LLM inference in C/C++
C++ ggml
Commits on master
kv-cache : optimize restoring non-contiguous cells (#27991)
2d8d612 itsnotoger committed 2 months ago
opencl: tune the quant paths for Intel Xe-LP GPUs to improve its TG and PP performance (#26438)
010be96 Hongqiang Wang committed 2 months ago
ui: copy the displayed text of grouped agentic responses (#27832)
774ee0e Pascal committed 2 months ago
webgpu : avoid crash when offset is not multiple of 4 in WebGPU ggml_backend_tensor_get() implementation (#28045)
8e53fce fairydreaming committed 2 months ago
CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were restricted to 1 token (#27621)
41ef91f ynankani committed 2 months ago
sycl : Enhance to get the free memory of Intel GPU (#27968)
a32af33 Neo Zhang committed 2 months ago
spec : fuse the DFlash encoder into the KV cache injection (#27310)
662a0b0 王金旭 committed 2 months ago
vulkan: tune mat-vec rows for batched inference on Strix Halo (#27909)
2cdae80 Simon Teixidor committed 2 months ago
ggml : add MUL_MAT to the list of ops that may need additional memory (for WebGPU) (#28071)
557614e fairydreaming committed 2 months ago
vulkan: top_k radix select for k >= 1024 for Qwen 3.8 Flash Next (#28032)
daef7b6 Ruben Ortlam committed 2 months ago
metal : add remaining Q4_1/Q5_0/Q5_1 fa-vec tunings for M2 (#28017)
bd55e6a codemonkey committed 2 months ago
ggml : fix ggml_backend_buft_get_alloc_size() guard (#28038)
6d1479c Georgi Gerganov committed 2 months ago
kv-cells: stop the sequence scan once all sequences are seen (#28011)
62acc89 Pascal committed 2 months ago
CUDA: use the fast mm_ids_helper path for any n_expert_used (#27978)
f1793c1 Pascal committed 2 months ago
dflash: pass missing NVFP4 scales to attention operations (#28000)
cc231cb JamePeng committed 2 months ago
common: rename --tensor-read-lazy to --lazy-mode, add -lzm shorthand (#27969)
bebc935 Georgi Gerganov committed 2 months ago
ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC (#27960)
73f56d1 Georgi Gerganov committed 2 months ago