LLM inference in C/C++
C++ ggml
Commits on master
vulkan: fix stale prealloc_y reuse across flash attention and soft_max (#29591)
806eee9 François-Xavier Gsell committed 1 month ago
vulkan: sparse flash attention for quantized K/V (#29639)
b3daa07 François-Xavier Gsell committed 1 month ago
llama : fix unexpected graph reallocation in the k-pool models (#29958)
c173a53 Georgi Gerganov committed 1 month ago
kv-cache: fix restoring mismatched KV cache rotation by saving exact rotation metadata (#28498)
2107910 Evan Huus committed 1 month ago
ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 (#28479)
9f12cd4 Alan Tseng committed 1 month ago
vulkan : Fix undeclared identifiers when -DGGML_VULKAN_RUN_TESTS=ON (#29912)
ebe18be Ed Addario committed 1 month ago
webgpu: add MMVQ support for Q1_0/Q5_0/Q5_1/Q3_K/Q5_K/Q6_K/MXFP4 (#29483)
8216c84 Masashi Yoshimura committed 1 month ago
cuda: tile the lightning indexer over keys and tokens for 4 heads (#29901)
1b43d31 Pascal committed 1 month ago
CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size (#29633)
9d3aba6 ynankani committed 1 month ago
CUDA: prefer whole-tile FlashAttention scheduling for efficient two-stage kernels (#29435)
d89651a anujj committed 1 month ago
log, server: self contained colors, split child commands from logs in router mode (#29895)
a7fb71f Pascal committed 1 month ago
llama: support both embd + raw tokens in batch (#29622)
0bb496d Xuan-Son Nguyen committed 1 month ago
ggml-cpu: support BF16/FP16/FP32 K tails in tinyBLAS on x86 (#29806)
a7b94df SXX committed 1 month ago
ci : windows llvm build requires ninja multi-config (#29959)
2e7c58c Sigbjørn Skjæret committed 1 month ago
chat-peg-parser : clear current_tool when pending_tool_call is reset (#29942)
dbe4c3e Anas committed 1 month ago
cuda : move blocks_per_col to where it is used (#29939)
2bc5635 Adrien Gallouët committed 1 month ago
CUDA: fix MMQ memory fault if n_expert >> n_ubatch (#29941)
dd26678 Johannes Gäßler committed 1 month ago
imatrix: calculate activation-based statistics for new format (GGUF) imatrices (#14891)
0504396 Ed Addario committed 1 month ago
spec : fix n-gram drafts rejected at temp > 0 after truncation (#29924)
8330e96 Pranesh Gonegandla committed 1 month ago
common : prepare load_from_models_dir() for path conversion (#29674)
6716df6 Adrien Gallouët committed 1 month ago