LLM inference in C/C++
C++ ggml
fit : count nextn (MTP) blocks in n_gpu_layers so front layers stay on GPU (#26177)
Jonas Jankaitis0324696b8e5f
diff --git a/common/fit.cpp b/common/fit.cppindex c79221cb00..c82d066ad4 100644--- a/common/fit.cpp+++ b/common/fit.cpp@@ -136,7 +136,7 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(devs.push_back(llama_model_get_device(model, i));}- hp_ngl = llama_model_n_layer(model);+ hp_ngl = llama_model_n_layer(model) + llama_model_n_layer_nextn(model);hp_n_ctx_train = llama_model_n_ctx_train(model);hp_n_expert = llama_model_n_expert(model);