Skip to content

LLM inference in C/C++

C++ ggml

fit : count nextn (MTP) blocks in n_gpu_layers so front layers stay on GPU (#26177)

Jonas Jankaitis0324696b8e5f
diff --git a/common/fit.cpp b/common/fit.cpp
index c79221cb00..c82d066ad4 100644
--- a/common/fit.cpp
+++ b/common/fit.cpp
@@ -136,7 +136,7 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
devs.push_back(llama_model_get_device(model, i));
}
- hp_ngl = llama_model_n_layer(model);
+ hp_ngl = llama_model_n_layer(model) + llama_model_n_layer_nextn(model);
hp_n_ctx_train = llama_model_n_ctx_train(model);
hp_n_expert = llama_model_n_expert(model);