Skip to content

LLM inference in C/C++

C++ ggml

server: avoid unnecessary checkpoint restore when new tokens are present (#24110)

Yongyue Sun6f3a9f3dee3c
diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp
index 28b1158c77..28f738c3fe 100644
--- a/tools/server/server-context.cpp
+++ b/tools/server/server-context.cpp
@@ -2782,8 +2782,11 @@ private:
llama_pos pos_next = slot.prompt.tokens.pos_next(n_past);
+ // ref: https://github.com/ggml-org/llama.cpp/pull/24110
+ const bool has_new_tokens = (n_past < slot.task->n_tokens());
+
// the largest pos_min required for a checkpoint to be useful
- const auto pos_min_thold = std::max(0, pos_next - n_swa - 1);
+ const auto pos_min_thold = std::max(0, pos_next - n_swa - (has_new_tokens ? 0 : 1));
if (n_past > 0 && n_past <= slot.prompt.n_tokens()) {
const auto pos_min = llama_memory_seq_pos_min(llama_get_memory(ctx_tgt), slot.id);