Skip to content

Distribute and run LLMs with a single file.

Recognize cache_prompt parameter in OpenAI API

Georgi Gerganov8cb9fd8479fc
diff --git a/llama.cpp/README.llamafile b/llama.cpp/README.llamafile
index 9510b544..bf6ab964 100644
--- a/llama.cpp/README.llamafile
+++ b/llama.cpp/README.llamafile
@@ -20,8 +20,8 @@ ORIGIN
allocation, scheduling, backends, etc. So far, we've curated up to:
https://github.com/ggerganov/llama.cpp
- 8d6d9f033b8101f929e445cf45b39e1557ca7934
- Fri Dec 1 11:41:56 2023 +0200
+ fe680e3d1080a765e5d3150ffd7bab189742898d
+ Thu Dec 7 22:26:54 2023 +0200
See the llamafile git history re: cherry-picks for further details.
diff --git a/llama.cpp/server/server.cpp b/llama.cpp/server/server.cpp
index a81b498b..172e4a26 100644
--- a/llama.cpp/server/server.cpp
+++ b/llama.cpp/server/server.cpp
@@ -2380,6 +2380,7 @@ json oaicompat_completion_params_parse(
// Map OpenAI parameters to llama.cpp parameters
llama_params["prompt"] = format_chatml(body["messages"]); // OpenAI 'messages' to llama.cpp 'prompt'
+ llama_params["cache_prompt"] = json_value(body, "cache_prompt", false);
llama_params["temperature"] = json_value(body, "temperature", 0.8);
llama_params["top_k"] = json_value(body, "top_k", 40);
llama_params["top_p"] = json_value(body, "top_p", 0.95);