LLM inference in C/C++
C++ ggml
server : forward llama.cpp sampling params on /v1/messages (#30856)
David Garcia261700b896b9
diff --git a/tools/server/README.md b/tools/server/README.mdindex 2886bca158..71310d9030 100644--- a/tools/server/README.md+++ b/tools/server/README.md@@ -1719,6 +1719,8 @@ See [Anthropic Messages API documentation](https://docs.anthropic.com/en/api/mes`top_k`: Top-k sampling+`extra_params`: llama.cpp parameters that the Anthropic API has no field for, such as `seed`, `min_p` or `repeat_penalty`, as an object (e.g. `{"seed": 42, "min_p": 0.05}`). Its keys are merged into the converted request and behave as on `/v1/chat/completions`.+`stop_sequences`: Array of stop sequences`stream`: Enable streaming (default: false)@@ -1739,7 +1741,8 @@ curl http://localhost:9931/v1/messages \"system": "You are a helpful assistant.","messages": [{"role": "user", "content": "Hello!"}- ]+ ],+ "extra_params": {"seed": 42, "min_p": 0.05}}'```diff --git a/tools/server/server-chat.cpp b/tools/server/server-chat.cppindex c5473417c3..1c2326f33c 100644--- a/tools/server/server-chat.cpp+++ b/tools/server/server-chat.cpp@@ -615,6 +615,13 @@ json server_chat_convert_anthropic_to_oai(const json & body) {}}+ // llama.cpp params (e.g. seed, min_p) go under extra_params and are merged into the converted body+ if (body.contains("extra_params") && body.at("extra_params").is_object()) {+ for (const auto & [key, value] : body.at("extra_params").items()) {+ oai_body[key] = value;+ }+ }+return oai_body;}