Skip to content

LLM inference in C/C++

C++ ggml

server : forward llama.cpp sampling params on /v1/messages (#30856)

David Garcia261700b896b9
diff --git a/tools/server/README.md b/tools/server/README.md
index 2886bca158..71310d9030 100644
--- a/tools/server/README.md
+++ b/tools/server/README.md
@@ -1719,6 +1719,8 @@ See [Anthropic Messages API documentation](https://docs.anthropic.com/en/api/mes
`top_k`: Top-k sampling
+`extra_params`: llama.cpp parameters that the Anthropic API has no field for, such as `seed`, `min_p` or `repeat_penalty`, as an object (e.g. `{"seed": 42, "min_p": 0.05}`). Its keys are merged into the converted request and behave as on `/v1/chat/completions`.
+
`stop_sequences`: Array of stop sequences
`stream`: Enable streaming (default: false)
@@ -1739,7 +1741,8 @@ curl http://localhost:9931/v1/messages \
"system": "You are a helpful assistant.",
"messages": [
{"role": "user", "content": "Hello!"}
- ]
+ ],
+ "extra_params": {"seed": 42, "min_p": 0.05}
}'
```
diff --git a/tools/server/server-chat.cpp b/tools/server/server-chat.cpp
index c5473417c3..1c2326f33c 100644
--- a/tools/server/server-chat.cpp
+++ b/tools/server/server-chat.cpp
@@ -615,6 +615,13 @@ json server_chat_convert_anthropic_to_oai(const json & body) {
}
}
+ // llama.cpp params (e.g. seed, min_p) go under extra_params and are merged into the converted body
+ if (body.contains("extra_params") && body.at("extra_params").is_object()) {
+ for (const auto & [key, value] : body.at("extra_params").items()) {
+ oai_body[key] = value;
+ }
+ }
+
return oai_body;
}