Distribute and run LLMs with a single file.
API server
When llamafile runs in server mode, it exposes the HTTP APIs inherited from
llama-server, llama.cpp's server component. The default base address is
http://127.0.0.1:8080.
The server provides:
- OpenAI-compatible APIs under
/v1. - An Anthropic-compatible Messages API.
- Native llama.cpp endpoints for lower-level server features.
- An experimental internal endpoint for the bundled Web UI's built-in tools.
Note that the supported fields and behavior come from the llama.cpp version bundled into a particular llamafile release and can change between releases.
Address and authentication
Use --host and --port to change the listening address and port.
--api-prefix prepends a path to every endpoint. For example,
--api-prefix /llama changes /v1/chat/completions to
/llama/v1/chat/completions.
Configure authentication with --api-key or --api-key-file. Protected
requests accept either header:
Authorization: Bearer YOUR_API_KEY
X-Api-Key: YOUR_API_KEY
The second form is also compatible with clients that send Anthropic's
x-api-key header; HTTP header names are case-insensitive. When no API key is
configured, omit the header.
[!WARNING] As a general rule, avoid making your server accessible to more clients than you actually need. Start with the default host binding, which listens on localhost only, and change it only if you need to access the server from another machine. When you do this, configure
--api-keyor--api-key-fileso reachable clients still need authentication.Also note: the Web UI uses CORS to limit server responses to a given set of browser origins. While this helps limiting server access to some extent, it should not be used instead of authentication.
OpenAI-compatible APIs
Chat Completions
POST /v1/chat/completions accepts OpenAI-style chat messages, e.g.:
curl http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "local-model",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a limerick about llamas."}
]
}'
The model field is required by the OpenAI-style schema even when only one
model is loaded. Clients commonly use a placeholder such as local-model. Set
--alias if you want a stable model name in requests and responses.
OpenAI SDKs read their base URL and API key from environment variables:
export OPENAI_BASE_URL=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=no-key-required
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="local-model",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
The SDK reads both variables automatically. It requires a non-empty API key
value even when the llamafile server does not require authentication. If you
set an API key via the --api-key argument as described above, set it as your
OPENAI_API_KEY.
Responses
POST /v1/responses accepts OpenAI Responses-style input. The server converts
it internally to a Chat Completions request:
curl http://127.0.0.1:8080/v1/responses \
-H 'Content-Type: application/json' \
-d '{
"model": "local-model",
"instructions": "You are a helpful assistant.",
"input": "Explain why llamas hum.",
"max_output_tokens": 256
}'
input can be a string or an array of supported input items. Streaming is
available with "stream": true. previous_response_id is not supported;
send the conversation history needed for the next request in input.
Other OpenAI-compatible endpoints
| Endpoint | Purpose |
|---|---|
GET /v1/models |
List model metadata and capabilities. |
POST /v1/completions |
Generate text from a raw prompt. |
POST /v1/embeddings |
Create embeddings with a pooling-enabled model. |
POST /v1/rerank |
Rank documents with a reranker model. /v1/reranking is an alias. |
POST /v1/audio/transcriptions |
Transcribe audio with a compatible model. |
POST /v1/chat/completions/input_tokens |
Count tokens for a Chat Completions request. |
POST /v1/responses/input_tokens |
Count tokens for a Responses request. |
POST /v1/chat/completions/control |
Control an in-progress chat completion that enabled realtime reasoning control. |
Model and build capabilities determine whether specialized endpoints such as embeddings, reranking, multimodal input, and transcription can succeed.
Anthropic-compatible API
Messages
POST /v1/messages accepts Anthropic Messages-style requests, including a
system prompt, content blocks, stop sequences, streaming, and tools:
curl http://127.0.0.1:8080/v1/messages \
-H 'Content-Type: application/json' \
-d '{
"model": "local-model",
"max_tokens": 512,
"system": "You are a helpful assistant.",
"messages": [
{"role": "user", "content": "Explain why llamas hum."}
]
}'
The main supported request fields are:
| Field | Description |
|---|---|
model |
Model identifier. Required by the Anthropic-style schema. |
messages |
Conversation messages. |
max_tokens |
Maximum generated tokens. Defaults to 4096 when omitted. |
system |
System prompt as a string or text-block array. |
temperature, top_p, top_k |
Sampling controls. |
stop_sequences |
Strings that stop generation. |
stream |
Return server-sent events when true. |
tools, tool_choice |
Anthropic-style function definitions and selection mode. |
POST /v1/messages/count_tokens accepts the same conversation shape and
returns its input-token count without generating a response:
{
"input_tokens": 10
}
Tool calling
The OpenAI- and Anthropic-compatible APIs can ask a model to call functions, but they do not execute those functions. A client must run the agent loop:
- Send tool definitions and conversation messages to the model.
- Inspect the response for requested tool calls.
- Validate the arguments and ask the user for any required approval.
- Execute each approved call in the client or through
POST /tools. - Return each result in the selected API's tool-result format.
- Call the model again and repeat until it returns a normal answer.
--tools is needed only when the client will execute llama.cpp's enabled
built-in tools through /tools; client-defined functions
do not require it.
The three compatible APIs use different tool-call shapes:
| API | Send tool definitions as | Requested call appears as | Return the result as |
|---|---|---|---|
| OpenAI Chat Completions | tools array with nested function objects |
assistant message.tool_calls[] |
next request message with role: "tool" and the matching tool_call_id |
| OpenAI Responses | tools array of function objects |
output[] item with type: "function_call" |
next request input[] item with type: "function_call_output" |
| Anthropic Messages | tools array with input_schema |
assistant content block with type: "tool_use" |
next user message content block with type: "tool_result" |
To execute one of llama.cpp's enabled built-in tools, call POST /tools with
the tool name and arguments selected by the model. This internal endpoint is
experimental and does not display the Web UI's permission prompt. See
Built-in local tools for tool discovery, security notes,
and the current /tools interface.
Native llama.cpp endpoints
Native endpoints expose lower-level features that are not part of the OpenAI or Anthropic schemas:
| Endpoint | Purpose |
|---|---|
GET /health |
Report readiness. /v1/health is an alias. |
GET /props, POST /props |
Read or update server properties. |
GET /models |
List models and their load state. |
POST /completion |
Generate from a raw prompt with native llama.cpp options. |
POST /tokenize, POST /detokenize |
Convert between text and token IDs. |
POST /apply-template |
Apply the active chat template without inference. |
POST /infill |
Perform fill-in-the-middle code completion. |
POST /embedding |
Create embeddings in the native response format. |
POST /rerank |
Rank documents in the native response format. |
GET /slots, POST /slots/:id_slot |
Inspect slots and manage prompt-cache state. |
GET /lora-adapters, POST /lora-adapters |
Inspect or change loaded LoRA adapter scales. |
GET /metrics |
Return Prometheus metrics when enabled with --metrics. |
GET /tools, POST /tools |
List and invoke experimental built-in or MCP tools. |
Consult the bundled server help and upstream server documentation before building automation around native endpoints, which change more frequently than the compatibility endpoints.
Errors
API errors normally use an OpenAI-style envelope:
{
"error": {
"code": 400,
"message": "Description of the invalid request",
"type": "invalid_request_error"
}
}
Common error types include invalid_request_error, authentication_error,
permission_error, not_found_error, server_error, and
not_supported_error. The HTTP status code carries the same general category.
Further reference
This page is an overview of the endpoints most useful to llamafile users. For
all accepted generation parameters and lower-level response fields, see the
llama.cpp server README
and check llamafile --server --help for the options available in the current
executable.