Skip to content

πŸ™Œ OpenHands: AI-Driven Development

docs: local serving with ollama documented (#8807)

Alexanderd81d2f62cbc1
diff --git a/docs/usage/llms/local-llms.mdx b/docs/usage/llms/local-llms.mdx
index 40c1ad94ba..39e58005be 100644
--- a/docs/usage/llms/local-llms.mdx
+++ b/docs/usage/llms/local-llms.mdx
@@ -126,6 +126,18 @@ vllm serve all-hands/openhands-lm-32b-v0.1 \
--enable-prefix-caching
```
+### Create an OpenAI-Compatible Endpoint with Ollama
+
+- Install Ollama following [the official documentation](https://ollama.com/download).
+- For Ollama configuration, use `ollama/<modelname>` as custom model in web. Api key also can be set to `ollama`.
+- Example launch command for Devstral LM 24B:
+
+```bash
+OLLAMA_CONTEXT_LENGTH=32768 OLLAMA_HOST=0.0.0.0:11434 OLLAMA_KEEP_ALIVE=-1 nohup ollama serve&
+#The minimum context size is ~8196, even the system prompt won't fit smaller
+ollama pull devstral:latest
+```
+
## Advanced: Run and Configure OpenHands
### Run OpenHands