π OpenHands: AI-Driven Development
Add instruction to use Arctic Inference (#9547)
Ryan H. Tran8bc9207c2421
diff --git a/docs/usage/llms/local-llms.mdx b/docs/usage/llms/local-llms.mdxindex 0e75fb581c..a1d14d5c92 100644--- a/docs/usage/llms/local-llms.mdx+++ b/docs/usage/llms/local-llms.mdx@@ -179,6 +179,23 @@ If you are interested in further improved inference speed, you can also try Snowof vLLM, [ArcticInference](https://www.snowflake.com/en/engineering-blog/fast-speculative-decoding-vllm-arctic/),which can achieve up to 2x speedup in some cases.+1. Install the Arctic Inference library that automatically patches vLLM:++```bash+pip install git+https://github.com/snowflakedb/ArcticInference.git+```++2. Run the launch command with speculative decoding enabled:++```bash+vllm serve mistralai/Devstral-Small-2505 \+ --host 0.0.0.0 --port 8000 \+ --api-key mykey \+ --tensor-parallel-size 2 \+ --served-model-name Devstral-Small-2505 \+ --speculative-config '{"method": "suffix"}'+```+### Run OpenHands (Alternative Backends)#### Using Docker