ODS V3 Pre-Release: Public testing and refinement ahead of the official V3 launch. Turn your PC, Mac, or Linux box into a private AI server.
AMD GPUs now run on llama.cpp
ODS used to serve models on AMD GPUs through Lemonade Server. It now runs
upstream llama.cpp's llama-server
on every platform, so the same engine, model files and API serve NVIDIA, AMD,
Apple and CPU installs:
- Linux with an AMD GPU: the official llama.cpp Vulkan container image, by default. A ROCm image is available as an option (see Choosing Vulkan or ROCm).
- Windows with an AMD GPU (the Portal): the official
llama-server.exeVulkan build, which runs on Windows. The rest of the stack stays in WSL. - Windows with an AMD GPU (the native installer): the same
llama-server.exebuild. - An OpenAI-compatible server you run yourself (including your own Lemonade Server): ODS connects to it as an external model server.
ODS no longer uses Lemonade Server. It never uninstalls or reconfigures a Lemonade Server installed on your computer. On Linux it removes the Docker image and volumes of the Lemonade container it used to run only when you uninstall ODS. If ODS installed Lemonade Server for you and you don't use it yourself, you can remove it (see Removing what ODS left behind).
What an upgrade does
On Linux and in the Windows Portal, the upgrade keeps the model you selected and its context size, as described below. Your model files stay where they are and are not downloaded again.
Linux with an AMD GPU
Rerun the installer of the new release, as for any update (see
How do I update ODS?). ods update only
refreshes container images, and the source updater (ods-update.sh update)
refuses a stack that builds its own image, as the Lemonade-era AMD stack did.
The installer:
- Moves the Lemonade-era settings in
.envto llama.cpp before anything else reads them, and prints "Moved the AMD settings from Lemonade to the llama.cpp runtime; your model files and active model are kept."ODS_MODE=lemonadebecomeslocal,LLM_BACKENDbecomesllama-server,LLM_API_BASE_PATHbecomes/v1, theAMD_INFERENCE_*settings describe the llama.cpp container, and the retired settings are removed. The install log names the keys it changed, never their values. - Keeps
GGUF_FILE, the active model and its context. llama.cpp serves that GGUF file fromdata/models. If Lemonade's model id named a different file, the installer says so. - Sets
AMD_INFERENCE_BACKEND=vulkan, whatever backend Lemonade used. Instinct cards get ROCm instead, because they have no Vulkan driver. - Removes values the Lemonade-era installer wrote:
HSA_OVERRIDE_GFX_VERSION=11.5.1andROCBLAS_USE_HIPBLASLT=1for Lemonade's Strix Halo build, and aLLAMA_SERVER_IMAGEmeant for another backend.LLAMA_ARG_SPLIT_MODE=rowbecomeslayer, because Vulkan has no row split. - Downloads the pinned llama.cpp image and starts the stack. The
llama-serverservice keeps its ports:127.0.0.1:11434on the host (OLLAMA_PORT) andllama-server:8080inside the ODS network. - Deletes the files ODS shipped for Lemonade when you never edited them, and names the edited ones it kept.
If the model router held a Lemonade route, the host agent replaces it and proves the model on llama.cpp before the router serves it again.
The upgrade leaves the Lemonade container's Docker volumes and its
ods-lemonade-server:latest image in place and prints the command that
removes them (see Removing what ODS left behind).
On a computer without an AMD GPU, the upgrade only removes or clears the retired settings.
Windows (Portal)
Run the newer install.ps1 over your existing installation, with the same
Ubuntu distribution and runtime directory (see the
Windows Quickstart).
Setup:
- Downloads the pinned
llama-server.exeVulkan build into%LOCALAPPDATA%\ODS\llama.cpp\. It checks the download's size and SHA-256 before extracting it. - Checks that it starts and that it sees your AMD GPU (
--version,--list-devices). While this runs, Lemonade keeps serving and nothing else changes. - Stops only the Lemonade task ODS created, starts the new task
ODSLlamaServerRuntime-<your SID>on the same port, and waits until it serves your model. When the task's plan names a model that is still in the model store, setup keeps that model and its context; otherwise it uses the recommended model. - If the new runtime does not come up, setup restores the previous files, restarts the Lemonade task if it was enabled, and stops with the reason. Your Lemonade setup is back as it was.
- Once the model is proven, setup removes the old ODS Lemonade task and launcher files and shows a one-time notice. Lemonade Server itself, its cache and its settings are not touched.
- Runs the Linux installer in Ubuntu, which moves its settings from Lemonade to the native llama-server route and prints "Moved the Windows GPU route from Lemonade to the native llama-server settings."
The new runtime listens on 127.0.0.1 only and requires an API key, which
ODS generates and keeps in a file only your Windows account can read. The
key never appears on a command line: setup passes it to the Linux installer
in an environment variable, and Ubuntu keeps it in .env as
LLAMA_SERVER_API_KEY.
The runtime's plan (portal-runtime) and the model store (models) stay in
%LOCALAPPDATA%\ODS\lemonade, which keeps its name.
Windows (native installer)
Rerun ods\installers\windows\install-windows.ps1 from the new source. The
installer:
- Downloads and checks
llama-server.exeas the Portal does, before it changes.envor any runtime. If that fails, nothing was changed. - Writes
.envfor llama-server and selects the model for your GPU, as on every rerun. A model file already indata\modelsthat matches its checksum is not downloaded again. - Disables ODS's
ODSLemonadeRuntimetask and stops only the processes it started. Lemonade Server itself is not changed. A task with that name that ODS did not write is left alone, and the installer says why. - Copies the verified runtime to
<install>\llama-server, keeps its API key file, launch options and log in%LOCALAPPDATA%\ODS\native-runtime, and starts the model through the newODSNativeLlamaRuntimetask, which runs at sign-in. - Once llama-server has proven its model, unregisters
ODSLemonadeRuntime, deletes thelogs\lemonade-launch.task.ps1launcher ODS wrote, and shows a one-time notice.
If you connected ODS to your own Lemonade
Linux installs made with --use-existing-lemonade are moved to the generic
external model server option by an installer rerun or, in a git checkout,
by ods-update.sh update. Your server keeps running as before; ODS does not
start or stop it.
- ODS keeps the server's address as
EXTERNAL_LLM_URL, without a trailing/api/v1,/v1or/api, and LiteLLM calls its/v1endpoint. - It keeps the model from
LEMONADE_MODELasEXTERNAL_LLM_MODEL. Without a saved model, the upgrade changes nothing and stops with the installer options to use instead. - An API key you gave ODS for the server moves to
config/litellm/external-upstream.key, readable only by you. If that file already holds a different key, ODS keeps the file and says so. - The installer shows "ODS no longer manages Lemonade. Your Lemonade server is now used as a generic OpenAI-compatible endpoint (EXTERNAL_LLM_URL)."
To change the model, make it available in your server, then rerun the
installer with --external-llm-model <id>. The Dashboard's Models page
shows "Model changes managed externally"; Adopt loaded model is gone. See
Can ODS reuse a model already running in Ollama or LM Studio?
for the external server options.
AMD GAIA
The AMD GAIA library recipe is removed too, because GAIA's local models need Lemonade Server. An installed GAIA keeps running until you disable it. The changelog explains how to remove it.
Retired settings and options
These .env settings are no longer used. They still validate for one
release, so an older .env does not stop an update. Upgrades remove them, and
the Dashboard's Settings page lists one only while your .env has it and
lets you clear it:
LEMONADE_EXTERNAL, LEMONADE_HOST_TRANSPORT, LEMONADE_BASE_URL,
LEMONADE_CONTAINER_BASE_URL, LEMONADE_API_BASE_PATH, LEMONADE_MODEL,
LEMONADE_API_KEY, LITELLM_LEMONADE_API_KEY, LEMONADE_SERVER_IMAGE,
LEMONADE_LLAMACPP, LEMONADE_LLAMACPP_ROCM_BIN, LLAMA_CPP_REF,
AMDGPU_TARGET, HSA_XNACK.
These values are retired the same way, and upgrades rewrite them:
ODS_MODE=lemonade (read as local), LLM_BACKEND=lemonade,
AMD_INFERENCE_RUNTIME=lemonade, AMD_INFERENCE_BACKEND auto, cpu or
npu, and AMD_INFERENCE_RUNTIME_MODE external-lemonade or
windows-legacy-lemonade.
Installer options:
| Old option | Use instead |
|---|---|
--use-existing-lemonade [--lemonade-url U] [--lemonade-model M] |
--external-llm-url U --external-llm-provider openai-compatible [--external-llm-model M] |
--lemonade-api-key K (with --use-existing-lemonade) |
--external-llm-key-file PATH, a file only you can read that holds the key |
--lemonade-url, --lemonade-model, --lemonade-host-transport, --lemonade-context-size, --lemonade-gpu-name, --lemonade-gpu-vram-mb (sent by older Windows setup) |
--native-llm-url, --native-llm-model, --native-llm-host-transport, --native-llm-context-size, --native-llm-gpu-name, --native-llm-gpu-vram-mb |
ODS_MODE=lemonade in the installer's environment |
Nothing: AMD installs use ODS_MODE=local |
The old options still work for one release and print the replacement. Without
--lemonade-url, --use-existing-lemonade uses http://localhost:13305.
The Dashboard API's /api/models/external-observation and
/api/models/external-adopt, and the host agent's
/v1/model/external-observation, /v1/model/external-adopt and
/v1/runtime/lemonade/ensure, answer HTTP 410 (external_lemonade_removed)
for one release.
Choosing Vulkan or ROCm on Linux
Vulkan is the default. The image brings its own Vulkan driver (Mesa RADV), so
the host needs only the amdgpu kernel driver's /dev/dri render node and
the video and render groups. It covers RDNA cards and APUs, Strix Halo
included.
The ROCm image is opt-in, except on Instinct cards: they have no Vulkan
driver, so the installer selects ROCm for them. The ROCm image is about 7 GB,
and it also needs the ROCm compute device /dev/kfd. To switch, rerun the
installer with the backend set, for example from your ODS source folder:
AMD_INFERENCE_BACKEND=rocm ./install.sh
The installer keeps the choice in .env, downloads the ROCm image and adds
docker-compose.amd-rocm.yml to the stack. To go back, rerun it with
AMD_INFERENCE_BACKEND=vulkan.
The ROCm image is built for gfx908, gfx90a, gfx942, gfx1030, gfx1100, gfx1101,
gfx1102, gfx1150, gfx1151, gfx1200 and gfx1201. For gfx1031 to gfx1036 the
installer sets HSA_OVERRIDE_GFX_VERSION=10.3.0, and for gfx1103 11.0.0.
For any other GPU it sets no override and warns that the model may not load;
set AMD_INFERENCE_BACKEND=vulkan if it does not.
Removing what ODS left behind
Windows (Portal). If setup's notice says ODS installed Lemonade Server for
you and you don't use it yourself, uninstall it from Settings > Apps. ODS's
own Lemonade task and launchers are already gone. A backup of the previous
runtime plan stays in %LOCALAPPDATA%\ODS\lemonade\, in a folder named
portal-runtime.lemonade-backup-<time>-<id>, until you delete it. Lemonade-era
logs there are kept for diagnosis; you can delete them too. Do not delete the
lemonade folder itself, or its portal-runtime and models folders: the
llama.cpp runtime uses them.
If setup said it left a former task registered because it changed or restarted after ODS stopped it, remove it from normal PowerShell once you no longer need it:
$odsSid = [Security.Principal.WindowsIdentity]::GetCurrent().User.Value
Unregister-ScheduledTask -TaskPath '\' -TaskName "ODSLemonadeRuntime-$odsSid" -Confirm:$false
Windows (native installer). Uninstall Lemonade Server from
Settings > Apps as above if the notice says ODS installed it. The
installer already removed the ODSLemonadeRuntime task. If it reported that
it left the task registered (disabled), remove it from normal PowerShell once
you no longer need it:
Unregister-ScheduledTask -TaskPath '\' -TaskName 'ODSLemonadeRuntime' -Confirm:$false
Linux. The volumes of the former Lemonade container and its image are no
longer used. Your model files are in data/models, not in these volumes.
ods-uninstall.sh removes them (the volumes unless you pass --keep-data).
To free the space now, run what the upgrade printed. With the default Compose
project name ods:
docker volume rm ods_lemonade-cache ods_lemonade-llama ods_lemonade-recipe
docker image rm ods-lemonade-server:latest
If you set COMPOSE_PROJECT_NAME, the volume names start with that name
instead. Files the installer kept because you edited them are named in its
output; nothing uses them now, so delete them when you no longer need them.
If an upgrade stops
- Windows (Portal). A failure before the new runtime has proven its model
restores the previous runtime, as described above, and setup ends with "The
llama.cpp runtime did not start" and the reason. If restoring also fails,
setup says so: ODS changed only its own files under
%LOCALAPPDATA%\ODS\lemonade, and its backup is kept there. Rerun setup. - Windows (native installer). A failure while preparing llama.cpp changes
nothing. A failure after that leaves the former
ODSLemonadeRuntimetask registered but disabled; Lemonade is not restarted. Fix the cause the installer names and rerun it. - Linux. The
.envchange is all or nothing: if it fails, the installer stops before any other step and nothing was changed. If a later step fails, rerun the installer. The settings move changes nothing on a.envthat has already moved.
This release has no way back to a Lemonade runtime that ODS runs. To keep using Lemonade, run it yourself and connect ODS to it as an external server.
Troubleshooting
llama-server.exeexits at once with a missing DLL (0xC0000135). Install the Microsoft Visual C++ 2015-2022 Redistributable (x64), for example withwinget install Microsoft.VCRedist.2015+.x64, then rerun setup.- Smart App Control or an application control policy blocks it. The llama.cpp binaries are not signed; ODS pins them by SHA-256 instead. Setup names the policy it found. ODS never changes these settings.
- No usable Vulkan device is found (Windows). Update the AMD Adrenalin driver from amd.com/support and restart Windows, then rerun setup. ODS does not fall back to the CPU silently: a new install continues on the CPU and says so, and an existing install stops without changing anything.
/dev/drior its render node is missing (Linux). The installer warns about it. Theamdgpudriver may not be loaded: trysudo modprobe amdgpuor reboot. The ROCm image also needs/dev/kfd(sudo modprobe amdkfd).- Another program holds the port (Windows). Setup names it. Set
AMD_INFERENCE_PORTto a free port in the same PowerShell window, for example$env:AMD_INFERENCE_PORT = "18080", and rerun setup. - A path or model name with non-ASCII characters (Windows). llama.cpp on Windows cannot open such a path unless the drive has 8.3 short names; setup says so. Use an ASCII user or folder name, and rename a model file whose name is not ASCII.
- Setup warns that the model needs more memory than Vulkan reports (Windows). llama.cpp keeps the rest in system memory, which is slower; a larger UMA frame buffer in the BIOS or a smaller model avoids that.
- Chat stopped working after an interrupted update. Rerun the same installer. It recognizes the migrated runtime and finishes the remaining steps; on Linux and in the Portal it keeps your model.