ODS V3 Pre-Release: Public testing and refinement ahead of the official V3 launch. Turn your PC, Mac, or Linux box into a private AI server.
Python ai-agentsamdcomfyuidockerllama-cppllmlocal-ain8nnvidiaopen-webuiragself-hostedspeech-to-textstrix-halotext-to-speechworkflow-automation
services:
llama-server:
environment:
NVIDIA_VISIBLE_DEVICES: "${LLAMA_SERVER_GPU_UUIDS:?LLAMA_SERVER_GPU_UUIDS must be set for the NVIDIA multi-GPU overlay}"
# NVIDIA_VISIBLE_DEVICES alone does not limit llama-server: the base
# overlay reserves `count: all`, Docker then appends its own
# NVIDIA_VISIBLE_DEVICES=all, and CDI ignores the variable (all four
# GPUs stayed visible on a 4x RTX PRO 6000 host, 2026-10-27). CUDA
# honors CUDA_VISIBLE_DEVICES, in the listed order, so llama.cpp sees
# exactly the GPUs the assignment gave it and LLAMA_ARG_TENSOR_SPLIT
# applies to them.
CUDA_VISIBLE_DEVICES: "${LLAMA_SERVER_GPU_UUIDS:?LLAMA_SERVER_GPU_UUIDS must be set for the NVIDIA multi-GPU overlay}"
LLAMA_ARG_SPLIT_MODE: "${LLAMA_ARG_SPLIT_MODE:-none}"
LLAMA_ARG_TENSOR_SPLIT: "${LLAMA_ARG_TENSOR_SPLIT:-}"
deploy:
resources:
reservations:
devices:
- driver: nvidia
capabilities: [gpu]