ODS V3 Pre-Release: Public testing and refinement ahead of the official V3 launch. Turn your PC, Mac, or Linux box into a private AI server.
Python ai-agentsamdcomfyuidockerllama-cppllmlocal-ain8nnvidiaopen-webuiragself-hostedspeech-to-textstrix-halotext-to-speechworkflow-automation
fix(vllm): keep every kernel cache in the mounted cache folder (#8294)
Mike Bradley4e7e5112e3d9
diff --git a/ods/docker-compose.vllm.yml b/ods/docker-compose.vllm.ymlindex 4b7d46a92..e57632867 100644--- a/ods/docker-compose.vllm.yml+++ b/ods/docker-compose.vllm.yml@@ -48,6 +48,12 @@ services:# in docker-compose.nvidia.yml); 4 GiB is the driver's maximum.- CUDA_CACHE_PATH=/root/.cache/nv/ComputeCache- CUDA_CACHE_MAXSIZE=4294967296+ # The other kernel caches default outside /root/.cache (~/.tilelang,+ # ~/.triton, /tmp/torchinductor_root), so every recreate rebuilt them:+ # GLM-5.3-Flash JIT-compiles 136 TileLang kernels on each start.+ - TILELANG_CACHE_DIR=/root/.cache/tilelang+ - TRITON_CACHE_DIR=/root/.cache/triton+ - TORCHINDUCTOR_CACHE_DIR=/root/.cache/torchinductorvolumes:# The model's snapshot folder under data/models, read-only.- ./data/models/${VLLM_MODEL_DIR:?VLLM_MODEL_DIR must name the model folder under data/models}:/model:ro,zdiff --git a/ods/tests/test-vllm-engine-service.sh b/ods/tests/test-vllm-engine-service.shindex e21e89281..48ed0d78e 100755--- a/ods/tests/test-vllm-engine-service.sh+++ b/ods/tests/test-vllm-engine-service.sh@@ -85,8 +85,9 @@ env = dict(item.split("=", 1) for item in env) if isinstance(env, list) else {k:for key, value in {"VLLM_ENGINE_READY_TIMEOUT_S": "3600", "HF_HUB_OFFLINE": "1","VLLM_NO_USAGE_STATS": "1", "DO_NOT_TRACK": "1"}.items():check(env.get(key) == value, f"vllm-server must set {key}={value}")-check(env.get("CUDA_CACHE_PATH", "").startswith("/root/.cache/"),- "the CUDA kernel cache must live in the mounted cache folder")+for key in ("CUDA_CACHE_PATH", "TILELANG_CACHE_DIR", "TRITON_CACHE_DIR", "TORCHINDUCTOR_CACHE_DIR"):+ check(env.get(key, "").startswith("/root/.cache/"),+ f"{key} must point into the mounted cache folder, or a recreate rebuilds that kernel cache")# Mounts: one model folder read-only, the compile/autotune cache writable.mounts = {}