Skip to content

armgit / ODS

Public Imported from GitHub
0

ODS V3 Pre-Release: Public testing and refinement ahead of the official V3 launch. Turn your PC, Mac, or Linux box into a private AI server.

fix(vllm): keep every kernel cache in the mounted cache folder (#8294)

Mike Bradley4e7e5112e3d9
diff --git a/ods/docker-compose.vllm.yml b/ods/docker-compose.vllm.yml
index 4b7d46a92..e57632867 100644
--- a/ods/docker-compose.vllm.yml
+++ b/ods/docker-compose.vllm.yml
@@ -48,6 +48,12 @@ services:
# in docker-compose.nvidia.yml); 4 GiB is the driver's maximum.
- CUDA_CACHE_PATH=/root/.cache/nv/ComputeCache
- CUDA_CACHE_MAXSIZE=4294967296
+ # The other kernel caches default outside /root/.cache (~/.tilelang,
+ # ~/.triton, /tmp/torchinductor_root), so every recreate rebuilt them:
+ # GLM-5.3-Flash JIT-compiles 136 TileLang kernels on each start.
+ - TILELANG_CACHE_DIR=/root/.cache/tilelang
+ - TRITON_CACHE_DIR=/root/.cache/triton
+ - TORCHINDUCTOR_CACHE_DIR=/root/.cache/torchinductor
volumes:
# The model's snapshot folder under data/models, read-only.
- ./data/models/${VLLM_MODEL_DIR:?VLLM_MODEL_DIR must name the model folder under data/models}:/model:ro,z
diff --git a/ods/tests/test-vllm-engine-service.sh b/ods/tests/test-vllm-engine-service.sh
index e21e89281..48ed0d78e 100755
--- a/ods/tests/test-vllm-engine-service.sh
+++ b/ods/tests/test-vllm-engine-service.sh
@@ -85,8 +85,9 @@ env = dict(item.split("=", 1) for item in env) if isinstance(env, list) else {k:
for key, value in {"VLLM_ENGINE_READY_TIMEOUT_S": "3600", "HF_HUB_OFFLINE": "1",
"VLLM_NO_USAGE_STATS": "1", "DO_NOT_TRACK": "1"}.items():
check(env.get(key) == value, f"vllm-server must set {key}={value}")
-check(env.get("CUDA_CACHE_PATH", "").startswith("/root/.cache/"),
- "the CUDA kernel cache must live in the mounted cache folder")
+for key in ("CUDA_CACHE_PATH", "TILELANG_CACHE_DIR", "TRITON_CACHE_DIR", "TORCHINDUCTOR_CACHE_DIR"):
+ check(env.get(key, "").startswith("/root/.cache/"),
+ f"{key} must point into the mounted cache folder, or a recreate rebuilds that kernel cache")
# Mounts: one model folder read-only, the compile/autotune cache writable.
mounts = {}