Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(docker): keep torch's native Triton JIT off so a GPU image can infer without a C compiler

Bhushan Kinge846a75d3fe28
diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml
index d08718f8..cea69f13 100644
--- a/.github/workflows/docker.yml
+++ b/.github/workflows/docker.yml
@@ -34,6 +34,9 @@ jobs:
print(torch.__version__, cusparselt.version())
'
docker run --rm --network none laya:spark pip check
+ # torch 2.14 compiles Triton kernels on the first CUDA inference unless this is set (#365).
+ - name: A CUDA image must keep torch's native Triton JIT off
+ run: docker run --rm --network none laya:spark sh -c 'test "$TORCH_DISABLE_NATIVE_JIT" = 1'
smoke:
# Native runners for both architectures; torch wheels install for the target, so
diff --git a/Dockerfile b/Dockerfile
index caa02f40..3806d4b1 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -33,7 +33,12 @@ LABEL org.opencontainers.image.title="Laya Docker quickstart" \
org.opencontainers.image.source="https://github.com/NandhaKishorM/laya" \
org.opencontainers.image.licenses="Apache-2.0"
+# torch 2.14 swaps some eager CUDA ops (bmm, topk, sum, norms) for Triton kernels that it
+# compiles on the first inference, which needs a C compiler this image does not carry: the
+# container reports healthy, then every request fails (#365). The stock kernels give the same
+# answers at the same latency.
ENV PATH="/opt/venv/bin:$PATH" \
+ TORCH_DISABLE_NATIVE_JIT=1 \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
USE_TF=0 \
diff --git a/docs/docker.md b/docs/docker.md
index ffc87706..27487236 100644
--- a/docs/docker.md
+++ b/docs/docker.md
@@ -48,6 +48,12 @@ The sample rejects unavailable CUDA before loading a checkpoint. Laya can still
fall back to CPU after a memory or inference error, so inspect its warnings.
Rebuild when switching between CPU and CUDA configurations.
+The image sets `TORCH_DISABLE_NATIVE_JIT=1`. PyTorch 2.14 otherwise replaces some eager
+CUDA ops with Triton kernels that it compiles on the first inference, which needs a C
+compiler the slim image does not carry: the container reports healthy and then fails every
+request (#365). The stock kernels give the same answers at the same latency. Set the same
+variable on a bare-metal install if `predict` fails with `Failed to find C compiler`.
+
This uses [Compose GPU reservations](https://docs.docker.com/compose/how-tos/gpu-support/).
Windows requires Docker Desktop's supported WSL2 GPU setup. Apple MPS,
AMD/ROCm and Intel GPU containers are outside this quickstart; use CPU unless
diff --git a/tests/test_packaging.py b/tests/test_packaging.py
index 7c685eaa..66971a35 100644
--- a/tests/test_packaging.py
+++ b/tests/test_packaging.py
@@ -184,6 +184,12 @@ check_true("compose.http/leaves the quickstart service alone",
check_true("Dockerfile/installs the serve extra", '".[serve]"' in dockerfile, dockerfile[:400])
check_true("Dockerfile/still runs pip check", "pip check" in dockerfile)
+# torch 2.14's eager Triton kernels compile on the first CUDA inference and need a C compiler the
+# slim runtime image does not have (#365). The kill switch keeps the stock kernels.
+check_true("Dockerfile/runtime stage disables torch's native Triton JIT (#365)",
+ re.search(r"^\s*TORCH_DISABLE_NATIVE_JIT=1", dockerfile.partition("AS runtime")[2], re.M) is not None,
+ "without TORCH_DISABLE_NATIVE_JIT=1 a GPU image serves 500s while /health stays green")
+
# Overrides for `laya` never reach `laya-serve`, a separate service. If the CUDA override does
# not repeat the args for laya-serve, that service silently serves on CPU.
check_true("compose.cuda/covers laya-serve too",