Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(docker): keep torch's native Triton JIT off so a GPU image can infer without a C compiler
Bhushan Kinge846a75d3fe28
diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.ymlindex d08718f8..cea69f13 100644--- a/.github/workflows/docker.yml+++ b/.github/workflows/docker.yml@@ -34,6 +34,9 @@ jobs:print(torch.__version__, cusparselt.version())'docker run --rm --network none laya:spark pip check+ # torch 2.14 compiles Triton kernels on the first CUDA inference unless this is set (#365).+ - name: A CUDA image must keep torch's native Triton JIT off+ run: docker run --rm --network none laya:spark sh -c 'test "$TORCH_DISABLE_NATIVE_JIT" = 1'smoke:# Native runners for both architectures; torch wheels install for the target, sodiff --git a/Dockerfile b/Dockerfileindex caa02f40..3806d4b1 100644--- a/Dockerfile+++ b/Dockerfile@@ -33,7 +33,12 @@ LABEL org.opencontainers.image.title="Laya Docker quickstart" \org.opencontainers.image.source="https://github.com/NandhaKishorM/laya" \org.opencontainers.image.licenses="Apache-2.0"+# torch 2.14 swaps some eager CUDA ops (bmm, topk, sum, norms) for Triton kernels that it+# compiles on the first inference, which needs a C compiler this image does not carry: the+# container reports healthy, then every request fails (#365). The stock kernels give the same+# answers at the same latency.ENV PATH="/opt/venv/bin:$PATH" \+ TORCH_DISABLE_NATIVE_JIT=1 \PYTHONUNBUFFERED=1 \PYTHONDONTWRITEBYTECODE=1 \USE_TF=0 \diff --git a/docs/docker.md b/docs/docker.mdindex ffc87706..27487236 100644--- a/docs/docker.md+++ b/docs/docker.md@@ -48,6 +48,12 @@ The sample rejects unavailable CUDA before loading a checkpoint. Laya can stillfall back to CPU after a memory or inference error, so inspect its warnings.Rebuild when switching between CPU and CUDA configurations.+The image sets `TORCH_DISABLE_NATIVE_JIT=1`. PyTorch 2.14 otherwise replaces some eager+CUDA ops with Triton kernels that it compiles on the first inference, which needs a C+compiler the slim image does not carry: the container reports healthy and then fails every+request (#365). The stock kernels give the same answers at the same latency. Set the same+variable on a bare-metal install if `predict` fails with `Failed to find C compiler`.+This uses [Compose GPU reservations](https://docs.docker.com/compose/how-tos/gpu-support/).Windows requires Docker Desktop's supported WSL2 GPU setup. Apple MPS,AMD/ROCm and Intel GPU containers are outside this quickstart; use CPU unlessdiff --git a/tests/test_packaging.py b/tests/test_packaging.pyindex 7c685eaa..66971a35 100644--- a/tests/test_packaging.py+++ b/tests/test_packaging.py@@ -184,6 +184,12 @@ check_true("compose.http/leaves the quickstart service alone",check_true("Dockerfile/installs the serve extra", '".[serve]"' in dockerfile, dockerfile[:400])check_true("Dockerfile/still runs pip check", "pip check" in dockerfile)+# torch 2.14's eager Triton kernels compile on the first CUDA inference and need a C compiler the+# slim runtime image does not have (#365). The kill switch keeps the stock kernels.+check_true("Dockerfile/runtime stage disables torch's native Triton JIT (#365)",+ re.search(r"^\s*TORCH_DISABLE_NATIVE_JIT=1", dockerfile.partition("AS runtime")[2], re.M) is not None,+ "without TORCH_DISABLE_NATIVE_JIT=1 a GPU image serves 500s while /health stays green")+# Overrides for `laya` never reach `laya-serve`, a separate service. If the CUDA override does# not repeat the args for laya-serve, that service silently serves on CPU.check_true("compose.cuda/covers laya-serve too",