Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
fix(compose): forward the autocast dtype overrides the runtime reads
Aashishd724a099bb62
diff --git a/compose.cuda.yaml b/compose.cuda.yamlindex 71dc27ed..fbdb90db 100644--- a/compose.cuda.yaml+++ b/compose.cuda.yaml@@ -8,6 +8,10 @@ services:TORCH_INDEX: "${LAYA_TORCH_INDEX:-cu128}"environment:LAYA_DEVICE: "${LAYA_DEVICE:-cuda}"+ # The runtime reads these when it picks the autocast dtype; empty keeps the checkpoint's+ # own `amp_dtype`. README's threshold section measures what fp16 vs bf16 decides differently.+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"deploy:resources:reservations:@@ -23,6 +27,10 @@ services:TORCH_INDEX: "${LAYA_TORCH_INDEX:-cu128}"environment:LAYA_DEVICE: "${LAYA_DEVICE:-cuda}"+ # The runtime reads these when it picks the autocast dtype; empty keeps the checkpoint's+ # own `amp_dtype`. README's threshold section measures what fp16 vs bf16 decides differently.+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"deploy:resources:reservations:diff --git a/compose.http.yaml b/compose.http.yamlindex 2c5c82aa..e02df2c6 100644--- a/compose.http.yaml+++ b/compose.http.yaml@@ -37,6 +37,10 @@ services:LAYA_HOST: "${LAYA_HOST:-0.0.0.0}"LAYA_PORT: "${LAYA_PORT:-8000}"LAYA_DEVICE: "${LAYA_DEVICE:-cpu}"+ # Autocast dtype the runtime reads when building the model; empty keeps the checkpoint's own+ # `amp_dtype`. Set LAYA_CUDA_AMP=fp16 to serve fp16 on a GPU host.+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"LAYA_PRELOAD: "${LAYA_PRELOAD:-0}"LAYA_MODELS: "${LAYA_MODELS:-}"LAYA_THREADS: "${LAYA_THREADS:-${OMP_NUM_THREADS:-4}}"diff --git a/compose.spark.yaml b/compose.spark.yamlindex a147f8f2..619c9b41 100644--- a/compose.spark.yaml+++ b/compose.spark.yaml@@ -13,6 +13,10 @@ services:TORCH_INDEX: "${LAYA_TORCH_INDEX:-cu130}"environment:LAYA_DEVICE: "${LAYA_DEVICE:-cuda}"+ # Autocast dtype the runtime reads; empty keeps the checkpoint's own `amp_dtype`. Both+ # services need it, because an override on `laya` never reaches `laya-serve`.+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"deploy:resources:reservations:@@ -29,6 +33,10 @@ services:TORCH_INDEX: "${LAYA_TORCH_INDEX:-cu130}"environment:LAYA_DEVICE: "${LAYA_DEVICE:-cuda}"+ # Autocast dtype the runtime reads; empty keeps the checkpoint's own `amp_dtype`. Both+ # services need it, because an override on `laya` never reaches `laya-serve`.+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"deploy:resources:reservations:diff --git a/compose.yaml b/compose.yamlindex a53523c8..70ea7013 100644--- a/compose.yaml+++ b/compose.yaml@@ -7,6 +7,12 @@ services:TORCH_VERSION: "${LAYA_TORCH_VERSION:-2.14.0}"environment:LAYA_DEVICE: "${LAYA_DEVICE:-cpu}"+ # The autocast dtype the runtime picks when it builds the model on the selected device+ # (`laya/agent.py`). An empty value means "use the checkpoint's own `amp_dtype`", which is+ # what these images have always served. README's threshold section measures what fp16 vs+ # bf16 decides differently.+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"LAYA_MODEL: "${LAYA_MODEL:-auto}"LAYA_MODEL_PATH: "${LAYA_MODEL_PATH:-}"LAYA_REQUEST_FILE: "${LAYA_REQUEST_FILE:-/opt/laya/examples/request.json}"diff --git a/docs/docker.md b/docs/docker.mdindex 27487236..7bf7347b 100644--- a/docs/docker.md+++ b/docs/docker.md@@ -68,6 +68,8 @@ work with `docker run -e`; Compose-only settings are identified below.| Variable | Default | Purpose || --- | --- | --- || `LAYA_DEVICE` | `cpu` / `cuda` | Device selected by the base / GPU configuration |+| `LAYA_CUDA_AMP` | unset (checkpoint's `amp_dtype`) | `fp16` or `bf16` for the CUDA forward. Not cosmetic: the README's threshold section measures bf16 flipping 3 of 864 argmaxes on the parity set where fp16 flips none |+| `LAYA_CPU_AMP` | unset | `bf16` opts the CPU forward into bf16; anything else leaves it fp32 || `LAYA_MODEL` | `auto` | Router alias: `auto`, `english`, `multilingual`, `typed-decisions` || `LAYA_MODEL_PATH` | unset | Compatible checkpoint path inside the container || `LAYA_REQUEST_FILE` | bundled request | JSON request path inside the container |@@ -83,7 +85,9 @@ work with `docker run -e`; Compose-only settings are identified below.| `LAYA_TORCH_VERSION` | `2.14.0` | **Compose build:** pinned PyTorch version |Compose forwards the runtime variables except `HF_HOME`, which stays aligned-with its fixed cache mount. If overriding `HF_HOME` in `docker run` or your own+with its fixed cache mount, and except `LAYA_MPS_AMP_MIN_ROWS`, the MPS row gate,+which no image here can reach because no container here can select MPS.+If overriding `HF_HOME` in `docker run` or your ownCompose file, provide a matching mount writable by UID 10001. Direct Dockerbuilds select PyTorch with `--build-arg TORCH_INDEX=cu128`; runtime `-e` cannotchange the installed wheel.