Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(compose): forward the autocast dtype overrides the runtime reads

Aashishd724a099bb62
diff --git a/compose.cuda.yaml b/compose.cuda.yaml
index 71dc27ed..fbdb90db 100644
--- a/compose.cuda.yaml
+++ b/compose.cuda.yaml
@@ -8,6 +8,10 @@ services:
TORCH_INDEX: "${LAYA_TORCH_INDEX:-cu128}"
environment:
LAYA_DEVICE: "${LAYA_DEVICE:-cuda}"
+ # The runtime reads these when it picks the autocast dtype; empty keeps the checkpoint's
+ # own `amp_dtype`. README's threshold section measures what fp16 vs bf16 decides differently.
+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"
+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"
deploy:
resources:
reservations:
@@ -23,6 +27,10 @@ services:
TORCH_INDEX: "${LAYA_TORCH_INDEX:-cu128}"
environment:
LAYA_DEVICE: "${LAYA_DEVICE:-cuda}"
+ # The runtime reads these when it picks the autocast dtype; empty keeps the checkpoint's
+ # own `amp_dtype`. README's threshold section measures what fp16 vs bf16 decides differently.
+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"
+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"
deploy:
resources:
reservations:
diff --git a/compose.http.yaml b/compose.http.yaml
index 2c5c82aa..e02df2c6 100644
--- a/compose.http.yaml
+++ b/compose.http.yaml
@@ -37,6 +37,10 @@ services:
LAYA_HOST: "${LAYA_HOST:-0.0.0.0}"
LAYA_PORT: "${LAYA_PORT:-8000}"
LAYA_DEVICE: "${LAYA_DEVICE:-cpu}"
+ # Autocast dtype the runtime reads when building the model; empty keeps the checkpoint's own
+ # `amp_dtype`. Set LAYA_CUDA_AMP=fp16 to serve fp16 on a GPU host.
+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"
+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"
LAYA_PRELOAD: "${LAYA_PRELOAD:-0}"
LAYA_MODELS: "${LAYA_MODELS:-}"
LAYA_THREADS: "${LAYA_THREADS:-${OMP_NUM_THREADS:-4}}"
diff --git a/compose.spark.yaml b/compose.spark.yaml
index a147f8f2..619c9b41 100644
--- a/compose.spark.yaml
+++ b/compose.spark.yaml
@@ -13,6 +13,10 @@ services:
TORCH_INDEX: "${LAYA_TORCH_INDEX:-cu130}"
environment:
LAYA_DEVICE: "${LAYA_DEVICE:-cuda}"
+ # Autocast dtype the runtime reads; empty keeps the checkpoint's own `amp_dtype`. Both
+ # services need it, because an override on `laya` never reaches `laya-serve`.
+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"
+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"
deploy:
resources:
reservations:
@@ -29,6 +33,10 @@ services:
TORCH_INDEX: "${LAYA_TORCH_INDEX:-cu130}"
environment:
LAYA_DEVICE: "${LAYA_DEVICE:-cuda}"
+ # Autocast dtype the runtime reads; empty keeps the checkpoint's own `amp_dtype`. Both
+ # services need it, because an override on `laya` never reaches `laya-serve`.
+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"
+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"
deploy:
resources:
reservations:
diff --git a/compose.yaml b/compose.yaml
index a53523c8..70ea7013 100644
--- a/compose.yaml
+++ b/compose.yaml
@@ -7,6 +7,12 @@ services:
TORCH_VERSION: "${LAYA_TORCH_VERSION:-2.14.0}"
environment:
LAYA_DEVICE: "${LAYA_DEVICE:-cpu}"
+ # The autocast dtype the runtime picks when it builds the model on the selected device
+ # (`laya/agent.py`). An empty value means "use the checkpoint's own `amp_dtype`", which is
+ # what these images have always served. README's threshold section measures what fp16 vs
+ # bf16 decides differently.
+ LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"
+ LAYA_CPU_AMP: "${LAYA_CPU_AMP:-}"
LAYA_MODEL: "${LAYA_MODEL:-auto}"
LAYA_MODEL_PATH: "${LAYA_MODEL_PATH:-}"
LAYA_REQUEST_FILE: "${LAYA_REQUEST_FILE:-/opt/laya/examples/request.json}"
diff --git a/docs/docker.md b/docs/docker.md
index 27487236..7bf7347b 100644
--- a/docs/docker.md
+++ b/docs/docker.md
@@ -68,6 +68,8 @@ work with `docker run -e`; Compose-only settings are identified below.
| Variable | Default | Purpose |
| --- | --- | --- |
| `LAYA_DEVICE` | `cpu` / `cuda` | Device selected by the base / GPU configuration |
+| `LAYA_CUDA_AMP` | unset (checkpoint's `amp_dtype`) | `fp16` or `bf16` for the CUDA forward. Not cosmetic: the README's threshold section measures bf16 flipping 3 of 864 argmaxes on the parity set where fp16 flips none |
+| `LAYA_CPU_AMP` | unset | `bf16` opts the CPU forward into bf16; anything else leaves it fp32 |
| `LAYA_MODEL` | `auto` | Router alias: `auto`, `english`, `multilingual`, `typed-decisions` |
| `LAYA_MODEL_PATH` | unset | Compatible checkpoint path inside the container |
| `LAYA_REQUEST_FILE` | bundled request | JSON request path inside the container |
@@ -83,7 +85,9 @@ work with `docker run -e`; Compose-only settings are identified below.
| `LAYA_TORCH_VERSION` | `2.14.0` | **Compose build:** pinned PyTorch version |
Compose forwards the runtime variables except `HF_HOME`, which stays aligned
-with its fixed cache mount. If overriding `HF_HOME` in `docker run` or your own
+with its fixed cache mount, and except `LAYA_MPS_AMP_MIN_ROWS`, the MPS row gate,
+which no image here can reach because no container here can select MPS.
+If overriding `HF_HOME` in `docker run` or your own
Compose file, provide a matching mount writable by UID 10001. Direct Docker
builds select PyTorch with `--build-arg TORCH_INDEX=cu128`; runtime `-e` cannot
change the installed wheel.