Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(serve): support reverse-proxy URL prefixes

Som Samantray1ca5e754c413
diff --git a/README.md b/README.md
index 4aadb0c4..a132f868 100644
--- a/README.md
+++ b/README.md
@@ -500,6 +500,10 @@ send `Authorization: Bearer <key>`). A client's `model` field is honoured when i
names a Laya checkpoint (`english`/`multilingual`/`typed-decisions`), otherwise
the router auto-selects by script/language.
+When publishing the server under a reverse-proxy prefix such as `/laya`, set
+`LAYA_ROOT_PATH=/laya`. The proxy should strip that prefix before forwarding;
+the setting controls FastAPI-generated URLs and leaves the internal routes unchanged.
+
Three things differ from Jev when you port a client:
* **Options per question.** A question's options share the checkpoint's option budget, `head_max_len` (192 tokens on `laya`, 256 on the other two), not Jev's cap of 255 options. In addition, the HTTP server (`laya.serve`) enforces an amplification guard of at most 100 choice options per question (`MAX_CHOICE_OPTIONS = 100`, rejected with 413 before inference). Once options overflow the token budget, around 20 options with a short description each, every option is trimmed to fit, so long or similar labels can reach the model reading the same ([Where Jev leads](#where-jev-leads)). Once they no longer fit the window at all, the library rejects the request with 422. For more candidates, narrow them first with `predict_shortlist` ([Honest limits](#honest-limits)).
diff --git a/compose.http.yaml b/compose.http.yaml
index cd7f7f65..3563b6e8 100644
--- a/compose.http.yaml
+++ b/compose.http.yaml
@@ -37,6 +37,8 @@ services:
LAYA_HOST: "${LAYA_HOST:-0.0.0.0}"
LAYA_PORT: "${LAYA_PORT:-8000}"
LAYA_DEVICE: "${LAYA_DEVICE:-cpu}"
+ # FastAPI's external URL prefix behind a reverse proxy; forward stripped paths.
+ LAYA_ROOT_PATH: "${LAYA_ROOT_PATH:-}"
# Autocast dtype the runtime reads when building the model; empty keeps the checkpoint's own
# `amp_dtype`. Set LAYA_CUDA_AMP=fp16 to serve fp16 on a GPU host.
LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"
diff --git a/docs/docker.md b/docs/docker.md
index 99696d21..03d2c55d 100644
--- a/docs/docker.md
+++ b/docs/docker.md
@@ -241,8 +241,13 @@ These apply to the `laya-serve` service only.
| `LAYA_MAX_CONCURRENT` | `16` | requests admitted at once; later ones get `503` (a value that does not parse, or is not positive, falls back to `16`) |
| `LAYA_LOG_LEVEL` | `info` | uvicorn log level |
| `LAYA_API_KEY` | (none) | when set, requires `Authorization: Bearer <key>` |
+| `LAYA_ROOT_PATH` | (empty) | public URL prefix for FastAPI when behind a reverse proxy; the proxy should strip it before forwarding |
| `LAYA_MAX_TOKEN_BUDGET` | `8192` | cap on per-request `max_len` and `head_max_len` overrides |
+For example, set `LAYA_ROOT_PATH=/laya` when publishing the API under `/laya`. The proxy must
+strip that prefix before forwarding to the container; this setting updates FastAPI's generated
+URLs and does not change the internal `/health` or `/v1/systemone` routes.
+
`LAYA_PRELOAD` defaults to `0` here rather than the package default of `1`, because
preloading makes the first boot download all three checkpoints. Set it to `1` for a
long-running deployment so the first request does not pay for the build.
diff --git a/docs/http-api.md b/docs/http-api.md
index a1bb894a..b240fa34 100644
--- a/docs/http-api.md
+++ b/docs/http-api.md
@@ -22,6 +22,7 @@ Everything is environment variables, so one image serves a laptop dev run and a
|---|---|---|
| `LAYA_HOST` | bind address | `0.0.0.0` |
| `LAYA_PORT` | bind port | `8000` |
+| `LAYA_ROOT_PATH` | public URL prefix when served behind a reverse proxy | empty |
| `LAYA_DEVICE` | torch device for every checkpoint | auto |
| `LAYA_PRELOAD` | build the checkpoints at startup, not lazily | `1` |
| `LAYA_MODELS` | comma list to preload (`english,multilingual,typed-decisions`); empty = all | all |
@@ -31,6 +32,11 @@ Everything is environment variables, so one image serves a laptop dev run and a
| `LAYA_LOG_LEVEL` | uvicorn log level | `info` |
| `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` |
+For a deployment published under a prefix such as `/laya`, set `LAYA_ROOT_PATH=/laya`.
+FastAPI uses it when generating OpenAPI and Swagger UI URLs. Configure the reverse proxy to
+strip `/laya` before forwarding requests to Laya; the app's routes remain `/health` and
+`/v1/systemone` internally.
+
For containers, including CUDA and ARM64 images, see [Docker quickstart](docker.md).
## Endpoints
diff --git a/laya/serve.py b/laya/serve.py
index ed08d39a..2b207869 100644
--- a/laya/serve.py
+++ b/laya/serve.py
@@ -27,6 +27,7 @@ env var meaning defaul
``LAYA_MAX_LOADED`` checkpoints kept resident at once. Below what 2
routing can choose, one reloads per switch.
``LAYA_API_KEY`` if set, require ``Authorization: Bearer <it>`` (none)
+``LAYA_ROOT_PATH`` public URL prefix behind a reverse proxy (none)
``LAYA_LOG_LEVEL`` uvicorn log level info
``LAYA_MAX_CONCURRENT`` cap on requests past auth at once; excess 16
gets 503 (see below)
@@ -367,6 +368,7 @@ def create_app(router: Optional[Any] = None):
app = FastAPI(
title="laya-serve",
summary="Laya System-1 decisions over the TypeSafe Jev /v1/systemone protocol",
+ root_path=os.environ.get("LAYA_ROOT_PATH", ""),
lifespan=lifespan,
)
diff --git a/nix/laya-serve.nix b/nix/laya-serve.nix
index a3c80398..2f5a0e28 100644
--- a/nix/laya-serve.nix
+++ b/nix/laya-serve.nix
@@ -44,6 +44,17 @@ in
description = "TCP port to listen on.";
};
+ rootPath = lib.mkOption {
+ type = lib.types.nullOr lib.types.str;
+ default = null;
+ example = "/laya";
+ description = ''
+ Public URL prefix used by FastAPI behind a reverse proxy (sets
+ `LAYA_ROOT_PATH`). The proxy should strip this prefix before forwarding
+ requests. null leaves the root path empty.
+ '';
+ };
+
device = lib.mkOption {
type = lib.types.str;
default = "cuda";
@@ -274,6 +285,8 @@ in
} // lib.optionalAttrs (cfg.threads != null) {
LAYA_THREADS = toString cfg.threads;
OMP_NUM_THREADS = toString cfg.threads;
+ } // lib.optionalAttrs (cfg.rootPath != null) {
+ LAYA_ROOT_PATH = cfg.rootPath;
} // lib.optionalAttrs (cfg.logLevel != null) {
LAYA_LOG_LEVEL = cfg.logLevel;
} // lib.optionalAttrs (cfg.maxConcurrent != null) {
diff --git a/tests/test_packaging.py b/tests/test_packaging.py
index 4633f226..656a2cfa 100644
--- a/tests/test_packaging.py
+++ b/tests/test_packaging.py
@@ -269,6 +269,8 @@ check_true("compose.http/the server reads the same variable",
# unreachable for the documented compose path -- `docker run -e` still works, compose does not.
check_true("compose.http/forwards the resident-checkpoint cap",
'LAYA_MAX_LOADED: "${LAYA_MAX_LOADED:-}"' in http)
+check_true("compose.http/forwards the FastAPI root path",
+ 'LAYA_ROOT_PATH: "${LAYA_ROOT_PATH:-}"' in http)
check_true("compose.http/shares the model cache",
"model-cache:/home/laya/.cache/huggingface" in http)
# The base service is what `docker compose run --rm laya` uses; publishing it a port or
@@ -470,7 +472,7 @@ def option_type(opt):
# Every new knob is opt-in: unset means the unit exports nothing and the runtime's own default
# applies, so a host that ignores them gets today's behaviour byte for byte.
-for opt in ("logLevel", "maxConcurrent", "cudaAmp", "cpuAmp", "mpsAmpMinRows",
+for opt in ("rootPath", "logLevel", "maxConcurrent", "cudaAmp", "cpuAmp", "mpsAmpMinRows",
"maxLoaded", "maxTokenBudget", "revision"):
_t = option_text(opt)
check_true("nix/module declares %s" % opt, _t != "", "option not found")
diff --git a/tests/test_serve.py b/tests/test_serve.py
index dab066ab..4cf327a6 100644
--- a/tests/test_serve.py
+++ b/tests/test_serve.py
@@ -81,6 +81,28 @@ def _budget_client(monkeypatch, api_key=None):
return TestClient(create_app(router=fake)), fake
+def test_root_path_from_environment_updates_openapi_and_keeps_routes(monkeypatch):
+ monkeypatch.setenv("LAYA_ROOT_PATH", "/laya")
+ app = create_app(router=FakeRouter())
+
+ assert app.root_path == "/laya"
+ with TestClient(app) as client:
+ assert client.get("/health").status_code == 200
+ assert client.get("/openapi.json").json()["servers"] == [{"url": "/laya"}]
+
+
[email protected]("root_path", [None, ""])
+def test_root_path_defaults_to_empty(monkeypatch, root_path):
+ if root_path is None:
+ monkeypatch.delenv("LAYA_ROOT_PATH", raising=False)
+ else:
+ monkeypatch.setenv("LAYA_ROOT_PATH", root_path)
+
+ app = create_app(router=FakeRouter())
+
+ assert app.root_path == ""
+
+
REQ = {
"model": "jev-1", # a non-Laya model id -> should be ignored, router auto-routes