Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(serve): support reverse-proxy URL prefixes
Som Samantray1ca5e754c413
diff --git a/README.md b/README.mdindex 4aadb0c4..a132f868 100644--- a/README.md+++ b/README.md@@ -500,6 +500,10 @@ send `Authorization: Bearer <key>`). A client's `model` field is honoured when inames a Laya checkpoint (`english`/`multilingual`/`typed-decisions`), otherwisethe router auto-selects by script/language.+When publishing the server under a reverse-proxy prefix such as `/laya`, set+`LAYA_ROOT_PATH=/laya`. The proxy should strip that prefix before forwarding;+the setting controls FastAPI-generated URLs and leaves the internal routes unchanged.+Three things differ from Jev when you port a client:* **Options per question.** A question's options share the checkpoint's option budget, `head_max_len` (192 tokens on `laya`, 256 on the other two), not Jev's cap of 255 options. In addition, the HTTP server (`laya.serve`) enforces an amplification guard of at most 100 choice options per question (`MAX_CHOICE_OPTIONS = 100`, rejected with 413 before inference). Once options overflow the token budget, around 20 options with a short description each, every option is trimmed to fit, so long or similar labels can reach the model reading the same ([Where Jev leads](#where-jev-leads)). Once they no longer fit the window at all, the library rejects the request with 422. For more candidates, narrow them first with `predict_shortlist` ([Honest limits](#honest-limits)).diff --git a/compose.http.yaml b/compose.http.yamlindex cd7f7f65..3563b6e8 100644--- a/compose.http.yaml+++ b/compose.http.yaml@@ -37,6 +37,8 @@ services:LAYA_HOST: "${LAYA_HOST:-0.0.0.0}"LAYA_PORT: "${LAYA_PORT:-8000}"LAYA_DEVICE: "${LAYA_DEVICE:-cpu}"+ # FastAPI's external URL prefix behind a reverse proxy; forward stripped paths.+ LAYA_ROOT_PATH: "${LAYA_ROOT_PATH:-}"# Autocast dtype the runtime reads when building the model; empty keeps the checkpoint's own# `amp_dtype`. Set LAYA_CUDA_AMP=fp16 to serve fp16 on a GPU host.LAYA_CUDA_AMP: "${LAYA_CUDA_AMP:-}"diff --git a/docs/docker.md b/docs/docker.mdindex 99696d21..03d2c55d 100644--- a/docs/docker.md+++ b/docs/docker.md@@ -241,8 +241,13 @@ These apply to the `laya-serve` service only.| `LAYA_MAX_CONCURRENT` | `16` | requests admitted at once; later ones get `503` (a value that does not parse, or is not positive, falls back to `16`) || `LAYA_LOG_LEVEL` | `info` | uvicorn log level || `LAYA_API_KEY` | (none) | when set, requires `Authorization: Bearer <key>` |+| `LAYA_ROOT_PATH` | (empty) | public URL prefix for FastAPI when behind a reverse proxy; the proxy should strip it before forwarding || `LAYA_MAX_TOKEN_BUDGET` | `8192` | cap on per-request `max_len` and `head_max_len` overrides |+For example, set `LAYA_ROOT_PATH=/laya` when publishing the API under `/laya`. The proxy must+strip that prefix before forwarding to the container; this setting updates FastAPI's generated+URLs and does not change the internal `/health` or `/v1/systemone` routes.+`LAYA_PRELOAD` defaults to `0` here rather than the package default of `1`, becausepreloading makes the first boot download all three checkpoints. Set it to `1` for along-running deployment so the first request does not pay for the build.diff --git a/docs/http-api.md b/docs/http-api.mdindex a1bb894a..b240fa34 100644--- a/docs/http-api.md+++ b/docs/http-api.md@@ -22,6 +22,7 @@ Everything is environment variables, so one image serves a laptop dev run and a|---|---|---|| `LAYA_HOST` | bind address | `0.0.0.0` || `LAYA_PORT` | bind port | `8000` |+| `LAYA_ROOT_PATH` | public URL prefix when served behind a reverse proxy | empty || `LAYA_DEVICE` | torch device for every checkpoint | auto || `LAYA_PRELOAD` | build the checkpoints at startup, not lazily | `1` || `LAYA_MODELS` | comma list to preload (`english,multilingual,typed-decisions`); empty = all | all |@@ -31,6 +32,11 @@ Everything is environment variables, so one image serves a laptop dev run and a| `LAYA_LOG_LEVEL` | uvicorn log level | `info` || `LAYA_MAX_CONCURRENT` | requests admitted past auth at once; excess gets `503` | `16` |+For a deployment published under a prefix such as `/laya`, set `LAYA_ROOT_PATH=/laya`.+FastAPI uses it when generating OpenAPI and Swagger UI URLs. Configure the reverse proxy to+strip `/laya` before forwarding requests to Laya; the app's routes remain `/health` and+`/v1/systemone` internally.+For containers, including CUDA and ARM64 images, see [Docker quickstart](docker.md).## Endpointsdiff --git a/laya/serve.py b/laya/serve.pyindex ed08d39a..2b207869 100644--- a/laya/serve.py+++ b/laya/serve.py@@ -27,6 +27,7 @@ env var meaning defaul``LAYA_MAX_LOADED`` checkpoints kept resident at once. Below what 2routing can choose, one reloads per switch.``LAYA_API_KEY`` if set, require ``Authorization: Bearer <it>`` (none)+``LAYA_ROOT_PATH`` public URL prefix behind a reverse proxy (none)``LAYA_LOG_LEVEL`` uvicorn log level info``LAYA_MAX_CONCURRENT`` cap on requests past auth at once; excess 16gets 503 (see below)@@ -367,6 +368,7 @@ def create_app(router: Optional[Any] = None):app = FastAPI(title="laya-serve",summary="Laya System-1 decisions over the TypeSafe Jev /v1/systemone protocol",+ root_path=os.environ.get("LAYA_ROOT_PATH", ""),lifespan=lifespan,)diff --git a/nix/laya-serve.nix b/nix/laya-serve.nixindex a3c80398..2f5a0e28 100644--- a/nix/laya-serve.nix+++ b/nix/laya-serve.nix@@ -44,6 +44,17 @@ indescription = "TCP port to listen on.";};+ rootPath = lib.mkOption {+ type = lib.types.nullOr lib.types.str;+ default = null;+ example = "/laya";+ description = ''+ Public URL prefix used by FastAPI behind a reverse proxy (sets+ `LAYA_ROOT_PATH`). The proxy should strip this prefix before forwarding+ requests. null leaves the root path empty.+ '';+ };+device = lib.mkOption {type = lib.types.str;default = "cuda";@@ -274,6 +285,8 @@ in} // lib.optionalAttrs (cfg.threads != null) {LAYA_THREADS = toString cfg.threads;OMP_NUM_THREADS = toString cfg.threads;+ } // lib.optionalAttrs (cfg.rootPath != null) {+ LAYA_ROOT_PATH = cfg.rootPath;} // lib.optionalAttrs (cfg.logLevel != null) {LAYA_LOG_LEVEL = cfg.logLevel;} // lib.optionalAttrs (cfg.maxConcurrent != null) {diff --git a/tests/test_packaging.py b/tests/test_packaging.pyindex 4633f226..656a2cfa 100644--- a/tests/test_packaging.py+++ b/tests/test_packaging.py@@ -269,6 +269,8 @@ check_true("compose.http/the server reads the same variable",# unreachable for the documented compose path -- `docker run -e` still works, compose does not.check_true("compose.http/forwards the resident-checkpoint cap",'LAYA_MAX_LOADED: "${LAYA_MAX_LOADED:-}"' in http)+check_true("compose.http/forwards the FastAPI root path",+ 'LAYA_ROOT_PATH: "${LAYA_ROOT_PATH:-}"' in http)check_true("compose.http/shares the model cache","model-cache:/home/laya/.cache/huggingface" in http)# The base service is what `docker compose run --rm laya` uses; publishing it a port or@@ -470,7 +472,7 @@ def option_type(opt):# Every new knob is opt-in: unset means the unit exports nothing and the runtime's own default# applies, so a host that ignores them gets today's behaviour byte for byte.-for opt in ("logLevel", "maxConcurrent", "cudaAmp", "cpuAmp", "mpsAmpMinRows",+for opt in ("rootPath", "logLevel", "maxConcurrent", "cudaAmp", "cpuAmp", "mpsAmpMinRows","maxLoaded", "maxTokenBudget", "revision"):_t = option_text(opt)check_true("nix/module declares %s" % opt, _t != "", "option not found")diff --git a/tests/test_serve.py b/tests/test_serve.pyindex dab066ab..4cf327a6 100644--- a/tests/test_serve.py+++ b/tests/test_serve.py@@ -81,6 +81,28 @@ def _budget_client(monkeypatch, api_key=None):return TestClient(create_app(router=fake)), fake+def test_root_path_from_environment_updates_openapi_and_keeps_routes(monkeypatch):+ monkeypatch.setenv("LAYA_ROOT_PATH", "/laya")+ app = create_app(router=FakeRouter())++ assert app.root_path == "/laya"+ with TestClient(app) as client:+ assert client.get("/health").status_code == 200+ assert client.get("/openapi.json").json()["servers"] == [{"url": "/laya"}]++[email protected]("root_path", [None, ""])+def test_root_path_defaults_to_empty(monkeypatch, root_path):+ if root_path is None:+ monkeypatch.delenv("LAYA_ROOT_PATH", raising=False)+ else:+ monkeypatch.setenv("LAYA_ROOT_PATH", root_path)++ app = create_app(router=FakeRouter())++ assert app.root_path == ""++REQ = {"model": "jev-1", # a non-Laya model id -> should be ignored, router auto-routes