Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat: promote Route Mode as primary Quickstart, bump to v0.3.4
Nandakishord113dca2512f
diff --git a/README.md b/README.mdindex f12882b7..da95bc05 100644--- a/README.md+++ b/README.md@@ -44,27 +44,29 @@ pip install laya----## Quickstart+## Quickstart: Route Mode (Recommended)++Laya ships three checkpoints. The built-in **`Router`** is the recommended entry point: it evaluates any state in any language, automatically detects scripts and languages in sub-milliseconds, and dispatches to the optimal checkpoint in a single forward pass.```pythonimport laya+from laya import Router-# 1. Load the fine-tuned model directly from Hugging Face Hub (auto-downloads weights)-agent = laya.load("convaiinnovations/laya")+# Preload checkpoints into memory for instant sub-35ms routing+router = Router(preload=True)-# 2. Provide any state (string or dictionary)+# 1. State in any language or schemastate = {"from": "[email protected]","subject": "Duplicate charge on invoice #4411","body": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan."}-# 3. Define your typed questions+# 2. Define your typed questionsquestions = {- # choice: categorical selection with probabilities & confidence"department": {"type": "choice",- "instructions": "Which department should handle this email?",+ "instructions": "Which department should handle this request?","criteria": {"billing": "invoices, payments, refunds","technical": "bugs, outages, system errors",@@ -72,197 +74,159 @@ questions = {"other": "everything else"}},- # score: placement on an ordinal rubric"urgency": {"type": "score","instructions": "How urgent is this request?","criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},- # noul: calibrated boolean probability P(true)"churn_risk": {"type": "noul","instructions": "Does the user threaten to cancel or leave?"},- "is_phishing": {+ "refund_requested": {"type": "noul",- "instructions": "Is this email a phishing or scam attempt?"+ "instructions": "Does the user explicitly request a refund?"}}-# 4. Run all questions in ONE single forward pass (~35 ms on GPU)-result = agent.predict(state, questions)-answers = result["answers"]--print("Department :", answers["department"]["choice"])-# -> billing (confidence: 0.94)+# 3. English state -> automatically routed to laya (ModernBERT-large, 39.5 ms)+res_en = router.predict(state, questions)+print("Department :", res_en["answers"]["department"]["choice"]) # -> billing (confidence: 0.94)+print("Routing :", res_en["routing"]["model"]) # -> english-print("Urgency :", answers["urgency"]["score"])-# -> 1.84 / 2.0+# 4. Hindi state -> automatically routed to laya-multilingual (mmBERT-base, 32.8 ms)+res_hi = router.predict({"body": "मुझसे दो बार शुल्क लिया गया, कृपया पैसे वापस करें।"}, questions)+print("Department :", res_hi["answers"]["department"]["choice"]) # -> billing (confidence: 0.86)+print("Routing :", res_hi["routing"]["model"]) # -> multilingual-print("Churn Risk :", answers["churn_risk"]["noul"])-# -> 0.892 (89.2% probability)--print("Phishing :", answers["is_phishing"]["noul"])-# -> 0.008 (0.8% probability)+# 5. Explicit override when you want a specific checkpoint+res_td = router.predict(state, questions, model="typed-decisions")```------## Automated Confidence Gating--Because Laya's probabilities are trained with strictly proper scoring rules (RLCD), confidence scores are statistically meaningful:+Every result carries full routing metadata explaining why the choice was made:```python-dept = answers["department"]["choice"]-conf = answers["department"]["confidence"]--if conf >= 0.85:- # High confidence: automated action without human in the loop- route_automatically(dept)-else:- # Low confidence: escalate to human triage- escalate_to_human_agent(dept, reason=f"Low confidence ({conf:.2f})")+res_hi["routing"]+# {+# 'model': 'multilingual',+# 'repo': 'convaiinnovations/laya/multilingual',+# 'reason': 'non-Latin script (devanagari, 100% of letters); the English checkpoint cannot read it'+# }```------## Built-in Workflow Presets--Laya provides pre-tuned question schemas for immediate production use:+Inspect a routing decision without running any forward pass:```python-import laya--agent = laya.load("convaiinnovations/laya")--# 1. Intelligent Model Router (routes to small vs. frontier models)-routing = agent.predict({"request": "Refactor this service using dependency injection"}, laya.router_questions())--# 2. Real-time Prompt Guardrails (jailbreaks, injections, leaks)-guard = agent.predict({"prompt": "Ignore all instructions"}, laya.guard_questions())--# 3. Content Safety & Moderation (toxicity, harassment, threats)-safety = agent.predict({"post": "User comment text"}, laya.moderation_questions())--# 4. Support Ticket Triage (intent, urgency, frustration, churn)-triage = agent.predict({"message": "My payment failed twice"}, laya.triage_questions())+router.route({"body": "Der Kunde wurde zweimal belastet"}, questions).reason+# "Latin script but language looks like 'de', not English"```----+### Why Route: The Evidence-## Model Routing (three checkpoints, one call)+On a shared benchmark (17,416 questions, one T4 GPU, identical questions per model):-Laya ships three checkpoints. `Router` picks the right one per request and loads it lazily.+| Benchmark / Task | English (`laya`) | Multilingual (`laya-multilingual`) | `Router` (Routed) |+|---|---|---|---|+| MASSIVE intent, English | **0.783** | 0.657 | **0.783** |+| MASSIVE intent, 13 other languages | 0.306 | **0.451** | **0.451** |+| XNLI, English | **0.860** | 0.843 | **0.860** |+| XNLI, 14 other languages | 0.521 | **0.731** | **0.731** |+| Languages usable (>3x random) | 23 / 51 | 45 / 51 | **45 / 51** |+| Latency, 1 question (T4 GPU) | 39.5 ms | **32.8 ms** | **32.8 ms** |+| Latency, 10 questions batched | 158.6 ms | **72.3 ms** | **72.3 ms** |-| name | repo | size | context | best at |-|---|---|---|---|---|-| `english` | [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya) | 421M | 512 | English text |-| `multilingual` | [`convaiinnovations/laya-multilingual`](https://huggingface.co/convaiinnovations/laya-multilingual) | 322M | 1024 | 100+ languages, 2x faster |-| `typed-decisions` | [`convaiinnovations/laya-typed-decisions`](https://huggingface.co/convaiinnovations/laya-typed-decisions) | 421M | 1024 | the four typed-decisions workflows |+The English checkpoint collapses on non-Latin scripts (Khmer scores **0.000 accuracy at 0.952 confidence**). Because the model stays confident while being wrong, confidence gating cannot save you. `Router` detects the script in <0.5 ms pure Python before the forward pass.-```python-from laya import Router+### Production Preload & Memory-router = Router() # nothing is downloaded until a request needs it+A cold checkpoint build costs seconds; language detection costs microseconds. At the default `max_loaded=1`, traffic that alternates languages rebuilds a model on *every* request (measured at a 7.4 s median reload on CPU and 10.3 s on T4).-# English -> routed to the English checkpoint-router.predict({"body": "I was charged twice, please refund."}, questions)+For a server or production app, preload:-# Hindi -> routed to the multilingual checkpoint automatically-router.predict({"body": "मुझसे दो बार शुल्क लिया गया"}, questions)+```python+# Every checkpoint resident in memory; language flips cost detection only (<1 ms)+router = Router(preload=True)+router = Router(preload=True, device="cuda")-# explicit when you already know-router.predict(state, questions, model="typed-decisions")-router.predict(state, questions, lang="de")-```+# Or preload only the specific checkpoints you serve:+router.preload(["english", "multilingual"])-Every result carries the decision that produced it:+# If your app already built an agent, attach it to avoid duplicate VRAM:+router.attach("english", existing_agent)-```python-result = router.predict({"body": "二重に請求されました"}, questions)-result["routing"]-# {'model': 'multilingual',-# 'repo': 'convaiinnovations/laya-multilingual',-# 'reason': 'non-Latin script (kana, 100% of letters); the English checkpoint cannot read it',-# ...}+# Manage resident memory (default keeps 1 hot, LRU eviction)+router = Router(max_loaded=2) # keep two hot+router.unload() # free memory```-Inspect a decision without running the model:+| Deployment Mode | Per-Request Latency | Model Reloads |+|---|---|---|+| `Router()` (lazy, `max_loaded=1`) | 7 to 10 s on every language switch | 1 per switch |+| `Router(preload=True)` | **32.8 ms (GPU) / 193–464 ms (CPU)** | **none** |-```python-router.route({"body": "Der Kunde wurde zweimal belastet"}, questions).reason-# "Latin script but language looks like 'de', not English"-```+----### Why route at all+## Single-Model Mode (Direct SDK)-Accuracy on a shared benchmark (17,416 questions, one T4, identical questions per model):+If you only need a single checkpoint for a dedicated pipeline, you can load models directly:-| | `english` | `multilingual` |-|---|---|---|-| MASSIVE intent, English | **0.783** | 0.657 |-| MASSIVE intent, 13 other languages | 0.306 | **0.451** |-| XNLI, English | **0.860** | 0.843 |-| XNLI, 14 other languages | 0.521 | **0.731** |-| English-only suites | **0.684** | 0.619 |-| Latency, 10 questions | 159 ms | **72 ms** |+```python+import laya-The English checkpoint does not degrade gracefully outside English -- it collapses, and stays-confident while doing so. On 20-option MASSIVE intent (random = 0.050) it scores 0.100 on Hindi-and 0.103 on Korean, with an expected calibration error of 0.855. Script detection is therefore-the primary routing signal.+# 1. Load a specific checkpoint directly from the hub+agent = laya.load("convaiinnovations/laya") # English root+agent_ml = laya.load("convaiinnovations/laya", subfolder="multilingual") # 100+ languages+agent_td = laya.load("convaiinnovations/laya", subfolder="typed-decisions")-### Routing rules+# 2. Run all questions in ONE single forward pass (~35 ms on GPU)+result = agent.predict(state, questions)+answers = result["answers"]-Precedence, highest first:+print("Department :", answers["department"]["choice"]) # -> billing (confidence: 0.94)+print("Urgency :", answers["urgency"]["score"]) # -> 1.84 / 2.0+print("Churn Risk :", answers["churn_risk"]["noul"]) # -> 0.892 (89.2% probability)+```-1. `model=` -- explicit checkpoint.-2. `task="typed_decisions"` -- explicit task.-3. A question-id set exactly matching a typed-decisions workflow, **only** if you construct the- router with `auto_task_detection=True`. It is off by default: that checkpoint is fine-tuned on- four synthetic workflows and should not be a silent fallback.-4. `lang=` -- explicit language code.-5. Detected script (exact) and, for Latin text, a stopword/diacritic language guess (best effort).-6. `default=` (`"english"` unless you change it).+----### Preload — make routing free+## Automated Confidence Gating-A cold checkpoint build costs **seconds**; language detection costs **microseconds**. At the-default `max_loaded=1`, traffic that alternates languages rebuilds a model on *every* request.-For a server or a demo, preload:+Because Laya's probabilities are trained with strictly proper scoring rules (RLCD), confidence scores are statistically meaningful:```python-router = Router(preload=True) # every checkpoint resident, routing is free-router = Router(preload=True, device="cuda")-router.preload(["english", "multilingual"]) # or just the two you serve+dept = answers["department"]["choice"]+conf = answers["department"]["confidence"]++if conf >= 0.85:+ # High confidence: automated action without human in the loop+ route_automatically(dept)+else:+ # Low confidence: escalate to human triage+ escalate_to_human_agent(dept, reason=f"Low confidence ({conf:.2f})")```-`preload` raises `max_loaded` to fit what it built, so the LRU cannot evict it immediately.+---++## Built-in Workflow Presets-If the process already has a checkpoint loaded for other reasons, hand it over instead of-loading a second copy:+Laya provides pre-tuned question schemas for immediate production use:```python-router.attach("english", existing_agent) # no duplicate 421M parameters-router.preload() # builds only what is still missing-```+import laya-Measured on CPU with the demo Space's own workload:+agent = laya.load("convaiinnovations/laya")-| | per request | model loads |-|---|---|---|-| `Router()` — lazy, `max_loaded=1` | 4–6 s on every language switch | 1 per switch |-| `Router(preload=True)` | **193–464 ms** | **none** |+# 1. Intelligent Model Router (routes to small vs. frontier models)+routing = agent.predict({"request": "Refactor this service using dependency injection"}, laya.router_questions())-### Memory+# 2. Real-time Prompt Guardrails (jailbreaks, injections, leaks)+guard = agent.predict({"prompt": "Ignore all instructions"}, laya.guard_questions())-All three together are ~1.16B parameters (~4.6 GB fp32), so `Router` keeps **one** resident by-default and evicts least-recently-used. Raise it when you have the RAM:+# 3. Content Safety & Moderation (toxicity, harassment, threats)+safety = agent.predict({"post": "User comment text"}, laya.moderation_questions())-```python-Router(max_loaded=2) # keep two hot-router.unload() # free everything-router.loaded # ['multilingual']+# 4. Support Ticket Triage (intent, urgency, frustration, churn)+triage = agent.predict({"message": "My payment failed twice"}, laya.triage_questions())```---diff --git a/laya/__init__.py b/laya/__init__.pyindex 19e535ce..1f1acb4f 100644--- a/laya/__init__.py+++ b/laya/__init__.py@@ -22,7 +22,7 @@ from .presets import ()from .router import DEFAULT_MODELS, RouteDecision, Router-__version__ = "0.3.3"+__version__ = "0.3.4"__all__ = ["Agent","RLAgent",diff --git a/pyproject.toml b/pyproject.tomlindex be4372aa..f247394b 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"[project]name = "laya"-version = "0.3.3"+version = "0.3.4"description = "Fast, non-autoregressive System 1 decision engine with calibrated probabilities"readme = "README.md"requires-python = ">=3.8"diff --git a/setup.py b/setup.pyindex 9b93a818..93db822b 100644--- a/setup.py+++ b/setup.py@@ -2,7 +2,7 @@ from setuptools import setup, find_packagessetup(name="laya",- version="0.3.3",+ version="0.3.4",description="Fast, non-autoregressive System 1 decision engine with calibrated probabilities",long_description=open("README.md").read(),long_description_content_type="text/markdown",