Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
laya: the Laya decision engine (Convai Innovations, ModernBERT + decision head)
JustVuggfbf985527b6f
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex d445d75b..82dac24f 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -278,7 +278,7 @@ jobs:run: |cd crc=0- ENGINES="colibri glm53 inkling kimi_k3 olmoe qwen36 qwen38 deepseek_v41 mimo qwenimage"+ ENGINES="colibri glm53 inkling kimi_k3 olmoe qwen36 qwen38 deepseek_v41 mimo qwenimage laya"if [ "${{ matrix.v4 }}" = "1" ]; then ENGINES="$ENGINES deepseek-v4"; fifor t in $ENGINES; doecho "::group::$t"@@ -1088,6 +1088,19 @@ jobs:python3 tools/make_edge_tiny_tokenizer.py --vocab-size 64 qwen38_tiny_fp8QWEN38_TINY=qwen38_tiny_fp8 python3 -m unittest -v tests.test_qwen38_brio+ laya-tiny-check:+ name: Laya tiny oracle (vs the laya package) and POST /v1/systemone end to end+ runs-on: ubuntu-latest+ steps:+ - uses: actions/checkout@v4+ - uses: actions/setup-python@v5+ with:+ python-version: '3.12'+ - name: Install the fixture generator's one dependency+ run: pip install numpy+ - name: Generate the fixture, build laya, compare with the committed reference+ run: make -C c laya-tiny-check+qwenimage-tiny-check:name: Qwen-Image tiny oracle (text encoder, DiT, scheduler, VAE vs diffusers)runs-on: ubuntu-latestdiff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex 2fcc7ead..b2cf5e43 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -67,7 +67,7 @@ jobs:- name: Build enginesrun: |cd c- for t in colibri glm53 inkling kimi_k3 olmoe qwen36 qwen38 deepseek_v41 mimo qwenimage; do+ for t in colibri glm53 inkling kimi_k3 olmoe qwen36 qwen38 deepseek_v41 mimo qwenimage laya; domake $t ${{ matrix.make_args }}donels -lh colibri${{ matrix.ext }} glm53${{ matrix.ext }} inkling${{ matrix.ext }} kimi_k3${{ matrix.ext }} olmoe${{ matrix.ext }} qwen36${{ matrix.ext }} qwen38${{ matrix.ext }} deepseek_v41${{ matrix.ext }} mimo${{ matrix.ext }}@@ -134,6 +134,7 @@ jobs:cp c/qwenimage${{ matrix.ext }} dist/qwenimage${{ matrix.ext }}cp c/deepseek_v41${{ matrix.ext }} dist/deepseek_v41${{ matrix.ext }}cp c/mimo${{ matrix.ext }} dist/mimo${{ matrix.ext }}+ cp c/laya${{ matrix.ext }} dist/laya${{ matrix.ext }}# deepseek_v4 -- underscore, not the hyphen of the make target: engine_for()# looks for "deepseek_v4" next to coli. Guarded, not unconditional, because# the macos-arm64 archive has no such engine to ship (#858).diff --git a/.gitignore b/.gitignoreindex 03d9c9be..3d7ee1d0 100644--- a/.gitignore+++ b/.gitignore@@ -21,6 +21,8 @@ c/kimi_k3.exec/olmoe.exec/qwen38c/qwen38.exe+c/laya+c/laya.exe# deepseek_v4 and the per-unit objects its amalgamated Makefile leaves behind.# The same omission as #858, one file over: every engine is listed here except# the one added last, so `make deepseek-v4` left 26 untracked .o files indiff --git a/README.it.md b/README.it.mdindex b22891d1..2eac5daf 100644--- a/README.it.md+++ b/README.it.md@@ -525,6 +525,7 @@ altre nove famiglie di modelli linguistici, e un motore genera immagini. Ognuna| **Qwen3-Coder-30B-A3B** (Alibaba) | 30B / 3B | [`Justvugg/Qwen3-Coder-30B-A3B-colibri-int4`](https://huggingface.co/Justvugg/Qwen3-Coder-30B-A3B-colibri-int4) (19 GB, int4-gs64), convertito da [`Qwen/Qwen3-Coder-30B-A3B-Instruct`](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct). MoE Qwen3 di sola attenzione sul motore Qwen3.6, 128 expert top-8, una sua forma XML per le chiamate ai tool e nessun thinking; in teacher forcing il container int4 sceglie il token top-1 della release bf16 nel 96,9% delle posizioni | `make -C c qwen36` | [qwen36.md](docs/qwen36.md#qwen3-coder-30b-a3b) || **OLMoE** (AI2) | 7B / 1B | convertito con `c/tools/convert_olmoe_merged.py`: container **int8**, ~7 GB | `make -C c olmoe` | - || **Qwen-Image-2.1** (Alibaba) | modello di immagini | [`Qwen/Qwen-Image-2.1`](https://huggingface.co/Qwen/Qwen-Image-2.1) (~33 GB), checkpoint diffusers ufficiale, **nessuna conversione**: il text encoder e il diffusion transformer vengono quantizzati a int8 durante il caricamento. Immagini dentro `coli chat`, `POST /v1/images/generations` su `coli serve`. Qwen Research License: solo uso non commerciale | `make -C c qwenimage` | [qwen-image.md](docs/qwen-image.md) |+| **Laya** (Convai Innovations) | modello di decisione, 421M | [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya) (842 MB), checkpoint ufficiale, **nessuna conversione**: un encoder ModernBERT con una testa di decisione che risponde a domande tipizzate (choice, score, noul) con probabilità calibrate invece di generare. Servito su `POST /v1/systemone` da `coli serve`. Apache-2.0 | `make -C c laya` | [laya.md](docs/laya.md) |Qwen3.6 offre tre container pre-convertiti: **int4-gs64** (consigliato: cosenomisurato rispetto all'ancora int8 da 0,98777 a 0,99313 e KL da 0,109 a 0,080diff --git a/README.ja.md b/README.ja.mdindex 3dfbd590..3d07e9f0 100644--- a/README.ja.md+++ b/README.ja.md@@ -485,6 +485,7 @@ GLM-5.2 がリファレンスモデルですが、同じストリーミング手| **Qwen3-Coder-30B-A3B**(Alibaba) | 30B / 3B | [`Justvugg/Qwen3-Coder-30B-A3B-colibri-int4`](https://huggingface.co/Justvugg/Qwen3-Coder-30B-A3B-colibri-int4)(19 GB、int4-gs64)、[`Qwen/Qwen3-Coder-30B-A3B-Instruct`](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct) から変換。Qwen3.6 エンジンで動く全層アテンションの Qwen3 MoE で、128 エキスパートの top-8、独自の XML 形式のツール呼び出しを持ち、思考モードはなし。teacher forcing で、int4 コンテナは 96.9% の位置で bf16 リリースと同じ top-1 トークンを選ぶ | `make -C c qwen36` | [qwen36.md](docs/qwen36.md#qwen3-coder-30b-a3b) || **OLMoE**(AI2) | 7B / 1B | `c/tools/convert_olmoe_merged.py` で変換 — **int8** コンテナ、約 7 GB | `make -C c olmoe` | — || **Qwen-Image-2.1**(Alibaba) | 画像モデル | [`Qwen/Qwen-Image-2.1`](https://huggingface.co/Qwen/Qwen-Image-2.1)(約 33 GB)、公式 diffusers チェックポイント、**変換不要**: テキストエンコーダと拡散トランスフォーマーはロード時に int8 に量子化。`coli chat` では画像をインライン表示し、`coli serve` では `POST /v1/images/generations` で提供。Qwen Research License: 非商用利用のみ | `make -C c qwenimage` | [qwen-image.md](docs/qwen-image.md) |+| **Laya**(Convai Innovations) | 判定モデル、421M | [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya)(842 MB)、公式チェックポイント、**変換不要**: ModernBERT エンコーダと判定ヘッドで、型付きの質問(choice、score、noul)に生成ではなく較正済みの確率で答える。`coli serve` の `POST /v1/systemone` で提供。Apache-2.0 | `make -C c laya` | [laya.md](docs/laya.md) |Qwen3.6 には変換済みコンテナが 3 つあります: **int4-gs64**(推奨 — int8 のアンカーに対するコサイン類似度は行単位と比べて 0.98777 → 0.99313、KL は 0.109 → 0.080 と計測されており、量子化誤差が約 44% 少ない)、diff --git a/README.md b/README.mdindex 828771fd..1a379091 100644--- a/README.md+++ b/README.md@@ -506,6 +506,7 @@ or from `model_index.json` for the image model):| **Qwen3-Coder-30B-A3B** (Alibaba) | 30B / 3B | [`Justvugg/Qwen3-Coder-30B-A3B-colibri-int4`](https://huggingface.co/Justvugg/Qwen3-Coder-30B-A3B-colibri-int4) (19 GB, int4-gs64), converted from [`Qwen/Qwen3-Coder-30B-A3B-Instruct`](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct). All-attention Qwen3 MoE on the Qwen3.6 engine, 128 experts top-8, its own XML tool-call form and no thinking; teacher forced, the int4 container picks the bf16 release's top-1 token at 96.9% of positions | `make -C c qwen36` | [qwen36.md](docs/qwen36.md#qwen3-coder-30b-a3b) || **OLMoE** (AI2) | 7B / 1B | converted with `c/tools/convert_olmoe_merged.py` — **int8** container, ~7 GB | `make -C c olmoe` | — || **Qwen-Image-2.1** (Alibaba) | image model | [`Qwen/Qwen-Image-2.1`](https://huggingface.co/Qwen/Qwen-Image-2.1) (~33 GB), official diffusers checkpoint, **no conversion**: the text encoder and the diffusion transformer are quantized to int8 while loading. Pictures inline in `coli chat`, `POST /v1/images/generations` on `coli serve`. Qwen Research License: non-commercial use only | `make -C c qwenimage` | [qwen-image.md](docs/qwen-image.md) |+| **Laya** (Convai Innovations) | decision model, 421M | [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya) (842 MB), official checkpoint, **no conversion**: a ModernBERT encoder with a decision head that answers typed questions (choice, score, noul) with calibrated probabilities instead of generating. Served on `POST /v1/systemone` by `coli serve`. Apache-2.0 | `make -C c laya` | [laya.md](docs/laya.md) |Qwen3.6 ships three pre-converted containers: **int4-gs64** (recommended — measuredcosine to the int8 anchor 0.98777 → 0.99313 and KL 0.109 → 0.080 against per-row, i.e.diff --git a/README.zh-CN.md b/README.zh-CN.mdindex 5f119952..2f7c4211 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -440,6 +440,7 @@ GLM-5.2 是参考模型,但同样的流式方法还能运行另外九个语言| **Qwen3-Coder-30B-A3B**(Alibaba) | 30B / 3B | [`Justvugg/Qwen3-Coder-30B-A3B-colibri-int4`](https://huggingface.co/Justvugg/Qwen3-Coder-30B-A3B-colibri-int4)(19 GB,int4-gs64),由 [`Qwen/Qwen3-Coder-30B-A3B-Instruct`](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct) 转换而来。运行在 Qwen3.6 引擎上的全注意力 Qwen3 MoE,128 个专家 top-8,有自己的 XML 工具调用格式,不带思考;在 teacher forcing 下,int4 容器在 96.9% 的位置上选出与 bf16 发布版相同的 top-1 token | `make -C c qwen36` | [qwen36.md](docs/qwen36.md#qwen3-coder-30b-a3b) || **OLMoE**(AI2) | 7B / 1B | 用 `c/tools/convert_olmoe_merged.py` 转换,**int8** 容器,约 7 GB | `make -C c olmoe` | 无 || **Qwen-Image-2.1**(Alibaba) | 图像模型 | [`Qwen/Qwen-Image-2.1`](https://huggingface.co/Qwen/Qwen-Image-2.1)(约 33 GB),官方 diffusers checkpoint,**无需转换**:文本编码器与扩散 transformer 在加载时量化为 int8。`coli chat` 中直接显示图片,`coli serve` 提供 `POST /v1/images/generations`。Qwen Research License:仅限非商业用途 | `make -C c qwenimage` | [qwen-image.md](docs/qwen-image.md) |+| **Laya**(Convai Innovations) | 决策模型,421M | [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya)(842 MB),官方 checkpoint,**无需转换**:ModernBERT 编码器加决策头,对类型化问题(choice、score、noul)给出校准后的概率,而不是生成文本。由 `coli serve` 在 `POST /v1/systemone` 上提供。Apache-2.0 | `make -C c laya` | [laya.md](docs/laya.md) |Qwen3.6 提供三个预转换容器:**int4-gs64**(推荐:与 per-row 相比,对 int8 基准的余弦相似度实测从 0.98777 提升到 0.99313,KL 从 0.109 降到 0.080,即量化误差减少约 44%)、作为 A/B 基线的diff --git a/README.zh-TW.md b/README.zh-TW.mdindex 8b70e84e..c8dfc62f 100644--- a/README.zh-TW.md+++ b/README.zh-TW.md@@ -441,6 +441,7 @@ GLM-5.2 是參考模型,但同樣的串流方法還能執行另外九個語言| **Qwen3-Coder-30B-A3B**(Alibaba) | 30B / 3B | [`Justvugg/Qwen3-Coder-30B-A3B-colibri-int4`](https://huggingface.co/Justvugg/Qwen3-Coder-30B-A3B-colibri-int4)(19 GB,int4-gs64),由 [`Qwen/Qwen3-Coder-30B-A3B-Instruct`](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct) 轉換而來。執行在 Qwen3.6 引擎上的全注意力 Qwen3 MoE,128 個專家 top-8,有自己的 XML 工具呼叫格式,不帶思考;在 teacher forcing 下,int4 容器在 96.9% 的位置上選出與 bf16 發布版相同的 top-1 token | `make -C c qwen36` | [qwen36.md](docs/qwen36.md#qwen3-coder-30b-a3b) || **OLMoE**(AI2) | 7B / 1B | 以 `c/tools/convert_olmoe_merged.py` 轉換,**int8** 容器,約 7 GB | `make -C c olmoe` | 無 || **Qwen-Image-2.1**(Alibaba) | 圖像模型 | [`Qwen/Qwen-Image-2.1`](https://huggingface.co/Qwen/Qwen-Image-2.1)(約 33 GB),官方 diffusers checkpoint,**無需轉換**:文字編碼器與擴散 transformer 在載入時量化為 int8。`coli chat` 中直接顯示圖片,`coli serve` 提供 `POST /v1/images/generations`。Qwen Research License:僅限非商業用途 | `make -C c qwenimage` | [qwen-image.md](docs/qwen-image.md) |+| **Laya**(Convai Innovations) | 決策模型,421M | [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya)(842 MB),官方 checkpoint,**無需轉換**:ModernBERT 編碼器加決策頭,對型別化問題(choice、score、noul)給出校準後的機率,而不是生成文字。由 `coli serve` 在 `POST /v1/systemone` 上提供。Apache-2.0 | `make -C c laya` | [laya.md](docs/laya.md) |Qwen3.6 提供三個預先轉換的容器:**int4-gs64**(推薦:與 per-row 相比,對 int8 基準的餘弦相似度實測從 0.98777 提升到 0.99313,KL 從 0.109 降到 0.080,即量化誤差減少約 44%)、作為 A/B 基準的diff --git a/c/Makefile b/c/Makefileindex 2a91104c..74e3756d 100644--- a/c/Makefile+++ b/c/Makefile@@ -1263,6 +1263,28 @@ endifqwenimage$(EXE): qwenimage.c $(VK_OBJ) $(VK_SPV) .build-config$(CC) $(CFLAGS) qwenimage.c $(VK_OBJ) -o qwenimage$(EXE) $(LDFLAGS)+# Laya (Convai Innovations), a decision model: ModernBERT encoder + decision+# head, served on POST /v1/systemone through the DECIDE command (docs/laya.md).+# One file; qi_gemm.h is its matrix kernel, decide_serve.h its serve loop.+ifneq ($(EXE),)+.PHONY: laya+laya: laya$(EXE)+endif+laya$(EXE): laya.c .build-config+ $(CC) $(CFLAGS) laya.c -o laya$(EXE) $(LDFLAGS)++# Tiny Laya checkpoint + the laya package's own answers on it. The generator is+# numpy and deterministic; laya_tiny/ref.json is committed and comes from the+# `laya` package (tools/make_laya_ref.py, maintainer-run, fixture SHA-256+# pinned), so CI needs neither torch nor network. The second module is the same+# fixture through `coli serve`'s gateway: POST /v1/systemone end to end.+.PHONY: laya-tiny-generate laya-tiny-check+laya-tiny-generate:+ $(PYTHON) tools/make_laya_tiny.py --output ./laya_tiny++laya-tiny-check: laya-tiny-generate laya$(EXE)+ LAYA_TINY_REQUIRED=1 $(PYTHON) -m unittest -v tests.test_laya_tiny tests.test_decision_serve+# In make check: every format and ragged edges against the naive product, and# the int8 x int8 VNNI path against the exact integer sum.tests/test_qi_gemm$(EXE): tests/test_qi_gemm.c@@ -2645,7 +2667,7 @@ check:$(MAKE) portable$(MAKE) test-install: colibri$(EXE) glm53$(EXE) inkling$(EXE) kimi_k3$(EXE) olmoe$(EXE) qwen36$(EXE) qwen38$(EXE) deepseek_v41$(EXE) mimo$(EXE) qwenimage$(EXE) \+install: colibri$(EXE) glm53$(EXE) inkling$(EXE) kimi_k3$(EXE) olmoe$(EXE) qwen36$(EXE) qwen38$(EXE) deepseek_v41$(EXE) mimo$(EXE) qwenimage$(EXE) laya$(EXE) \$(if $(filter 1,$(COLI_V4_SUPPORTED)),deepseek-v4,)$(INSTALL) -d $(DESTDIR)$(BINDIR)$(INSTALL) -d $(DESTDIR)$(LIBEXECDIR)@@ -2659,6 +2681,7 @@ install: colibri$(EXE) glm53$(EXE) inkling$(EXE) kimi_k3$(EXE) olmoe$(EXE) qwen3$(INSTALL) -m 755 qwen36$(EXE) $(DESTDIR)$(LIBEXECDIR)/qwen36$(EXE)$(INSTALL) -m 755 qwen38$(EXE) $(DESTDIR)$(LIBEXECDIR)/qwen38$(EXE)$(INSTALL) -m 755 qwenimage$(EXE) $(DESTDIR)$(LIBEXECDIR)/qwenimage$(EXE)+ $(INSTALL) -m 755 laya$(EXE) $(DESTDIR)$(LIBEXECDIR)/laya$(EXE)$(INSTALL) -m 755 deepseek_v41$(EXE) $(DESTDIR)$(LIBEXECDIR)/deepseek_v41$(EXE)$(INSTALL) -m 755 mimo$(EXE) $(DESTDIR)$(LIBEXECDIR)/mimo$(EXE)@if [ -f deepseek_v4$(EXE) ]; then \diff --git a/c/family_registry.py b/c/family_registry.pyindex 5a4e83f9..552db884 100644--- a/c/family_registry.py+++ b/c/family_registry.py@@ -1636,6 +1636,52 @@ FAMILIES = (modality="image",tokenizer_file="processor/tokenizer.json",),+ FamilyDescriptor(+ id="laya",+ # A Laya checkpoint has no config.json at its root: resolve_model reads+ # rl_agent_config.json and the encoder's own config.json, and names it+ # laya_<encoder model_type>. The English and typed-decisions checkpoints+ # are ModernBERT-large; laya-multilingual is mmBERT-base, also model_type+ # modernbert, and is told apart by its geometry.+ model_types=("laya_modernbert",),+ display_name="Laya",+ display_scale="421M",+ display_variants=(+ DisplayVariant((("hidden_size", 1024), ("num_hidden_layers", 28)), "Laya", "421M"),+ DisplayVariant((("hidden_size", 768), ("num_hidden_layers", 22)),+ "Laya multilingual", "322M", model_id="laya-multilingual"),+ ),+ engine_artifact="laya",+ engine_aliases=(),+ engine_group="laya",+ internal_arch="laya",+ build_target="laya",+ process_names=("laya",),+ default_model_id="laya",+ cli_adapter="laya",+ gateway_adapter="laya",+ planner_id="laya",+ planner_geometry=None,+ planner_unsupported_reason=(+ "a decision model has no KV cache and no experts: it keeps its weights "+ "resident (about 1.7 GB in f32 for the 421M checkpoint) and reads at "+ "most max_len tokens per question"),+ expert_inventory=lambda _name, _size, _config, _dtype=None: (),+ config_section="root",+ # max_len 512 (the English checkpoint) by default; the multilingual+ # encoder reads up to 8192. No generation, so the output budgets are the+ # placeholders every descriptor carries, and no context variable: the+ # engine's own COLI_LAYA_MAX_LEN overrides the checkpoint's max_len.+ limits=FamilyLimits(512, 8192, 1, 1, 1, 0, ""),+ capabilities=FamilyCapabilities(False, False, False, False, decision=True),+ has_gateway_adapter=True,+ # one-shot `coli run` has nothing to run: a decision needs questions,+ # which come over POST /v1/systemone from `coli serve` / `coli web`+ has_cli_adapter=False,+ supports_accelerator=False,+ modality="decision",+ tokenizer_file="tokenizer/tokenizer.json",+ ),)@@ -1759,9 +1805,44 @@ def tuning_replay_prompt(family, prompt):MODEL_INDEX = "model_index.json"+# A Laya checkpoint (and anything trained with its code) carries this instead of+# a root config.json; its encoder's config.json sits in encoder/.+DECISION_CONFIG = "rl_agent_config.json"+++def _resolve_decision_checkpoint(model):+ """A Laya-style decision checkpoint: rl_agent_config.json at the root, the+ encoder's config.json under encoder/. The family is keyed on the encoder,+ laya_<model_type>, and the encoder config is the family config (its geometry+ names the variant)."""+ def load(path, what):+ try:+ value = json.loads(path.read_text(encoding="utf-8"))+ except OSError as error:+ raise FamilyConfigError(f"cannot read {what}: {model}") from error+ except json.JSONDecodeError as error:+ raise FamilyConfigError(f"invalid {what}: {error}") from error+ if not isinstance(value, dict):+ raise FamilyConfigError(f"{what} is not a JSON object")+ return value+ config = load(model / DECISION_CONFIG, DECISION_CONFIG)+ encoder = load(model / "encoder" / "config.json", "encoder/config.json")+ model_type = "laya_" + _normalize_model_type(encoder.get("model_type"))+ try:+ family = _BY_TYPE[model_type]+ except KeyError as error:+ raise UnknownFamilyError(f"unsupported decision checkpoint: an "+ f"{encoder.get('model_type')} encoder") from error+ if family.modality != "decision":+ raise UnknownFamilyError(f"unsupported decision checkpoint: {model_type}")+ return ResolvedFamily(family, model_type, config, encoder, str(model))++def resolve_model(model_dir):model = Path(model_dir).expanduser().resolve()path = model / "config.json"+ if not path.is_file() and (model / DECISION_CONFIG).is_file():+ return _resolve_decision_checkpoint(model)if not path.is_file() and (model / MODEL_INDEX).is_file():# A diffusers pipeline (Qwen-Image): the root carries model_index.json# and each component keeps its own config.json in its own directory.diff --git a/c/laya.c b/c/laya.cnew file mode 100644index 00000000..2a8af7e9--- /dev/null+++ b/c/laya.c@@ -0,0 +1,1134 @@+/* laya.c -- Laya, a decision model by Convai Innovations+ * (convaiinnovations/laya on Hugging Face, Apache-2.0), in C.+ *+ * Laya does not generate. Given a state and typed questions (choice, score,+ * noul) it scores every option of every question in one forward pass and+ * returns calibrated probabilities. It is a bidirectional encoder,+ * ModernBERT-large in the English checkpoint, plus a decision head trained+ * from scratch: two transformer layers, an option scorer read at one [MASK]+ * marker per option, and an act/escalate head.+ *+ * The specification is the model's own Python: the `laya` package (common.py,+ * agent.py) and the ModernBERT modeling code it builds on. Each step below+ * names the function it follows:+ *+ * sequence common.build_head / build_sequence / render_options+ * encoder ModernBertModel.forward: LayerNorm'd embeddings, 28 pre-norm+ * layers (no norm before layer 0's attention), RoPE per layer+ * type (global theta 160000 on every third layer starting at 0,+ * local theta 10000 elsewhere with a +-64 window), GeGLU MLP,+ * final LayerNorm; no biases+ * head DecisionModel.forward: + type_emb[qtype], two post-attention+ * nn.TransformerEncoderLayer (norm_first, ReLU, d/64 heads),+ * scorer LN -> Linear -> GELU -> Linear at each marker,+ * act head on [h[0], top1, top1-top2, entropy, k/255]+ * calibration Agent._decode_answers: one temperature per (type, option+ * count) bucket, clamped to [0.5, 5] (common.clamp_temperature),+ * softmax over the question's options+ *+ * Weights are read as stored (F16 in the release) and kept in f32. The+ * forward runs every question of a request as one batch of rows; attention+ * stays per sequence, so a question never sees another's tokens.+ *+ * Two ways to run it:+ * SERVE=1 SNAP=<dir> laya the mux serve protocol, DECIDE only+ * (decide_serve.h; what the gateway runs)+ * laya --model DIR --records F a JSON array of DECIDE records; one+ * DECISION object per line (tests)+ * laya --model DIR --tokenize F a JSON array of strings; their ids+ */+#ifndef _GNU_SOURCE+#define _GNU_SOURCE+#endif+#include <math.h>+#include <stdarg.h>+#include <stdint.h>+#include <stdio.h>+#include <stdlib.h>+#include <string.h>+#include <time.h>+#ifdef _OPENMP+#include <omp.h>+#endif++#include "compat.h"+#include "json.h"+#include "st.h"+#include "tok.h"+#include "qwen38_nfc.h"+#include "qi_gemm.h"+#include "matmul_f32.h"+#include "serve_codec.h"+#include "decide_serve.h"+#include "omp_tune.h"++/* ------------------------------------------------------------------ utils */++static void die(const char *fmt, ...)+{+ va_list args;+ va_start(args, fmt);+ fprintf(stderr, "[laya] ");+ vfprintf(stderr, fmt, args);+ fputc('\n', stderr);+ va_end(args);+ exit(1);+}++static void *xmalloc(size_t size)+{+ void *p = malloc(size ? size : 1);+ if (!p) die("out of memory (%zu bytes)", size);+ return p;+}++static void *xcalloc(size_t count, size_t size)+{+ void *p = calloc(count ? count : 1, size ? size : 1);+ if (!p) die("out of memory (%zu x %zu bytes)", count, size);+ return p;+}++static char *read_text(const char *path)+{+ FILE *f = fopen(path, "rb");+ if (!f) return NULL;+ if (fseek(f, 0, SEEK_END) != 0) { fclose(f); return NULL; }+ long n = ftell(f);+ if (n < 0 || n > (1L << 30)) { fclose(f); return NULL; }+ rewind(f);+ char *b = (char *)xmalloc((size_t)n + 1);+ if (fread(b, 1, (size_t)n, f) != (size_t)n) { fclose(f); free(b); return NULL; }+ b[n] = 0;+ fclose(f);+ return b;+}++static jval *read_json(const char *dir, const char *name)+{+ char path[4096];+ snprintf(path, sizeof(path), "%s/%s", dir, name);+ char *text = read_text(path);+ if (!text) die("cannot read %s", path);+ jval *root = json_parse_checked(text);+ free(text);+ if (!root || root->t != J_OBJ) die("%s is not a JSON object", path);+ return root;+}++static double num_or(jval *o, const char *key, double fallback)+{+ jval *v = json_get(o, key);+ return v && v->t == J_NUM ? v->num : fallback;+}++static int bool_or(jval *o, const char *key, int fallback)+{+ jval *v = json_get(o, key);+ return v && v->t == J_BOOL ? v->boolean : fallback;+}++static double now_ms(void)+{+ struct timespec t;+ clock_gettime(CLOCK_MONOTONIC, &t);+ return t.tv_sec * 1e3 + t.tv_nsec / 1e6;+}++/* Python's floor division, for the one place the reference divides a value+ * that can be negative (build_head's per-option cap). */+static int floordiv(int a, int b)+{+ int q = a / b;+ if ((a % b) && ((a < 0) != (b < 0))) q--;+ return q;+}++/* ------------------------------------------------------------------ model */++typedef struct {+ float *attn_norm_w, *attn_norm_b; /* NULL on layer 0: ModernBERT has no norm there */+ QiMat wqkv, wo, wi, wo2;+ float *bqkv, *bo, *bi, *bo2; /* NULL without attention_bias / mlp_bias */+ float *mlp_norm_w, *mlp_norm_b;+ int global; /* full attention; otherwise the local window */+} EncLayer;++typedef struct {+ float *n1w, *n1b, *n2w, *n2b;+ QiMat in_proj, out_proj, lin1, lin2;+ float *in_proj_b, *out_proj_b, *lin1_b, *lin2_b;+} HeadLayer;++typedef struct {+ /* encoder config */+ int d, layers, heads, hd, inter, vocab, max_pos, window;+ float eps, theta_global, theta_local;+ int gelu_tanh;+ float *tok_emb; /* [vocab][d] */+ float *emb_norm_w, *emb_norm_b, *final_norm_w, *final_norm_b;+ EncLayer *L;+ float *rope_g, *rope_l; /* [max_pos][hd/2] cos then [max_pos][hd/2] sin */+ /* decision head */+ int head_layers, head_heads;+ HeadLayer *H;+ float *type_emb; /* [3][d] */+ float *sc_ln_w, *sc_ln_b, *sc2_w, sc2_b;+ QiMat sc1; float *sc1_b;+ int act_hidden, n_act;+ QiMat act1; float *act1_b;+ float *act2_w, *act2_b; /* [n_act][act_hidden] */+ char *action_names[DECIDE_MAX_ACTIONS];+ /* rl_agent_config.json */+ int max_len, head_max_len;+ double temperature[3];+ int n_buckets;+ char *bucket_key[32];+ double bucket_t[32];+ /* tokenizer */+ Tok tok;+ int nfc;+ int cls_id, sep_id, mask_id, pad_id;+ char *mask_str;+ unsigned char added_first[256]; /* first bytes of the added tokens */+ char model_name[128];+} Laya;++/* common.clamp_temperature: a usable temperature, confined to [0.5, 5]; a value+ * that is not a finite number (or is a JSON bool) falls back to 1.0. */+static double clamp_temperature(jval *v)+{+ if (!v || v->t != J_NUM || !isfinite(v->num)) return 1.0;+ double t = v->num;+ return t < 0.5 ? 0.5 : t > 5.0 ? 5.0 : t;+}++static float *load_f32(shards *S, const char *name, int64_t expect)+{+ st_tensor *t = st_find(S, name);+ if (!t) die("model.safetensors has no tensor %s", name);+ if (expect >= 0 && t->numel != expect)+ die("%s has %lld elements, expected %lld", name, (long long)t->numel, (long long)expect);+ float *out = (float *)xmalloc((size_t)t->numel * sizeof(float));+ st_read_f32(S, name, out, 0);+ return out;+}++static float *load_opt(shards *S, const char *name, int64_t expect)+{+ return st_find(S, name) ? load_f32(S, name, expect) : NULL;+}++static QiMat load_mat(shards *S, const char *name, int N, int K)+{+ QiMat m = {QI_F32, N, K, load_f32(S, name, (int64_t)N * K), NULL, 0};+ return m;+}++/* COLI_LAYA_INT8=1: the large matrices as int8 with one scale per row. Opt-in,+ * and only worth it where it was measured to leave the decisions unchanged. */+static void maybe_int8(QiMat *m)+{+ const char *flag = getenv("COLI_LAYA_INT8");+ if (!flag || strcmp(flag, "1")) return;+ int8_t *q = (int8_t *)xmalloc((size_t)m->N * m->K);+ float *sc = (float *)xmalloc((size_t)m->N * sizeof(float));+ qi_quantize_i8((const float *)m->w, m->N, m->K, q, sc);+ free((void *)m->w);+ m->fmt = QI_I8; m->w = q; m->sc = sc;+}++static void rope_table(float *table, int max_pos, int hd, float theta)+{+ /* ModernBertRotaryEmbedding.compute_default_rope_parameters, in float32 like+ * the reference: inv_freq = 1 / theta^(2i/hd), angle = pos * inv_freq. */+ int half = hd / 2;+ float *cosv = table, *sinv = table + (size_t)max_pos * half;+ for (int i = 0; i < half; i++) {+ float inv = 1.0f / powf(theta, (float)(2 * i) / (float)hd);+ for (int p = 0; p < max_pos; p++) {+ float a = (float)p * inv;+ cosv[(size_t)p * half + i] = cosf(a);+ sinv[(size_t)p * half + i] = sinf(a);+ }+ }+}++static const char *special_content(jval *cfg, const char *key)+{+ jval *v = json_get(cfg, key);+ if (v && v->t == J_STR) return v->str;+ if (v && v->t == J_OBJ) { jval *c = json_get(v, "content"); if (c && c->t == J_STR) return c->str; }+ return NULL;+}++static void load_tokenizer(Laya *M, const char *dir)+{+ char path[4096];+ snprintf(path, sizeof(path), "%s/tokenizer/tokenizer.json", dir);+ jval *tj = read_json(dir, "tokenizer/tokenizer.json");+ jval *model = json_get(tj, "model"), *mt = model ? json_get(model, "type") : NULL;+ if (!mt || mt->t != J_STR || strcmp(mt->str, "BPE"))+ die("tokenizer: only a byte-level BPE tokenizer is supported (this is %s)",+ mt && mt->t == J_STR ? mt->str : "unknown");+ jval *pt = json_get(tj, "pre_tokenizer"), *ptt = pt ? json_get(pt, "type") : NULL;+ if (!ptt || ptt->t != J_STR || strcmp(ptt->str, "ByteLevel") || !bool_or(pt, "use_regex", 1) ||+ bool_or(pt, "add_prefix_space", 0))+ die("tokenizer: expected a ByteLevel pre-tokenizer with the GPT-2 regex and no prefix space");+ jval *norm = json_get(tj, "normalizer");+ if (norm && norm->t == J_OBJ) {+ jval *nt = json_get(norm, "type");+ if (nt && nt->t == J_STR && !strcmp(nt->str, "NFC")) M->nfc = 1;+ else die("tokenizer: unsupported normalizer %s", nt && nt->t == J_STR ? nt->str : "?");+ } else if (norm && norm->t != J_NULL) {+ die("tokenizer: unsupported normalizer");+ }+ /* Added tokens: the reference matches them on the raw text (special ones) or+ * on the normalized text (the rest); NFC cannot create or break any of these+ * ASCII strings, so matching once after normalizing is the same split. The+ * one lstrip/rstrip token is [MASK], which never reaches the tokenizer:+ * build_sequence replaces it in every text first. */+ jval *added = json_get(tj, "added_tokens");+ for (int i = 0; added && i < added->len; i++) {+ jval *a = added->kids[i], *c = json_get(a, "content");+ if (!c || c->t != J_STR || !c->str[0]) continue;+ int is_mask = M->mask_str && !strcmp(c->str, M->mask_str);+ if (!is_mask && (bool_or(a, "lstrip", 0) || bool_or(a, "rstrip", 0) || bool_or(a, "single_word", 0)))+ die("tokenizer: added token %s strips whitespace, which this engine does not implement", c->str);+ M->added_first[(unsigned char)c->str[0]] = 1;+ }+ json_free(tj);+ tok_load(&M->tok, path);+ if (!M->tok.gpt2) die("tokenizer: not the GPT-2 byte-level pre-tokenizer");+}++static void load_model(Laya *M, const char *dir)+{+ memset(M, 0, sizeof(*M));+ /* rl_agent_config.json: the head and the calibration */+ jval *rc = read_json(dir, "rl_agent_config.json");+ M->head_layers = (int)num_or(rc, "head_layers", 2);+ M->max_len = (int)num_or(rc, "max_len", 512);+ M->head_max_len = (int)num_or(rc, "head_max_len", 192);+ const char *env;+ if ((env = getenv("COLI_LAYA_MAX_LEN")) && atoi(env) > 0) M->max_len = atoi(env);+ if ((env = getenv("COLI_LAYA_HEAD_MAX_LEN")) && atoi(env) > 0) M->head_max_len = atoi(env);+ jval *name = json_get(rc, "model_name");+ snprintf(M->model_name, sizeof(M->model_name), "%s", name && name->t == J_STR ? name->str : "laya");+ jval *temps = json_get(rc, "temperature");+ for (int i = 0; i < 3; i++)+ M->temperature[i] = temps && temps->t == J_ARR && temps->len == 3 ? clamp_temperature(temps->kids[i]) : 1.0;+ if (temps && (temps->t != J_ARR || temps->len != 3))+ die("rl_agent_config.json: temperature must be a list of 3 numbers");+ jval *tbo = json_get(rc, "temperature_by_options");+ for (int i = 0; tbo && tbo->t == J_OBJ && i < tbo->len && M->n_buckets < 32; i++) {+ M->bucket_key[M->n_buckets] = strdup(tbo->keys[i]);+ M->bucket_t[M->n_buckets++] = clamp_temperature(tbo->kids[i]);+ }+ jval *costs = json_get(rc, "act_costs");+ M->n_act = 1 + (costs && costs->t == J_OBJ ? costs->len : 0);+ if (M->n_act > DECIDE_MAX_ACTIONS) die("rl_agent_config.json: too many act_costs");+ M->action_names[0] = strdup("act");+ for (int i = 1; i < M->n_act; i++) M->action_names[i] = strdup(costs->keys[i - 1]);+ json_free(rc);++ /* encoder/config.json: ModernBERT */+ jval *ec = read_json(dir, "encoder/config.json");+ jval *mtype = json_get(ec, "model_type");+ if (!mtype || mtype->t != J_STR || strcmp(mtype->str, "modernbert"))+ die("encoder/config.json: model_type must be modernbert");+ M->d = (int)num_or(ec, "hidden_size", 0);+ M->layers = (int)num_or(ec, "num_hidden_layers", 0);+ M->heads = (int)num_or(ec, "num_attention_heads", 0);+ M->inter = (int)num_or(ec, "intermediate_size", 0);+ M->vocab = (int)num_or(ec, "vocab_size", 0);+ M->max_pos = (int)num_or(ec, "max_position_embeddings", 8192);+ M->window = (int)num_or(ec, "local_attention", 128) / 2;+ M->eps = (float)num_or(ec, "norm_eps", 1e-5);+ if (M->d < 1 || M->layers < 1 || M->heads < 1 || M->d % M->heads || M->inter < 1 || M->vocab < 1 ||+ M->max_pos < 1 || M->d > 16384 || M->layers > 256 || M->max_pos > 1 << 20)+ die("encoder/config.json: invalid geometry");+ M->hd = M->d / M->heads;+ if (M->hd % 2) die("encoder/config.json: head dimension must be even");+ jval *act = json_get(ec, "hidden_activation");+ if (act && act->t == J_STR) {+ if (!strcmp(act->str, "gelu")) M->gelu_tanh = 0;+ else if (!strcmp(act->str, "gelu_pytorch_tanh") || !strcmp(act->str, "gelu_new")) M->gelu_tanh = 1;+ else die("encoder/config.json: unsupported hidden_activation %s", act->str);+ }+ /* RoPE: transformers 5 writes rope_parameters per layer type; older configs+ * global_rope_theta / local_rope_theta (defaults 160000 / 10000). */+ M->theta_global = (float)num_or(ec, "global_rope_theta", 160000.0);+ M->theta_local = (float)num_or(ec, "local_rope_theta", 10000.0);+ jval *rp = json_get(ec, "rope_parameters");+ if (rp && rp->t == J_OBJ) {+ jval *f = json_get(rp, "full_attention"), *s = json_get(rp, "sliding_attention");+ if (f && f->t == J_OBJ) M->theta_global = (float)num_or(f, "rope_theta", M->theta_global);+ if (s && s->t == J_OBJ) M->theta_local = (float)num_or(s, "rope_theta", M->theta_local);+ jval *ft = f ? json_get(f, "rope_type") : NULL, *stp = s ? json_get(s, "rope_type") : NULL;+ if ((ft && ft->t == J_STR && strcmp(ft->str, "default")) ||+ (stp && stp->t == J_STR && strcmp(stp->str, "default")))+ die("encoder/config.json: only the default RoPE is supported");+ }+ M->L = (EncLayer *)xcalloc((size_t)M->layers, sizeof(EncLayer));+ jval *types = json_get(ec, "layer_types");+ int every = (int)num_or(ec, "global_attn_every_n_layers", 3);+ for (int l = 0; l < M->layers; l++) {+ if (types && types->t == J_ARR && types->len == M->layers && types->kids[l]->t == J_STR)+ M->L[l].global = !strcmp(types->kids[l]->str, "full_attention");+ else+ M->L[l].global = every > 0 ? l % every == 0 : 1;+ }+ json_free(ec);++ /* tokenizer: the special tokens by name, from tokenizer_config.json */+ jval *tc = read_json(dir, "tokenizer/tokenizer_config.json");+ const char *cls = special_content(tc, "cls_token"), *sep = special_content(tc, "sep_token");+ const char *mask = special_content(tc, "mask_token"), *pad = special_content(tc, "pad_token");+ if (!cls || !sep || !mask) die("tokenizer_config.json: needs cls_token, sep_token and mask_token");+ M->mask_str = strdup(mask);+ char *cls_s = strdup(cls), *sep_s = strdup(sep), *pad_s = pad ? strdup(pad) : NULL;+ json_free(tc);+ load_tokenizer(M, dir);+ M->cls_id = tok_id_of(&M->tok, cls_s);+ M->sep_id = tok_id_of(&M->tok, sep_s);+ M->mask_id = tok_id_of(&M->tok, M->mask_str);+ M->pad_id = pad_s ? tok_id_of(&M->tok, pad_s) : -1;+ if (M->cls_id < 0 || M->sep_id < 0 || M->mask_id < 0)+ die("tokenizer: cls/sep/mask are not added tokens of tokenizer.json");+ free(cls_s); free(sep_s); free(pad_s);++ /* weights */+ shards S;+ st_init(&S, dir);+ int d = M->d, I = M->inter;+ M->tok_emb = load_f32(&S, "encoder.embeddings.tok_embeddings.weight", (int64_t)M->vocab * d);+ M->emb_norm_w = load_f32(&S, "encoder.embeddings.norm.weight", d);+ M->emb_norm_b = load_opt(&S, "encoder.embeddings.norm.bias", d);+ M->final_norm_w = load_f32(&S, "encoder.final_norm.weight", d);+ M->final_norm_b = load_opt(&S, "encoder.final_norm.bias", d);+ char n[256];+ for (int l = 0; l < M->layers; l++) {+ EncLayer *E = &M->L[l];+#define NAME(fmt) (snprintf(n, sizeof(n), "encoder.layers.%d." fmt, l), n)+ if (l > 0) {+ E->attn_norm_w = load_f32(&S, NAME("attn_norm.weight"), d);+ E->attn_norm_b = load_opt(&S, NAME("attn_norm.bias"), d);+ }+ E->wqkv = load_mat(&S, NAME("attn.Wqkv.weight"), 3 * d, d);+ E->bqkv = load_opt(&S, NAME("attn.Wqkv.bias"), 3 * d);+ E->wo = load_mat(&S, NAME("attn.Wo.weight"), d, d);+ E->bo = load_opt(&S, NAME("attn.Wo.bias"), d);+ E->mlp_norm_w = load_f32(&S, NAME("mlp_norm.weight"), d);+ E->mlp_norm_b = load_opt(&S, NAME("mlp_norm.bias"), d);+ E->wi = load_mat(&S, NAME("mlp.Wi.weight"), 2 * I, d);+ E->bi = load_opt(&S, NAME("mlp.Wi.bias"), 2 * I);+ E->wo2 = load_mat(&S, NAME("mlp.Wo.weight"), d, I);+ E->bo2 = load_opt(&S, NAME("mlp.Wo.bias"), d);+ maybe_int8(&E->wqkv); maybe_int8(&E->wo); maybe_int8(&E->wi); maybe_int8(&E->wo2);+#undef NAME+ }+ M->head_heads = d / 64 > 1 ? d / 64 : 1; /* DecisionModel: nhead = max(1, d // 64) */+ if (d % M->head_heads) die("head: hidden size %d does not split into %d heads", d, M->head_heads);+ M->H = (HeadLayer *)xcalloc((size_t)(M->head_layers > 0 ? M->head_layers : 1), sizeof(HeadLayer));+ for (int l = 0; l < M->head_layers; l++) {+ HeadLayer *H = &M->H[l];+#define NAME(fmt) (snprintf(n, sizeof(n), "head.layers.%d." fmt, l), n)+ H->n1w = load_f32(&S, NAME("norm1.weight"), d);+ H->n1b = load_f32(&S, NAME("norm1.bias"), d);+ H->n2w = load_f32(&S, NAME("norm2.weight"), d);+ H->n2b = load_f32(&S, NAME("norm2.bias"), d);+ H->in_proj = load_mat(&S, NAME("self_attn.in_proj_weight"), 3 * d, d);+ H->in_proj_b = load_f32(&S, NAME("self_attn.in_proj_bias"), 3 * d);+ H->out_proj = load_mat(&S, NAME("self_attn.out_proj.weight"), d, d);+ H->out_proj_b = load_f32(&S, NAME("self_attn.out_proj.bias"), d);+ H->lin1 = load_mat(&S, NAME("linear1.weight"), 4 * d, d);+ H->lin1_b = load_f32(&S, NAME("linear1.bias"), 4 * d);+ H->lin2 = load_mat(&S, NAME("linear2.weight"), d, 4 * d);+ H->lin2_b = load_f32(&S, NAME("linear2.bias"), d);+ maybe_int8(&H->in_proj); maybe_int8(&H->out_proj); maybe_int8(&H->lin1); maybe_int8(&H->lin2);+#undef NAME+ }+ M->type_emb = load_f32(&S, "type_emb.weight", 3 * d);+ M->sc_ln_w = load_f32(&S, "scorer.0.weight", d);+ M->sc_ln_b = load_f32(&S, "scorer.0.bias", d);+ M->sc1 = load_mat(&S, "scorer.1.weight", d, d);+ M->sc1_b = load_f32(&S, "scorer.1.bias", d);+ M->sc2_w = load_f32(&S, "scorer.3.weight", d);+ float *b = load_f32(&S, "scorer.3.bias", 1);+ M->sc2_b = b[0];+ free(b);+ st_tensor *a1 = st_find(&S, "act_head.0.weight");+ if (!a1 || a1->rank != 2 || a1->shape[1] != d + 4) die("act_head.0.weight: expected [hidden, %d]", d + 4);+ M->act_hidden = (int)a1->shape[0];+ M->act1 = load_mat(&S, "act_head.0.weight", M->act_hidden, d + 4);+ M->act1_b = load_f32(&S, "act_head.0.bias", M->act_hidden);+ st_tensor *a2 = st_find(&S, "act_head.2.weight");+ if (!a2 || a2->rank != 2 || a2->shape[0] != M->n_act || a2->shape[1] != M->act_hidden)+ die("act_head.2.weight: expected [%d, %d] (one row per action of act_costs)", M->n_act, M->act_hidden);+ M->act2_w = load_f32(&S, "act_head.2.weight", (int64_t)M->n_act * M->act_hidden);+ M->act2_b = load_f32(&S, "act_head.2.bias", M->n_act);+ st_destroy(&S);++ int half = M->hd / 2;+ M->rope_g = (float *)xmalloc((size_t)M->max_pos * half * 2 * sizeof(float));+ M->rope_l = (float *)xmalloc((size_t)M->max_pos * half * 2 * sizeof(float));+ rope_table(M->rope_g, M->max_pos, M->hd, M->theta_global);+ rope_table(M->rope_l, M->max_pos, M->hd, M->theta_local);+ if (M->max_len > M->max_pos) M->max_len = M->max_pos;+}++/* -------------------------------------------------------------- tokenizer */++/* tokenizer.json applied to one text: NFC, the added tokens split out+ * (leftmost, longest), then the GPT-2 pre-tokenizer and BPE on what is left+ * (tok.h). Returns the ids; *count their number. */+static int *laya_encode(Laya *M, const char *text, size_t len, int *count)+{+ char *norm = NULL;+ size_t norm_len = len;+ const char *p = text;+ if (M->nfc) {+ if (q38_nfc_normalize(text, len, &norm, &norm_len) != 0) die("NFC normalization failed");+ p = norm;+ }+ int cap = (int)norm_len + 8, n = 0;+ int *out = (int *)xmalloc((size_t)cap * sizeof(int));+ const unsigned char *u = (const unsigned char *)p;+ int i = 0, L = (int)norm_len;+ Tok *T = &M->tok;+ while (i < L) {+ int hit = -1, hit_len = 0, hit_id = -1;+ for (int j = i; j < L && hit < 0; j++) {+ if (!M->added_first[u[j]]) continue;+ for (int k = 0; k < T->nsp; k++) {+ int sl = T->sp[k].len;+ if (sl > 0 && j + sl <= L && !memcmp(u + j, T->sp[k].str, (size_t)sl)) {+ hit = j; hit_len = sl; hit_id = T->sp[k].id;+ break;+ }+ }+ }+ int end = hit < 0 ? L : hit;+ if (end > i) pretok_chunk_gpt2(T, u, i, end, out, &n, cap);+ if (hit < 0) break;+ if (n < cap) out[n++] = hit_id;+ i = hit + hit_len;+ }+ free(norm);+ *count = n;+ return out;+}++/* str.replace(needle, " ") */+static char *replace_all(const char *s, const char *needle)+{+ size_t nl = strlen(needle), sl = strlen(s);+ char *out = (char *)xmalloc(sl + 1);+ size_t o = 0;+ for (size_t i = 0; i < sl;) {+ if (nl && i + nl <= sl && !memcmp(s + i, needle, nl)) { out[o++] = ' '; i += nl; }+ else out[o++] = s[i++];+ }+ out[o] = 0;+ return out;+}++/* ------------------------------------------------------------ the sequence */++typedef struct {+ int *ids, n;+ int *markers, n_markers;+ int qtype;+ int state_tokens, state_used;+} Seq;++static void seq_free(Seq *s) { free(s->ids); free(s->markers); memset(s, 0, sizeof(*s)); }++/* common.render_options: the option texts in label-index order. */+static char *render_option(const DecideQuestion *q, int i)+{+ const DecideOption *o = &q->options[i];+ const char *text = o->text && o->text[0] ? o->text : NULL;+ size_t size = strlen(o->label) + (o->text ? strlen(o->text) : 0) + 64;+ char *out = (char *)xmalloc(size);+ if (q->type == DECIDE_CHOICE) {+ if (text) snprintf(out, size, "%s: %s", o->label, text);+ else snprintf(out, size, "%s", o->label);+ } else if (q->type == DECIDE_SCORE) {+ snprintf(out, size, "level %d: %s", i, o->text ? o->text : "");+ } else {+ /* noul: [false, true], each with its default description */+ const char *fallback = i == 0 ? "no, the statement does not hold" : "yes, the statement holds";+ snprintf(out, size, "%s: %s", o->label, text ? text : fallback);+ }+ return out;+}++/* common.build_head + build_sequence:+ * [CLS] <type> question: <ins> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP]+ * Returns 1, or 0 with the reason when not every option marker fits. */+static int build_sequence(Laya *M, const DecideQuestion *q, const int *state, int n_state,+ int truncate_left, Seq *out, char *err, size_t cap)+{+ memset(out, 0, sizeof(*out));+ out->qtype = q->type;+ const char *tname = decide_type_name(q->type);+ char *ins = replace_all(q->instructions, M->mask_str);+ size_t hl = strlen(tname) + strlen(ins) + 16;+ char *head_text = (char *)xmalloc(hl);+ snprintf(head_text, hl, "%s question: %s", tname, ins);+ int n_head = 0;+ int *head = laya_encode(M, head_text, strlen(head_text), &n_head);+ free(head_text); free(ins);++ int K = q->n_options;+ int **opt = (int **)xcalloc((size_t)K, sizeof(int *));+ int *olen = (int *)xcalloc((size_t)K, sizeof(int));+ int total = 0;+ for (int i = 0; i < K; i++) {+ char *rendered = render_option(q, i);+ char *clean = replace_all(rendered, M->mask_str);+ size_t cl = strlen(clean);+ char *spaced = (char *)xmalloc(cl + 2);+ spaced[0] = ' ';+ memcpy(spaced + 1, clean, cl + 1);+ int n = 0;+ int *ids = laya_encode(M, spaced, cl + 1, &n);+ if (n > 48) n = 48; /* truncation=True, max_length=48 */+ opt[i] = (int *)xmalloc((size_t)(n + 1) * sizeof(int));+ opt[i][0] = M->mask_id;+ memcpy(opt[i] + 1, ids, (size_t)n * sizeof(int));+ olen[i] = n + 1;+ total += n + 1;+ free(ids); free(spaced); free(clean); free(rendered);+ }+ int budget = M->head_max_len - total;+ if (budget < 16) { /* shrink every option evenly */+ int per = floordiv(M->head_max_len - 16, K > 1 ? K : 1);+ if (per < 4) per = 4;+ total = 0;+ for (int i = 0; i < K; i++) { if (olen[i] > per) olen[i] = per; total += olen[i]; }+ budget = M->head_max_len - total;+ }+ int keep = budget > 8 ? budget : 8;+ if (n_head > keep) n_head = keep;++ int room_cap = 2 + n_head + total + 1 + n_state + 1;+ int *ids = (int *)xmalloc((size_t)room_cap * sizeof(int));+ int *markers = (int *)xmalloc((size_t)K * sizeof(int));+ int n = 0;+ ids[n++] = M->cls_id;+ memcpy(ids + n, head, (size_t)n_head * sizeof(int)); n += n_head;+ ids[n++] = M->sep_id;+ for (int i = 0; i < K; i++) {+ markers[i] = n;+ memcpy(ids + n, opt[i], (size_t)olen[i] * sizeof(int));+ n += olen[i];+ }+ ids[n++] = M->sep_id;+ int room = M->max_len - n - 1;+ if (room < 0) room = 0;+ int take = n_state < room ? n_state : room;+ const int *from = truncate_left ? state + (n_state - take) : state;+ memcpy(ids + n, from, (size_t)take * sizeof(int)); n += take;+ ids[n++] = M->sep_id;+ if (n > M->max_len) n = M->max_len;+ int kept = 0;+ for (int i = 0; i < K; i++) if (markers[i] < M->max_len) markers[kept++] = markers[i];++ for (int i = 0; i < K; i++) free(opt[i]);+ free(opt); free(olen); free(head);+ out->ids = ids; out->n = n; out->markers = markers; out->n_markers = kept;+ out->state_tokens = n_state; out->state_used = take;+ if (kept != K) {+ snprintf(err, cap, "questions.%s: only %d of its %d option markers fit in max_len=%d with "+ "head_max_len=%d spent on the question; lower head_max_len, raise max_len, or use "+ "fewer options", q->id, kept, K, M->max_len, M->head_max_len);+ return 0;+ }+ return 1;+}++/* ------------------------------------------------------------- the forward */++static inline float gelu(float x, int tanh_form)+{+ if (tanh_form) return 0.5f * x * (1.0f + tanhf(0.7978845608028654f * (x + 0.044715f * x * x * x)));+ return 0.5f * x * (1.0f + erff(x * 0.70710678118654752f));+}++/* LayerNorm over rows of `d`: biased variance, eps inside the root. */+static void layernorm(float *out, const float *x, const float *w, const float *b, int rows, int d, float eps)+{+ #pragma omp parallel for schedule(static)+ for (int r = 0; r < rows; r++) {+ const float *xr = x + (size_t)r * d;+ float *o = out + (size_t)r * d;+ double mean = 0, var = 0;+ for (int i = 0; i < d; i++) mean += xr[i];+ mean /= d;+ for (int i = 0; i < d; i++) { double c = xr[i] - mean; var += c * c; }+ var /= d;+ float rstd = (float)(1.0 / sqrt(var + eps));+ for (int i = 0; i < d; i++) {+ float v = (float)(xr[i] - mean) * rstd * w[i];+ o[i] = b ? v + b[i] : v;+ }+ }+}++static void add_rows(float *x, const float *y, size_t n)+{+ #pragma omp parallel for schedule(static)+ for (size_t i = 0; i < n; i++) x[i] += y[i];+}++static void gemm(float *Y, const float *X, int M, const QiMat *W, const float *bias)+{+ qi_gemm(Y, X, M, W, bias);+}++/* Rotate q and k of every row in place: q*cos + rotate_half(q)*sin, positions+ * counted from each sequence's start. */+static void apply_rope(float *qkv, int T, const int *row_pos, int d, int heads, int hd,+ const float *table, int max_pos)+{+ int half = hd / 2;+ const float *cosv = table, *sinv = table + (size_t)max_pos * half;+ #pragma omp parallel for schedule(static)+ for (int r = 0; r < T; r++) {+ const float *c = cosv + (size_t)row_pos[r] * half, *s = sinv + (size_t)row_pos[r] * half;+ for (int part = 0; part < 2; part++) {+ float *base = qkv + (size_t)r * 3 * d + (size_t)part * d;+ for (int h = 0; h < heads; h++) {+ float *v = base + (size_t)h * hd;+ for (int i = 0; i < half; i++) {+ float a = v[i], b = v[i + half];+ v[i] = a * c[i] - b * s[i];+ v[i + half] = b * c[i] + a * s[i];+ }+ }+ }+ }+}++/* Softmax attention inside each sequence. qkv rows are [q | k | v], each d+ * wide with the heads side by side. window < 0 is full attention; otherwise+ * key j is visible to query i when |i - j| <= window. `only` (optional) marks+ * the query rows whose output is needed; the rest are skipped. */+static void attention(const float *qkv, float *out, int T, int d, int heads, int hd,+ const int *row_seq, const int *seq_off, int window, const unsigned char *only)+{+ const float scale = 1.0f / sqrtf((float)hd);+ #pragma omp parallel+ {+ int cap = 0;+ for (int r = 0; r < T; r++) {+ int L = seq_off[row_seq[r] + 1] - seq_off[row_seq[r]];+ if (L > cap) cap = L;+ }+ float *score = (float *)xmalloc((size_t)(cap > 0 ? cap : 1) * sizeof(float));+ float *acc = (float *)xmalloc((size_t)hd * sizeof(float));+ #pragma omp for schedule(dynamic, 8)+ for (long task = 0; task < (long)T * heads; task++) {+ int r = (int)(task / heads), h = (int)(task % heads);+ if (only && !only[r]) continue;+ int s = row_seq[r], start = seq_off[s], L = seq_off[s + 1] - start, i = r - start;+ int lo = 0, hi = L - 1;+ if (window >= 0) { lo = i - window < 0 ? 0 : i - window; hi = i + window > L - 1 ? L - 1 : i + window; }+ const float *q = qkv + (size_t)r * 3 * d + (size_t)h * hd;+ float mx = -INFINITY;+ for (int j = lo; j <= hi; j++) {+ const float *k = qkv + (size_t)(start + j) * 3 * d + d + (size_t)h * hd;+ float v = dot_f32_lanes(q, k, hd) * scale;+ score[j - lo] = v;+ if (v > mx) mx = v;+ }+ double sum = 0;+ for (int j = lo; j <= hi; j++) { float e = expf(score[j - lo] - mx); score[j - lo] = e; sum += e; }+ float inv = (float)(1.0 / sum);+ for (int t = 0; t < hd; t++) acc[t] = 0.f;+ for (int j = lo; j <= hi; j++) {+ const float *v = qkv + (size_t)(start + j) * 3 * d + 2 * d + (size_t)h * hd;+ float p = score[j - lo] * inv;+ for (int t = 0; t < hd; t++) acc[t] += p * v[t];+ }+ memcpy(out + (size_t)r * d + (size_t)h * hd, acc, (size_t)hd * sizeof(float));+ }+ free(score); free(acc);+ }+}++typedef struct {+ double *logits; /* [n_markers] */+ double actions[DECIDE_MAX_ACTIONS];+} SeqOut;++/* DecisionModel.forward over a batch of sequences. */+static void forward(Laya *M, Seq *seqs, int S, SeqOut *outs)+{+ int d = M->d, I = M->inter;+ int *seq_off = (int *)xmalloc((size_t)(S + 1) * sizeof(int));+ seq_off[0] = 0;+ for (int s = 0; s < S; s++) seq_off[s + 1] = seq_off[s] + seqs[s].n;+ int T = seq_off[S];+ int *row_seq = (int *)xmalloc((size_t)T * sizeof(int));+ int *row_pos = (int *)xmalloc((size_t)T * sizeof(int));+ for (int s = 0; s < S; s++)+ for (int i = 0; i < seqs[s].n; i++) { row_seq[seq_off[s] + i] = s; row_pos[seq_off[s] + i] = i; }++ float *x = (float *)xmalloc((size_t)T * d * sizeof(float));+ float *h = (float *)xmalloc((size_t)T * d * sizeof(float));+ float *att = (float *)xmalloc((size_t)T * d * sizeof(float));+ int wide = 3 * d; /* the widest activation: qkv, the GeGLU input or the head FFN */+ if (2 * I > wide) wide = 2 * I;+ if (4 * d > wide) wide = 4 * d;+ float *big = (float *)xmalloc((size_t)T * wide * sizeof(float));+ float *mid = (float *)xmalloc((size_t)T * (I > d ? I : d) * sizeof(float));++ /* embeddings: tok_embeddings, then LayerNorm */+ for (int r = 0; r < T; r++) {+ int id = seqs[row_seq[r]].ids[row_pos[r]];+ if (id < 0 || id >= M->vocab) id = 0;+ memcpy(h + (size_t)r * d, M->tok_emb + (size_t)id * d, (size_t)d * sizeof(float));+ }+ layernorm(x, h, M->emb_norm_w, M->emb_norm_b, T, d, M->eps);++ for (int l = 0; l < M->layers; l++) {+ EncLayer *E = &M->L[l];+ const float *in = x;+ if (E->attn_norm_w) { layernorm(h, x, E->attn_norm_w, E->attn_norm_b, T, d, M->eps); in = h; }+ gemm(big, in, T, &E->wqkv, E->bqkv);+ apply_rope(big, T, row_pos, d, M->heads, M->hd, E->global ? M->rope_g : M->rope_l, M->max_pos);+ attention(big, att, T, d, M->heads, M->hd, row_seq, seq_off, E->global ? -1 : M->window, NULL);+ gemm(h, att, T, &E->wo, E->bo);+ add_rows(x, h, (size_t)T * d);+ layernorm(h, x, E->mlp_norm_w, E->mlp_norm_b, T, d, M->eps);+ gemm(big, h, T, &E->wi, E->bi);+ #pragma omp parallel for schedule(static)+ for (int r = 0; r < T; r++) {+ const float *u = big + (size_t)r * 2 * I;+ float *g = mid + (size_t)r * I;+ for (int j = 0; j < I; j++) g[j] = gelu(u[j], M->gelu_tanh) * u[I + j];+ }+ gemm(h, mid, T, &E->wo2, E->bo2);+ add_rows(x, h, (size_t)T * d);+ }+ layernorm(h, x, M->final_norm_w, M->final_norm_b, T, d, M->eps);+ float *t = x; x = h; h = t;++ /* the decision head: + type_emb, then norm_first TransformerEncoderLayers */+ #pragma omp parallel for schedule(static)+ for (int r = 0; r < T; r++) {+ const float *te = M->type_emb + (size_t)seqs[row_seq[r]].qtype * d;+ float *xr = x + (size_t)r * d;+ for (int i = 0; i < d; i++) xr[i] += te[i];+ }+ int hh = M->head_heads, hhd = d / hh;+ for (int l = 0; l < M->head_layers; l++) {+ HeadLayer *H = &M->H[l];+ layernorm(h, x, H->n1w, H->n1b, T, d, 1e-5f);+ gemm(big, h, T, &H->in_proj, H->in_proj_b);+ attention(big, att, T, d, hh, hhd, row_seq, seq_off, -1, NULL);+ gemm(h, att, T, &H->out_proj, H->out_proj_b);+ add_rows(x, h, (size_t)T * d);+ layernorm(h, x, H->n2w, H->n2b, T, d, 1e-5f);+ gemm(big, h, T, &H->lin1, H->lin1_b);+ #pragma omp parallel for schedule(static)+ for (size_t i = 0; i < (size_t)T * 4 * d; i++) if (big[i] < 0.f) big[i] = 0.f;+ gemm(h, big, T, &H->lin2, H->lin2_b);+ add_rows(x, h, (size_t)T * d);+ }++ /* scorer at the markers: LayerNorm -> Linear -> GELU -> Linear(d, 1) */+ int nm = 0;+ for (int s = 0; s < S; s++) nm += seqs[s].n_markers;+ float *m = (float *)xmalloc((size_t)(nm > 0 ? nm : 1) * d * sizeof(float));+ float *mn = (float *)xmalloc((size_t)(nm > 0 ? nm : 1) * d * sizeof(float));+ float *m1 = (float *)xmalloc((size_t)(nm > 0 ? nm : 1) * d * sizeof(float));+ for (int s = 0, k = 0; s < S; s++)+ for (int i = 0; i < seqs[s].n_markers; i++, k++)+ memcpy(m + (size_t)k * d, x + (size_t)(seq_off[s] + seqs[s].markers[i]) * d, (size_t)d * sizeof(float));+ layernorm(mn, m, M->sc_ln_w, M->sc_ln_b, nm, d, 1e-5f);+ gemm(m1, mn, nm, &M->sc1, M->sc1_b);+ for (int s = 0, k = 0; s < S; s++) {+ int K = seqs[s].n_markers;+ outs[s].logits = (double *)xcalloc((size_t)(K > 0 ? K : 1), sizeof(double));+ for (int i = 0; i < K; i++, k++) {+ float *row = m1 + (size_t)k * d;+ for (int j = 0; j < d; j++) row[j] = gelu(row[j], 0);+ outs[s].logits[i] = (double)(dot_f32_lanes(row, M->sc2_w, d) + M->sc2_b);+ }+ }++ /* act head: [pooled h[0], top1, top1 - top2, normalized entropy, k / 255],+ * the distribution being the raw (untempered) softmax of the logits. */+ float *feat = (float *)xmalloc((size_t)S * (d + 4) * sizeof(float));+ float *hid = (float *)xmalloc((size_t)S * M->act_hidden * sizeof(float));+ for (int s = 0; s < S; s++) {+ int K = seqs[s].n_markers;+ float *f = feat + (size_t)s * (d + 4);+ memcpy(f, x + (size_t)seq_off[s] * d, (size_t)d * sizeof(float));+ float mx = -INFINITY;+ for (int i = 0; i < K; i++) if ((float)outs[s].logits[i] > mx) mx = (float)outs[s].logits[i];+ float *p = (float *)xmalloc((size_t)(K > 0 ? K : 1) * sizeof(float));+ float sum = 0.f;+ for (int i = 0; i < K; i++) { p[i] = expf((float)outs[s].logits[i] - mx); sum += p[i]; }+ float top1 = 0.f, top2 = 0.f, ent = 0.f;+ for (int i = 0; i < K; i++) {+ p[i] /= sum;+ if (p[i] > top1) { top2 = top1; top1 = p[i]; } else if (p[i] > top2) top2 = p[i];+ ent -= p[i] * logf(p[i] > 1e-9f ? p[i] : 1e-9f);+ }+ float kk = (float)(K < 2 ? 2 : K);+ f[d] = top1; f[d + 1] = top1 - top2; f[d + 2] = ent / logf(kk); f[d + 3] = kk / 255.0f;+ free(p);+ }+ gemm(hid, feat, S, &M->act1, M->act1_b);+ for (int s = 0; s < S; s++) {+ float *row = hid + (size_t)s * M->act_hidden;+ for (int j = 0; j < M->act_hidden; j++) row[j] = gelu(row[j], 0);+ double z[DECIDE_MAX_ACTIONS], mx = -INFINITY, sum = 0;+ for (int a = 0; a < M->n_act; a++) {+ z[a] = (double)(dot_f32_lanes(row, M->act2_w + (size_t)a * M->act_hidden, M->act_hidden) + M->act2_b[a]);+ if (z[a] > mx) mx = z[a];+ }+ for (int a = 0; a < M->n_act; a++) { z[a] = exp(z[a] - mx); sum += z[a]; }+ for (int a = 0; a < M->n_act; a++) outs[s].actions[a] = z[a] / sum;+ }++ free(feat); free(hid); free(m); free(mn); free(m1);+ free(x); free(h); free(att); free(big); free(mid);+ free(seq_off); free(row_seq); free(row_pos);+}++/* ------------------------------------------------------------ calibration */++/* common.temp_bucket */+static double temperature_for(Laya *M, int type, int k)+{+ const char *size = k <= 2 ? "2" : k <= 5 ? "3-5" : k <= 10 ? "6-10" : "11+";+ char key[32];+ snprintf(key, sizeof(key), "%s:%s", decide_type_name(type), size);+ for (int i = 0; i < M->n_buckets; i++) if (!strcmp(M->bucket_key[i], key)) return M->bucket_t[i];+ return M->temperature[type];+}++/* ------------------------------------------------------- the DECIDE call */++typedef struct {+ Laya *M;+ int dump_ids;+ Seq *last; /* the sequences of the last record, for --records --ids */+ int last_n;+} LayaCtx;++/* Most rows one forward takes before the batch is split: the activations of a+ * batch are about 12 * hidden floats per row. */+#define LAYA_MAX_BATCH_ROWS 16384++static int laya_decide(void *opaque, const DecideRecord *rec, DecideAnswer *answers,+ int *input_tokens, char *err, size_t cap)+{+ LayaCtx *ctx = (LayaCtx *)opaque;+ Laya *M = ctx->M;+ for (int i = 0; i < ctx->last_n; i++) seq_free(&ctx->last[i]);+ free(ctx->last); ctx->last = NULL; ctx->last_n = 0;++ char *state = replace_all(rec->state, M->mask_str);+ int n_state = 0;+ int *state_ids = laya_encode(M, state, strlen(state), &n_state);+ free(state);+ /* Agent._encode_state: a list (a conversation) keeps its newest turns */+ int truncate_left = rec->state_type == DECIDE_STATE_ARRAY;+ int Q = rec->n_questions;+ Seq *seqs = (Seq *)xcalloc((size_t)Q, sizeof(Seq));+ for (int q = 0; q < Q; q++) {+ if (!build_sequence(M, &rec->questions[q], state_ids, n_state, truncate_left, &seqs[q], err, cap)) {+ for (int i = 0; i <= q; i++) seq_free(&seqs[i]);+ free(seqs); free(state_ids);+ return 0;+ }+ }+ free(state_ids);+ SeqOut *outs = (SeqOut *)xcalloc((size_t)Q, sizeof(SeqOut));+ for (int start = 0; start < Q;) {+ int end = start, rows = 0;+ while (end < Q && (end == start || rows + seqs[end].n <= LAYA_MAX_BATCH_ROWS)) rows += seqs[end++].n;+ forward(M, seqs + start, end - start, outs + start);+ start = end;+ }+ int total = 0;+ for (int q = 0; q < Q; q++) {+ DecideAnswer *a = &answers[q];+ int K = seqs[q].n_markers;+ a->n = K;+ a->logits = (double *)xcalloc((size_t)K, sizeof(double));+ a->probs = (double *)xcalloc((size_t)K, sizeof(double));+ a->temperature = temperature_for(M, seqs[q].qtype, K);+ double mx = -INFINITY, sum = 0;+ for (int i = 0; i < K; i++) {+ a->logits[i] = outs[q].logits[i];+ double z = (double)(float)outs[q].logits[i] / a->temperature;+ a->probs[i] = z;+ if (z > mx) mx = z;+ }+ for (int i = 0; i < K; i++) { a->probs[i] = exp(a->probs[i] - mx); sum += a->probs[i]; }+ for (int i = 0; i < K; i++) a->probs[i] /= sum;+ a->n_actions = M->n_act;+ for (int i = 0; i < M->n_act; i++) { a->action_names[i] = M->action_names[i]; a->actions[i] = outs[q].actions[i]; }+ a->tokens = seqs[q].n;+ a->state_tokens = seqs[q].state_tokens;+ a->state_dropped = seqs[q].state_tokens - seqs[q].state_used;+ total += seqs[q].n;+ free(outs[q].logits);+ }+ free(outs);+ *input_tokens = total;+ if (ctx->dump_ids) { ctx->last = seqs; ctx->last_n = Q; }+ else { for (int q = 0; q < Q; q++) seq_free(&seqs[q]); free(seqs); }+ return 1;+}++/* ------------------------------------------------------------------- main */++static double rss_gb(void)+{+#ifdef __linux__+ FILE *f = fopen("/proc/self/statm", "r");+ long pages = 0, resident = 0;+ if (f) { if (fscanf(f, "%ld %ld", &pages, &resident) != 2) resident = 0; fclose(f); }+ return resident * (double)sysconf(_SC_PAGESIZE) / 1e9;+#else+ return 0.0;+#endif+}++static void print_json_string(FILE *out, const char *s)+{+ DecideBuf b = {0};+ decide_buf_string(&b, s);+ fputs(b.data ? b.data : "\"\"", out);+ free(b.data);+}++static int run_records(Laya *M, const char *path, int dump_ids)+{+ char *text = read_text(path);+ if (!text) die("cannot read %s", path);+ jval *root = json_parse_checked(text);+ if (!root || root->t != J_ARR) die("%s: expected a JSON array of records", path);+ LayaCtx ctx = {M, dump_ids, NULL, 0};+ /* Each record is the text of a DECIDE payload and goes through+ * decide_record_parse, the path the serve loop takes. */+ for (int i = 0; i < root->len; i++) {+ jval *r = root->kids[i];+ jval *payload = json_get(r, "payload");+ if (!payload || payload->t != J_STR) die("%s: record %d has no string \"payload\"", path, i);+ DecideRecord rec;+ char reason[1024];+ if (!decide_record_parse(payload->str, &rec, reason, sizeof(reason))) {+ printf("{\"error\":"); print_json_string(stdout, reason); printf("}\n");+ continue;+ }+ DecideAnswer *answers = (DecideAnswer *)xcalloc((size_t)rec.n_questions, sizeof(DecideAnswer));+ int tokens = 0;+ double t0 = now_ms();+ int ok = laya_decide(&ctx, &rec, answers, &tokens, reason, sizeof(reason));+ double ms = now_ms() - t0;+ if (!ok) {+ printf("{\"error\":"); print_json_string(stdout, reason); printf("}\n");+ } else {+ size_t len = 0;+ char *json = decide_answers_json(&rec, answers, tokens, ms, &len);+ if (dump_ids && json && len > 1) {+ json[len - 1] = 0; /* reopen the object */+ fputs(json, stdout);+ printf(",\"sequences\":[");+ for (int q = 0; q < ctx.last_n; q++) {+ printf("%s{\"ids\":[", q ? "," : "");+ for (int t = 0; t < ctx.last[q].n; t++) printf("%s%d", t ? "," : "", ctx.last[q].ids[t]);+ printf("],\"markers\":[");+ for (int t = 0; t < ctx.last[q].n_markers; t++) printf("%s%d", t ? "," : "", ctx.last[q].markers[t]);+ printf("]}");+ }+ printf("]}\n");+ } else if (json) {+ fputs(json, stdout); fputc('\n', stdout);+ }+ free(json);+ }+ fflush(stdout);+ decide_answers_free(answers, rec.n_questions);+ free(answers);+ decide_record_free(&rec);+ }+ json_free(root);+ free(text);+ return 0;+}++static int run_tokenize(Laya *M, const char *path)+{+ char *text = read_text(path);+ if (!text) die("cannot read %s", path);+ jval *root = json_parse_checked(text);+ if (!root || root->t != J_ARR) die("%s: expected a JSON array of strings", path);+ printf("[");+ for (int i = 0; i < root->len; i++) {+ jval *s = root->kids[i];+ if (s->t != J_STR) die("%s: entry %d is not a string", path, i);+ int n = 0;+ int *ids = laya_encode(M, s->str, strlen(s->str), &n);+ printf("%s[", i ? "," : "");+ for (int t = 0; t < n; t++) printf("%s%d", t ? "," : "", ids[t]);+ printf("]");+ free(ids);+ }+ printf("]\n");+ json_free(root);+ free(text);+ return 0;+}++int main(int argc, char **argv)+{+ const char *model = NULL, *records = NULL, *tokenize = NULL;+ int dump_ids = 0;+ for (int i = 1; i < argc; i++) {+ if (!strcmp(argv[i], "--model") && i + 1 < argc) model = argv[++i];+ else if (!strcmp(argv[i], "--records") && i + 1 < argc) records = argv[++i];+ else if (!strcmp(argv[i], "--tokenize") && i + 1 < argc) tokenize = argv[++i];+ else if (!strcmp(argv[i], "--ids")) dump_ids = 1;+ else if (!strcmp(argv[i], "--help") || !strcmp(argv[i], "-h")) {+ printf("usage: SERVE=1 SNAP=<dir> laya serve DECIDE on stdin/stdout\n"+ " laya --model DIR --records FILE [--ids]\n"+ " laya --model DIR --tokenize FILE\n");+ return 0;+ }+ /* a bare number is the gateway's cache-cap argument: nothing to cache here */+ }+ const char *serve = getenv("SERVE");+ int serving = serve && !strcmp(serve, "1") && !records && !tokenize;+ if (!model) model = getenv("SNAP");+ if (!model) die("no model: pass --model DIR or set SNAP");+ if (serving) coli_serve_stdio_init();+ coli_omp_tune_threads("laya"); /* physical cores: SMT halves the GEMMs here too */+ double t0 = now_ms();+ Laya *M = (Laya *)xcalloc(1, sizeof(Laya));+ load_model(M, model);+ fprintf(stderr, "[laya] %s: ModernBERT %d layers x %d, %d heads; head %d layers; max_len %d "+ "(head %d); loaded in %.1f s%s\n", M->model_name, M->layers, M->d, M->heads,+ M->head_layers, M->max_len, M->head_max_len, (now_ms() - t0) / 1e3,+ getenv("COLI_LAYA_INT8") && !strcmp(getenv("COLI_LAYA_INT8"), "1") ? ", int8 weights" : "");+ if (tokenize) return run_tokenize(M, tokenize);+ if (records) return run_records(M, records, dump_ids);+ if (!serving) die("nothing to do: set SERVE=1, or pass --records / --tokenize");+ int slots = getenv("KV_SLOTS") ? atoi(getenv("KV_SLOTS")) : 1;+ if (slots < 1) slots = 1;+ char caps[256];+ snprintf(caps, sizeof(caps), "decide=1 chat=0 max_len=%d head_max_len=%d", M->max_len, M->head_max_len);+ LayaCtx ctx = {M, 0, NULL, 0};+ return decide_serve(stdin, stdout, laya_decide, &ctx, caps, rss_gb(), slots);+}diff --git a/c/laya_tiny/encoder/config.json b/c/laya_tiny/encoder/config.jsonnew file mode 100644index 00000000..da1de364--- /dev/null+++ b/c/laya_tiny/encoder/config.json@@ -0,0 +1,60 @@+{+ "architectures": [+ "ModernBertForMaskedLM"+ ],+ "attention_bias": false,+ "attention_dropout": 0.0,+ "bos_token_id": 449,+ "cls_token_id": 449,+ "eos_token_id": 450,+ "sep_token_id": 450,+ "pad_token_id": 451,+ "classifier_activation": "gelu",+ "classifier_bias": false,+ "classifier_dropout": 0.0,+ "classifier_pooling": "mean",+ "decoder_bias": true,+ "deterministic_flash_attn": false,+ "dtype": "float32",+ "embedding_dropout": 0.0,+ "global_attn_every_n_layers": 3,+ "gradient_checkpointing": false,+ "hidden_activation": "gelu",+ "hidden_size": 128,+ "initializer_cutoff_factor": 2.0,+ "initializer_range": 0.02,+ "intermediate_size": 160,+ "layer_norm_eps": 1e-05,+ "layer_types": [+ "full_attention",+ "sliding_attention",+ "sliding_attention",+ "full_attention"+ ],+ "local_attention": 16,+ "max_position_embeddings": 512,+ "mlp_bias": false,+ "mlp_dropout": 0.0,+ "model_type": "modernbert",+ "norm_bias": false,+ "norm_eps": 1e-05,+ "num_attention_heads": 4,+ "num_hidden_layers": 4,+ "position_embedding_type": "absolute",+ "repad_logits_with_grad": false,+ "rope_parameters": {+ "full_attention": {+ "rope_theta": 160000.0,+ "rope_type": "default"+ },+ "sliding_attention": {+ "rope_theta": 10000.0,+ "rope_type": "default"+ }+ },+ "sparse_pred_ignore_index": -100,+ "sparse_prediction": false,+ "tie_word_embeddings": true,+ "transformers_version": "5.0.0",+ "vocab_size": 640+}diff --git a/c/laya_tiny/ref.json b/c/laya_tiny/ref.jsonnew file mode 100644index 00000000..502f85fc--- /dev/null+++ b/c/laya_tiny/ref.json@@ -0,0 +1,49 @@+{+ "generator":"tools/make_laya_ref.py",+ "reference":{"laya":"0.3.24","torch":"2.12.1+cu130","transformers":"5.12.1"},+ "fixture_sha256":{"model.safetensors":"53627dd9a910415c5a82a72fe24c23972239e5b0779a4ab79f05c2dc594eb4b4","tokenizer/tokenizer.json":"7cebe02653d85b702171356b461aa206b5b044ccaa92cc13a7711b632ee7f602","encoder/config.json":"3f8253b69e5afb49f3e8b8e80d2c437e75515ae057eb57b7152ae389f7eb1683","rl_agent_config.json":"54b75ac1ccbd7396fc02a7e253909aefd64d6b99a963491b727d9a330ecb6549"},+ "tolerance":{"probs":2e-05,"logits":0.0002,"act_probability":0.0002},+ "tokenizer":[+ {"text":"","ids":[]},+ {"text":" ","ids":[34]},+ {"text":" ","ids":[444]},+ {"text":" ","ids":[441]},+ {"text":"\n\n","ids":[12,12]},+ {"text":"\t","ids":[11]},+ {"text":"a b c d e","ids":[99,444,100,443,101,442,102,438,283]},+ {"text":"Hello world","ids":[74,103,110,110,113,362,317]},+ {"text":"don't I'll we've they're she'd 'S 'T","ids":[102,281,41,118,411,41,110,110,329,41,289,260,123,41,278,263,259,41,102,34,41,85,34,41,86]},+ {"text":"2026-10-02 3.14 1000000 12345678901","ids":[52,364,47,51,50,47,331,410,48,51,54,34,51,50,50,50,50,50,50,34,51,52,53,54,55,56,57,58,59,50,51]},+ {"text":"café café Å Ω Å","ids":[101,99,104,197,171,274,99,104,197,171,34,197,135,34,208,171,34,197,135]},+ {"text":"日本語のテキスト","ids":[232,153,167,232,158,174,234,172,160,229,131,176,229,133,136,229,132,175,229,132,187,229,133,138]},+ {"text":"🙂👍🏽","ids":[242,161,155,132,242,161,147,143,242,161,145,191]},+ {"text":"[SEP] in [CLS] text [PAD][UNK]","ids":[450,297,34,449,360,122,118,34,451,448]},+ {"text":"|||IP_ADDRESS||| and |||EMAIL_ADDRESS|||, |||PHONE_NUMBER|||","ids":[0,308,34,445,46,34,446]},+ {"text":"<|endoftext|><|padding|>","ids":[447,1]},+ {"text":"[unused0]x[unused1]","ids":[453,122,454]},+ {"text":"tab\tand\r\nnewline\n","ids":[118,99,100,11,99,276,15,12,112,378,110,262,103,12]},+ {"text":" leading spaces","ids":[443,268,99,102,273,263,114,99,101,261]},+ {"text":"trailing spaces ","ids":[118,116,370,273,263,114,99,101,261,443]},+ {"text":"mixed ,.;:!? punctuation!!! ... ???","ids":[111,107,122,377,34,46,48,61,60,35,65,275,119,112,298,119,272,299,35,35,35,34,48,48,48,34,65,65,65]},+ {"text":"über straße Ελληνικά русский عربي","ids":[197,190,100,266,358,116,99,197,161,103,34,208,151,208,189,208,189,208,185,208,191,208,187,208,188,208,174,34,211,130,211,133,211,131,211,131,210,188,210,186,210,187,34,218,187,218,179,218,170,219,140]},+ {"text":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","ids":[122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122,122]},+ {"text":"{\"subject\": \"Duplicate charge\", \"body\": \"Please fix this\"}","ids":[125,36,405,100,108,103,298,330,306,70,408,274,106,313,105,103,363,306,375,330,306,366,321,107,122,328,36,127]},+ {"text":"level 0: not urgent","ids":[268,320,34,50,60,326,413]},+ {"text":" choice question: Which department?","ids":[274,106,390,350,60,34,89,106,264,106,352,114,313,118,301,65]}+ ],+ "cases":[+ {"name":"readme_ticket","state":{"from":"[email protected]","subject":"Duplicate charge on invoice #4411","body":"Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan."},"questions":{"department":{"type":"choice","instructions":"Which department should handle this request?","criteria":{"billing":"invoices, payments, refunds","technical":"bugs, outages, system errors","sales":"pricing, new contracts","other":"everything else"}},"urgency":{"type":"score","instructions":"How urgent is this request?","criteria":["not urgent","soon","critical deadline or blocking issue"]},"churn_risk":{"type":"noul","instructions":"Does the user threaten to cancel or leave?"},"refund_requested":{"type":"noul","instructions":"Does the user explicitly request a refund?"}},"answers":[{"id":"department","ids":[449,334,390,350,60,34,89,106,264,106,352,114,313,118,301,263,106,450,452,296,339,273,60,427,113,264,261,46,429,117,452,360,334,112,382,60,296,119,105,117,46,324,452,263,310,261,60,327,264,273,46,279,378,420,452,34,287,259,116,60,423,118,337,283,110,403,450,125,36,104,116,343,330,306,345,66,99,101,388,48,101,343,363,306,405,100,108,103,298,330,306,70,408,274,106,313,105,103,34,281,427,390,34,37,54,54,51,51,363,306,375,330,306,74,107,46,329,267,266,103,296,293,268,102,258,121,300,425,34,79,313,334,48,34,366,356,260,282,408,258,302,292,349,329,436,422,355,275,110,312,48,36,127,450],"markers":[18,30,42,54],"logits":[1.8797698020935059,1.7290606498718262,1.510589599609375,1.3890326023101807],"probs":[0.28690940141677856,0.2633657455444336,0.23262393474578857,0.21710097789764404],"temperature":1.7601518630981445,"act_probability":0.014895780943334103,"state_tokens":86,"state_dropped":0},{"id":"urgency","ids":[449,117,101,269,103,350,60,34,74,392,413,323,328,357,65,450,452,325,34,50,60,326,413,452,325,34,51,60,263,113,281,452,325,347,60,421,316,382,352,99,102,110,262,103,349,419,323,405,103,450,125,36,104,116,343,330,306,345,66,99,101,388,48,101,343,363,306,405,100,108,103,298,330,306,70,408,274,106,313,105,103,34,281,427,390,34,37,54,54,51,51,363,306,375,330,306,74,107,46,329,267,266,103,296,293,268,102,258,121,300,425,34,79,313,334,48,34,366,356,260,282,408,258,302,292,349,329,436,422,355,275,110,312,48,36,127,450],"markers":[16,23,31],"logits":[1.3007258176803589,1.6083192825317383,1.6973261833190918],"probs":[0.2738581597805023,0.3501648008823395,0.3759770393371582],"temperature":1.2514300346374512,"act_probability":0.5394694209098816,"state_tokens":86,"state_dropped":0},{"id":"churn_risk","ids":[449,112,277,110,350,60,34,365,260,414,433,402,270,361,422,349,307,371,65,450,452,321,310,403,60,428,46,260,432,282,342,326,426,452,258,116,119,103,60,415,261,46,260,432,426,117,450,125,36,104,116,343,330,306,345,66,99,101,388,48,101,343,363,306,405,100,108,103,298,330,306,70,408,274,106,313,105,103,34,281,427,390,34,37,54,54,51,51,363,306,375,330,306,74,107,46,329,267,266,103,296,293,268,102,258,121,300,425,34,79,313,334,48,34,366,356,260,282,408,258,302,292,349,329,436,422,355,275,110,312,48,36,127,450],"markers":[20,33],"logits":[0.6505882143974304,-0.047032684087753296],"probs":[0.5870369672775269,0.41296300292015076],"temperature":1.983399510383606,"act_probability":0.9283646941184998,"state_tokens":86,"state_dropped":0},{"id":"refund_requested","ids":[449,112,277,110,350,60,34,365,260,414,283,122,318,316,110,123,357,265,356,65,450,452,321,310,403,60,428,46,260,432,282,342,326,426,452,258,116,119,103,60,415,261,46,260,432,426,117,450,125,36,104,116,343,330,306,345,66,99,101,388,48,101,343,363,306,405,100,108,103,298,330,306,70,408,274,106,313,105,103,34,281,427,390,34,37,54,54,51,51,363,306,375,330,306,74,107,46,329,267,266,103,296,293,268,102,258,121,300,425,34,79,313,334,48,34,366,356,260,282,408,258,302,292,349,329,436,422,355,275,110,312,48,36,127,450],"markers":[21,34],"logits":[0.7286667823791504,0.06424430012702942],"probs":[0.5829734802246094,0.41702648997306824],"temperature":1.983399510383606,"act_probability":0.9144636988639832,"state_tokens":86,"state_dropped":0}],"predict":{"model":"laya-rl-agent","answers":{"department":{"type":"choice","choice":"billing","probabilities":{"billing":0.2869,"technical":0.2634,"sales":0.2326,"other":0.2171},"confidence":0.0042,"answer_confidence":0.2869,"action":{"act_probability":0.0149}},"urgency":{"type":"score","score":1.1021,"legend":{"0":"not urgent","1":"soon","2":"critical deadline or blocking issue"},"probabilities":{"0":0.2739,"1":0.3502,"2":0.376},"confidence":0.0079,"answer_confidence":0.376,"action":{"act_probability":0.5395}},"churn_risk":{"type":"noul","noul":0.413,"confidence":0.587,"answer_confidence":0.587,"action":{"act_probability":0.9284}},"refund_requested":{"type":"noul","noul":0.417,"confidence":0.583,"answer_confidence":0.583,"action":{"act_probability":0.9145}}},"usage":{"input_tokens":560,"output_tokens":0,"state_tokens":86,"state_tokens_dropped":0,"truncated":false,"truncated_questions":[]}}},+ {"name":"plain_two_options","state":"The application crashes every time I open the settings page.","questions":{"is_bug":{"type":"choice","instructions":"Is this a bug report or a feature request?","criteria":{"bug":"something is broken","feature":"something new is wanted"}},"blocking":{"type":"noul","instructions":"Is the user blocked?","criteria":{"true":"the user cannot work at all","false":"the user can still work"}},"only_true":{"type":"noul","instructions":"Does it mention settings?","criteria":{"true":"the settings page is named"}}},"answers":[{"id":"is_bug","ids":[449,334,390,350,60,411,117,328,265,296,119,105,291,114,269,118,349,265,321,103,272,119,278,357,65,450,452,296,119,105,60,263,343,335,337,323,296,116,113,109,270,452,321,103,272,119,278,60,263,343,335,337,279,378,323,267,312,118,377,450,309,265,114,318,272,299,421,284,259,117,423,258,107,388,411,348,114,270,260,263,335,118,273,117,275,369,103,48,450],"markers":[26,41],"logits":[1.9618617296218872,1.8516799211502075],"probs":[0.5144452452659607,0.4855547845363617],"temperature":1.9063563346862793,"act_probability":0.0003925680066458881,"state_tokens":28,"state_dropped":0},{"id":"blocking","ids":[449,112,277,110,350,60,411,117,260,414,296,386,385,65,450,452,321,310,403,60,260,414,351,358,339,362,109,452,258,116,119,103,60,260,414,351,112,287,362,109,265,118,265,110,110,450,309,265,114,318,272,299,421,284,259,117,423,258,107,388,411,348,114,270,260,263,335,118,273,117,275,369,103,48,450],"markers":[15,27],"logits":[0.9451121091842651,0.9406300783157349],"probs":[0.5005649328231812,0.49943506717681885],"temperature":1.983399510383606,"act_probability":0.5351545214653015,"state_tokens":28,"state_dropped":0},{"id":"only_true","ids":[449,112,277,110,350,60,34,365,34,316,34,301,299,263,335,118,273,117,65,450,452,321,310,403,60,428,46,260,432,282,342,326,426,452,258,116,119,103,60,260,263,335,118,273,117,275,369,103,323,279,311,377,450,309,265,114,318,272,299,421,284,259,117,423,258,107,388,411,348,114,270,260,263,335,118,273,117,275,369,103,48,450],"markers":[20,33],"logits":[0.6375879645347595,0.09742391109466553],"probs":[0.5676679015159607,0.4323320984840393],"temperature":1.983399510383606,"act_probability":0.8332017064094543,"state_tokens":28,"state_dropped":0}],"predict":{"model":"laya-rl-agent","answers":{"is_bug":{"type":"choice","choice":"bug","probabilities":{"bug":0.5144,"feature":0.4856},"confidence":0.0006,"answer_confidence":0.5144,"action":{"act_probability":0.0004}},"blocking":{"type":"noul","noul":0.4994,"confidence":0.5006,"answer_confidence":0.5006,"action":{"act_probability":0.5352}},"only_true":{"type":"noul","noul":0.4323,"confidence":0.5677,"answer_confidence":0.5677,"action":{"act_probability":0.8332}}},"usage":{"input_tokens":246,"output_tokens":0,"state_tokens":28,"state_tokens_dropped":0,"truncated":false,"truncated_questions":[]}}},+ {"name":"choice_seven_structured","state":"We would like to schedule a demo of the product for our team next week.","questions":{"intent":{"type":"choice","instructions":"What does the sender want?","criteria":{"demo":"a product demonstration","refund":null,"cancel":"","pricing":{"desc":"price questions","examples":["cost","plan"]},"support":3,"partnership":["co-marketing","reselling"],"other":"anything else"}}},"answers":[{"id":"intent","ids":[449,334,390,350,60,34,89,106,272,282,342,260,263,270,102,266,267,312,118,65,450,452,352,111,113,60,265,452,356,452,422,452,327,264,273,60,34,452,359,399,269,118,60,452,275,313,118,112,380,452,34,287,259,116,60,450,89,103,267,398,354,361,431,259,102,119,268,265,352,111,113,348,104,260,430,425,355,434,279,379,118,329,103,109,48,450],"markers":[21,27,29,31,37,43,49],"logits":[1.3486415147781372,1.0380945205688477,1.065248966217041,1.2871294021606445,1.3605328798294067,1.2659051418304443,1.3090858459472656],"probs":[0.15821801126003265,0.11598161607980728,0.11917413771152496,0.14877910912036896,0.16011062264442444,0.14565470814704895,0.15208183228969574],"temperature":1.0000158548355103,"act_probability":0.0021321214735507965,"state_tokens":29,"state_dropped":0}],"predict":{"model":"laya-rl-agent","answers":{"intent":{"type":"choice","choice":"support","probabilities":{"demo":0.1582,"refund":0.116,"cancel":0.1192,"pricing":0.1488,"support":0.1601,"partnership":0.1457,"other":0.1521},"confidence":0.0036,"answer_confidence":0.1601,"action":{"act_probability":0.0021}}},"usage":{"input_tokens":86,"output_tokens":0,"state_tokens":29,"state_tokens_dropped":0,"truncated":false,"truncated_questions":[]}}},+ {"name":"choice_twelve_labels","state":"I love this product, it works great and the support team was very helpful.","questions":{"emotion":{"type":"choice","instructions":"Which emotion does the text express?","criteria":{"joy":null,"anger":null,"sadness":null,"fear":null,"surprise":null,"disgust":null,"trust":null,"anticipation":null,"neutral":null,"confusion":null,"gratitude":null,"pride":null}}},"answers":[{"id":"emotion","ids":[449,334,390,350,60,34,89,106,264,106,353,287,299,282,342,260,360,450,452,34,108,113,452,417,105,266,452,263,99,102,452,321,103,313,452,263,319,114,452,282,271,105,452,258,116,304,452,417,118,264,452,279,103,119,452,274,281,104,452,34,105,116,452,327,107,102,450,75,290,113,289,328,430,46,34,316,437,34,105,402,308,260,359,399,269,118,434,435,34,346,34,259,110,114,315,110,48,450],"markers":[18,22,26,30,34,38,42,46,50,54,58,62],"logits":[1.4752514362335205,2.2580246925354004,2.0820565223693848,1.692331075668335,2.339540958404541,1.5242490768432617,1.4365266561508179,0.9649323225021362,0.8330012559890747,1.23215913772583,1.101522445678711,1.613863468170166],"probs":[0.04672723263502121,0.22360335290431976,0.1572657972574234,0.07213117927312851,0.2631978988647461,0.05153816193342209,0.043244827538728714,0.016838854178786278,0.012933577410876751,0.028735768049955368,0.02212856337428093,0.06165480986237526],"temperature":0.5,"act_probability":0.006478143390268087,"state_tokens":30,"state_dropped":0}],"predict":{"model":"laya-rl-agent","answers":{"emotion":{"type":"choice","choice":"surprise","probabilities":{"joy":0.0467,"anger":0.2236,"sadness":0.1573,"fear":0.0721,"surprise":0.2632,"disgust":0.0515,"trust":0.0432,"anticipation":0.0168,"neutral":0.0129,"confusion":0.0287,"gratitude":0.0221,"pride":0.0617},"confidence":0.1622,"answer_confidence":0.2632,"action":{"act_probability":0.0065}}},"usage":{"input_tokens":98,"output_tokens":0,"state_tokens":30,"state_tokens_dropped":0,"truncated":false,"truncated_questions":[]}}},+ {"name":"scores","state":"This is the worst service I have ever used, nothing works and nobody answers.","questions":{"sentiment":{"type":"score","instructions":"What is the sender's tone?","criteria":["angry","negative","neutral","positive","delighted"]},"six_levels":{"type":"score","instructions":"How severe is the impact?","criteria":["none","minor","moderate","major","critical","catastrophic"]},"two_levels":{"type":"score","instructions":"Is any action needed?","criteria":["no action","action needed"]},"numeric_levels":{"type":"score","instructions":"Rate the urgency.","criteria":[1,2,3]}},"answers":[{"id":"sentiment","ids":[449,117,101,269,103,350,60,34,89,106,272,323,260,263,270,102,266,41,117,258,450,452,325,34,50,60,417,105,116,123,452,325,34,51,60,279,103,105,272,452,325,347,60,279,103,119,118,116,452,325,410,60,275,113,117,316,107,452,325,34,54,60,352,110,107,105,450,367,323,260,362,404,263,266,120,300,411,322,371,283,305,34,304,377,46,326,337,437,308,428,375,417,406,380,48,450],"markers":[21,30,39,48,57],"logits":[1.8515902757644653,1.7788968086242676,1.7535442113876343,1.3500280380249023,1.4620959758758545],"probs":[0.23416250944137573,0.22094792127609253,0.21651679277420044,0.15683968365192413,0.17153306305408478],"temperature":1.2514300346374512,"act_probability":0.647968590259552,"state_tokens":28,"state_dropped":0},{"id":"six_levels","ids":[449,117,101,269,103,350,60,34,74,392,263,103,305,103,323,260,34,450,452,325,34,50,60,279,281,103,452,325,34,51,60,34,111,262,452,325,347,60,34,111,302,266,452,325,410,60,34,111,99,108,452,325,34,54,60,421,316,382,452,325,34,55,60,274,272,284,450,367,323,260,362,404,263,266,120,300,411,322,371,283,305,34,304,377,46,326,337,437,308,428,375,417,406,380,48,450],"markers":[18,26,34,42,50,58],"logits":[1.3440673351287842,1.3813637495040894,1.1715118885040283,1.2259613275527954,1.7430481910705566,2.1922767162323],"probs":[0.13973473012447357,0.14396193623542786,0.121736541390419,0.12715019285678864,0.19220563769340515,0.275210976600647],"temperature":1.2514300346374512,"act_probability":0.43700361251831055,"state_tokens":28,"state_dropped":0},{"id":"two_levels","ids":[449,117,101,269,103,350,60,411,117,417,123,265,298,299,279,103,377,377,65,450,452,325,34,50,60,428,265,298,299,452,325,34,51,60,265,298,299,279,103,377,377,450,367,323,260,362,404,263,266,120,300,411,322,371,283,305,34,304,377,46,326,337,437,308,428,375,417,406,380,48,450],"markers":[20,29],"logits":[2.185020685195923,2.055086851119995],"probs":[0.5259337425231934,0.47406625747680664],"temperature":1.2514300346374512,"act_probability":0.31258225440979004,"state_tokens":28,"state_dropped":0},{"id":"numeric_levels","ids":[449,117,101,269,103,350,60,34,84,314,260,34,319,105,270,101,123,48,450,452,325,34,50,60,34,51,452,325,34,51,60,347,452,325,347,60,410,450,367,323,260,362,404,263,266,120,300,411,322,371,283,305,34,304,377,46,326,337,437,308,428,375,417,406,380,48,450],"markers":[19,26,32],"logits":[1.5683256387710571,1.6159822940826416,1.9428691864013672],"probs":[0.2951837480068207,0.30664166808128357,0.39817455410957336],"temperature":1.2514300346374512,"act_probability":0.44181326031684875,"state_tokens":28,"state_dropped":0}],"predict":{"model":"laya-rl-agent","answers":{"sentiment":{"type":"score","score":1.8106,"legend":{"0":"angry","1":"negative","2":"neutral","3":"positive","4":"delighted"},"probabilities":{"0":0.2342,"1":0.2209,"2":0.2165,"3":0.1568,"4":0.1715},"confidence":0.0072,"answer_confidence":0.2342,"action":{"act_probability":0.648}},"six_levels":{"type":"score","score":2.9138,"legend":{"0":"none","1":"minor","2":"moderate","3":"major","4":"critical","5":"catastrophic"},"probabilities":{"0":0.1397,"1":0.144,"2":0.1217,"3":0.1272,"4":0.1922,"5":0.2752},"confidence":0.0263,"answer_confidence":0.2752,"action":{"act_probability":0.437}},"two_levels":{"type":"score","score":0.4741,"legend":{"0":"no action","1":"action needed"},"probabilities":{"0":0.5259,"1":0.4741},"confidence":0.0019,"answer_confidence":0.5259,"action":{"act_probability":0.3126}},"numeric_levels":{"type":"score","score":1.103,"legend":{"0":"1","1":"2","2":"3"},"probabilities":{"0":0.2952,"1":0.3066,"2":0.3982},"confidence":0.0085,"answer_confidence":0.3982,"action":{"act_probability":0.4418}}},"usage":{"input_tokens":330,"output_tokens":0,"state_tokens":28,"state_tokens_dropped":0,"truncated":false,"truncated_questions":[]}}},+ {"name":"long_state_truncated","state":{"subject":"Invoice problem again","body":"Hello team, I am writing again about the invoice we received on the first of the month. The amount is twice what we agreed in the contract and the payment was already taken from our card. We asked for a refund last week and nobody answered. Our finance department needs the corrected invoice before Friday or we will have to escalate this to our lawyers and stop using the service. Please also check the other invoices of this year, because we think the same mistake happened in March and in April. Thank you, the accounts team."},"questions":{"department":{"type":"choice","instructions":"Which department should handle this request?","criteria":{"billing":"invoices, payments, refunds","technical":"bugs, outages, system errors","sales":"pricing, new contracts","other":"everything else"}},"legal_threat":{"type":"noul","instructions":"Does the sender threaten legal action?"}},"answers":[{"id":"department","ids":[449,334,390,350,60,34,89,106,264,106,352,114,313,118,301,263,106,450,452,296,339,273,60,427,113,264,261,46,429,117,452,360,334,112,382,60,296,119,105,117,46,324,452,263,310,261,60,327,264,273,46,279,378,420,452,34,287,259,116,60,423,118,337,283,110,403,450,125,36,405,100,108,103,298,330,306,75,112,120,390,327,113,100,268,111,265,105,99,262,363,306,375,330,306,74,103,110,110,113,434,46,411,265,111,267,116,316,273,265,105,99,262,265,100,277,118,260,427,390,329,291,333,107,289,102,34,281,260,321,107,116,404,348,104,260,34,111,395,106,48,34,309,265,111,277,112,118,323,258,121,300,267,106,272,329,265,105,278,377,450],"markers":[18,30,42,54],"logits":[2.1466763019561768,2.003511428833008,1.8001492023468018,1.445472002029419],"probs":[0.2928612232208252,0.2699838876724243,0.24052534997463226,0.196629598736763],"temperature":1.7601518630981445,"act_probability":0.014024795964360237,"state_tokens":290,"state_dropped":198},{"id":"legal_threat","ids":[449,112,277,110,350,60,34,365,260,263,270,102,266,433,402,270,307,105,310,265,298,299,65,450,452,321,310,403,60,428,46,260,432,282,342,326,426,452,258,116,119,103,60,415,261,46,260,432,426,117,450,125,36,405,100,108,103,298,330,306,75,112,120,390,327,113,100,268,111,265,105,99,262,363,306,375,330,306,74,103,110,110,113,434,46,411,265,111,267,116,316,273,265,105,99,262,265,100,277,118,260,427,390,329,291,333,107,289,102,34,281,260,321,107,116,404,348,104,260,34,111,395,106,48,34,309,265,111,277,112,118,323,258,121,300,267,106,272,329,265,105,278,377,297,260,420,308,260,429,435,265,110,278,99,102,123,258,99,109,450],"markers":[24,37],"logits":[0.7672969698905945,0.209745854139328],"probs":[0.5698180198669434,0.43018195033073425],"temperature":1.983399510383606,"act_probability":0.9386257529258728,"state_tokens":290,"state_dropped":182}],"predict":{"model":"laya-rl-agent","answers":{"department":{"type":"choice","choice":"billing","probabilities":{"billing":0.2929,"technical":0.27,"sales":0.2405,"other":0.1966},"confidence":0.0076,"answer_confidence":0.2929,"action":{"act_probability":0.014}},"legal_threat":{"type":"noul","noul":0.4302,"confidence":0.5698,"answer_confidence":0.5698,"action":{"act_probability":0.9386}}},"usage":{"input_tokens":320,"output_tokens":0,"state_tokens":290,"state_tokens_dropped":198,"truncated":true,"truncated_questions":["department","legal_threat"]}}},+ {"name":"conversation_truncated_left","state":[{"role":"user","content":"Hi, I cannot log in to my account since yesterday."},{"role":"assistant","content":"Sorry about that. Did you try to reset the password?"},{"role":"user","content":"Yes, twice, and the reset email never arrives."},{"role":"assistant","content":"Let me check the email address on the account."},{"role":"user","content":"It is the same as always. This is really urgent, I have a payment due today and I need the invoices from the portal."},{"role":"assistant","content":"I understand. I am escalating it now."},{"role":"user","content":"If it is not fixed today I will cancel the subscription."}],"questions":{"churn":{"type":"noul","instructions":"Does the user threaten to cancel?"},"topic":{"type":"choice","instructions":"What is the conversation about?","criteria":{"login":"access to the account","billing":"payments","other":"anything else"}}},"answers":[{"id":"churn","ids":[449,112,277,110,350,60,34,365,260,414,433,402,270,361,422,65,450,452,321,310,403,60,428,46,260,432,282,342,326,426,452,258,116,119,103,60,415,261,46,260,432,426,117,450,308,411,279,103,377,260,427,113,264,261,321,116,343,260,275,269,118,310,48,36,127,46,34,125,36,116,113,268,330,306,284,117,271,118,312,118,363,306,101,395,280,330,306,75,34,119,276,380,118,99,276,48,411,265,111,34,261,101,310,272,273,34,316,279,392,48,36,127,46,34,125,36,116,113,268,330,306,345,363,306,101,395,280,330,306,75,104,34,316,323,326,321,107,122,377,258,302,292,411,436,422,260,359,100,117,101,116,107,114,118,299,48,36,127,95,450],"markers":[17,30],"logits":[0.6236253976821899,-0.05764560401439667],"probs":[0.5850372314453125,0.4149627983570099],"temperature":1.983399510383606,"act_probability":0.9103093147277832,"state_tokens":360,"state_dropped":245},{"id":"topic","ids":[449,334,390,350,60,34,89,106,272,323,260,274,281,305,117,272,299,265,100,277,118,65,450,452,290,389,262,60,265,101,101,261,117,361,260,265,101,101,277,112,118,452,296,339,273,60,429,117,452,34,287,259,116,60,417,123,118,337,283,110,403,450,48,36,127,46,34,125,36,116,113,268,330,306,284,117,271,118,312,118,363,306,101,395,280,330,306,75,34,119,276,380,118,99,276,48,411,265,111,34,261,101,310,272,273,34,316,279,392,48,36,127,46,34,125,36,116,113,268,330,306,345,363,306,101,395,280,330,306,75,104,34,316,323,326,321,107,122,377,258,302,292,411,436,422,260,359,100,117,101,116,107,114,118,299,48,36,127,95,450],"markers":[23,41,48],"logits":[2.068683624267578,2.1091132164001465,2.00535249710083],"probs":[0.3346838653087616,0.3424603343009949,0.3228558599948883],"temperature":1.7601518630981445,"act_probability":0.008869393728673458,"state_tokens":360,"state_dropped":263}],"predict":{"model":"laya-rl-agent","answers":{"churn":{"type":"noul","noul":0.415,"confidence":0.585,"answer_confidence":0.585,"action":{"act_probability":0.9103}},"topic":{"type":"choice","choice":"billing","probabilities":{"login":0.3347,"billing":0.3425,"other":0.3229},"confidence":0.0003,"answer_confidence":0.3425,"action":{"act_probability":0.0089}}},"usage":{"input_tokens":320,"output_tokens":0,"state_tokens":360,"state_tokens_dropped":263,"truncated":true,"truncated_questions":["churn","topic"]}}},+ {"name":"json_instructions_and_state","state":{"ticket":{"id":4411,"amount":129.5,"currency":"EUR","paid":true,"refund":null,"ratio":1e-05,"big":1e+21,"tags":["billing","urgent"],"nested":{"a":[1,2.0,{"b":"c"}]}}},"questions":{"action":{"type":"choice","instructions":{"task":"pick the next action","context":"support desk","steps":["read","decide"]},"criteria":{"refund":"give the money back","escalate":"a human decides","close":"nothing to do"}},"paid":{"type":"noul","instructions":["Was","the","ticket","paid?"]}},"answers":[{"id":"action","ids":[449,334,390,350,60,34,125,36,118,284,109,330,306,114,264,109,260,279,379,118,265,450,452,356,60,34,105,107,289,260,34,111,281,103,123,296,99,101,109,452,34,261,101,310,314,60,265,322,407,312,352,101,107,102,261,452,274,110,113,403,60,326,337,361,282,113,450,125,36,118,264,294,118,330,34,125,36,107,102,330,34,54,54,51,51,46,306,311,277,112,118,330,34,51,52,59,48,55,46,306,101,319,116,270,101,123,330,306,71,87,84,363,306,114,99,107,102,330,258,116,119,103,46,306,278,336,330,279,119,110,110,46,306,116,272,107,113,330,34,51,103,47,50,55,46,306,100,107,105,330,34,51,103,45,52,51,46,306,118,450],"markers":[22,39,55],"logits":[1.8610424995422363,2.076181173324585,1.8813223838806152],"probs":[0.3183092176914215,0.35969293117523193,0.32199788093566895],"temperature":1.7601518630981445,"act_probability":0.0022632686886936426,"state_tokens":138,"state_dropped":46},{"id":"paid","ids":[449,112,277,110,350,60,34,93,36,89,284,363,306,118,259,363,306,118,264,294,118,363,306,114,99,107,102,65,36,95,450,452,321,310,403,60,428,46,260,432,282,342,326,426,452,258,116,119,103,60,415,261,46,260,432,426,117,450,125,36,118,264,294,118,330,34,125,36,107,102,330,34,54,54,51,51,46,306,311,277,112,118,330,34,51,52,59,48,55,46,306,101,319,116,270,101,123,330,306,71,87,84,363,306,114,99,107,102,330,258,116,119,103,46,306,278,336,330,279,119,110,110,46,306,116,272,107,113,330,34,51,103,47,50,55,46,306,100,107,105,330,34,51,103,45,52,51,46,306,118,369,117,330,34,93,36,100,339,273,450],"markers":[31,44],"logits":[1.6786537170410156,-0.037607043981552124],"probs":[0.7037694454193115,0.2962305545806885],"temperature":1.983399510383606,"act_probability":0.8384866714477539,"state_tokens":138,"state_dropped":37}],"predict":{"model":"laya-rl-agent","answers":{"action":{"type":"choice","choice":"escalate","probabilities":{"refund":0.3183,"escalate":0.3597,"close":0.322},"confidence":0.0014,"answer_confidence":0.3597,"action":{"act_probability":0.0023}},"paid":{"type":"noul","noul":0.2962,"confidence":0.7038,"answer_confidence":0.7038,"action":{"act_probability":0.8385}}},"usage":{"input_tokens":320,"output_tokens":0,"state_tokens":138,"state_tokens_dropped":46,"truncated":true,"truncated_questions":["action","paid"]}}},+ {"name":"special_text","state":"Order [MASK] arrived broken, see\tphoto.\r\n\r\nThe café [SEP] said |||IP_ADDRESS||| is blocked <|endoftext|> and [unused0] support said 'we're sorry'.","questions":{"damaged":{"type":"noul","instructions":"Was the item [MASK] damaged?"},"channel":{"type":"choice","instructions":"Which channel should answer?","criteria":{"email":"write an email [MASK] back","phone":"call\tthe customer","none":null}}},"answers":[{"id":"damaged","ids":[449,112,277,110,350,60,34,89,284,260,34,316,103,111,443,102,311,369,377,65,450,452,321,310,403,60,428,46,260,432,282,342,326,426,452,258,116,119,103,60,415,261,46,260,432,426,117,450,81,116,102,266,443,313,116,107,289,102,444,100,116,113,109,270,46,443,403,103,11,114,106,287,113,48,15,12,15,12,309,274,99,104,197,171,34,450,263,99,107,102,34,0,323,296,386,385,34,447,308,34,453,359,399,269,118,263,99,107,102,34,41,121,103,41,278,263,269,116,123,41,48,450],"markers":[21,34],"logits":[1.3261137008666992,0.4389176666736603],"probs":[0.6099996566772461,0.3900003433227539],"temperature":1.983399510383606,"act_probability":0.9083874821662903,"state_tokens":73,"state_dropped":0},{"id":"channel","ids":[449,334,390,350,60,34,89,106,264,106,274,106,312,112,103,110,444,117,106,398,443,312,406,266,65,450,452,424,60,267,116,316,103,417,424,443,100,99,101,109,452,275,106,281,103,60,274,310,110,11,118,259,274,304,118,343,266,452,279,281,103,450,81,116,102,266,443,313,116,107,289,102,444,100,116,113,109,270,46,443,403,103,11,114,106,287,113,48,15,12,15,12,309,274,99,104,197,171,34,450,263,99,107,102,34,0,323,296,386,385,34,447,308,34,453,359,399,269,118,263,99,107,102,34,41,121,103,41,278,263,269,116,123,41,48,450],"markers":[26,40,57],"logits":[1.6699668169021606,2.493666410446167,1.6683423519134521],"probs":[0.27810004353523254,0.4440564513206482,0.27784350514411926],"temperature":1.7601518630981445,"act_probability":0.0030960990116000175,"state_tokens":73,"state_dropped":0}],"predict":{"model":"laya-rl-agent","answers":{"damaged":{"type":"noul","noul":0.39,"confidence":0.61,"answer_confidence":0.61,"action":{"act_probability":0.9084}},"channel":{"type":"choice","choice":"phone","probabilities":{"email":0.2781,"phone":0.4441,"none":0.2778},"confidence":0.024,"answer_confidence":0.4441,"action":{"act_probability":0.0031}}},"usage":{"input_tokens":258,"output_tokens":0,"state_tokens":73,"state_tokens_dropped":0,"truncated":false,"truncated_questions":[]}}},+ {"name":"long_option","state":"The meeting notes say that the contract will be renewed if the price stays the same.","questions":{"renewal":{"type":"choice","instructions":"Will the contract be renewed?","criteria":{"yes":"the contract will be renewed under the same terms under the same terms under the same terms under the same terms under the same terms under the same terms under the same terms under the same terms under the same terms under the same terms under the same terms under the same terms ","no":"the contract ends","unclear":"the notes do not say"}}},"answers":[{"id":"renewal","ids":[449,334,390,350,60,34,89,339,260,420,296,103,412,270,378,377,65,450,452,415,261,60,260,420,436,296,103,412,270,378,377,34,119,276,452,428,60,260,420,34,270,102,117,452,34,119,112,101,268,313,60,260,326,261,282,113,326,263,292,450,309,34,388,335,273,326,261,263,292,433,272,260,420,436,296,103,412,270,378,377,34,107,104,260,327,300,358,292,117,260,263,311,103,48,450],"markers":[18,34,43],"logits":[1.6784346103668213,1.5342192649841309,1.520702600479126],"probs":[0.352656751871109,0.3249143958091736,0.3224288523197174],"temperature":1.7601518630981445,"act_probability":0.010438842698931694,"state_tokens":34,"state_dropped":0}],"predict":{"model":"laya-rl-agent","answers":{"renewal":{"type":"choice","choice":"yes","probabilities":{"yes":0.3527,"no":0.3249,"unclear":0.3224},"confidence":0.0008,"answer_confidence":0.3527,"action":{"act_probability":0.0104}}},"usage":{"input_tokens":95,"output_tokens":0,"state_tokens":34,"state_tokens_dropped":0,"truncated":false,"truncated_questions":[]}}},+ {"name":"many_options_shrunk","state":"Security incident: a phishing email asked employees to reset their passwords.","questions":{"category":{"type":"choice","instructions":"Classify the incident into exactly one of the categories below, considering the attack vector, the target and the likely impact on the organisation and its customers.","criteria":{"cat00":"category number 0 of the incident taxonomy","cat01":"category number 1 of the incident taxonomy","cat02":"category number 2 of the incident taxonomy","cat03":"category number 3 of the incident taxonomy","cat04":"category number 4 of the incident taxonomy","cat05":"category number 5 of the incident taxonomy","cat06":"category number 6 of the incident taxonomy","cat07":"category number 7 of the incident taxonomy","cat08":"category number 8 of the incident taxonomy","cat09":"category number 9 of the incident taxonomy","cat10":"category number 10 of the incident taxonomy","cat11":"category number 11 of the incident taxonomy","cat12":"category number 12 of the incident taxonomy","cat13":"category number 13 of the incident taxonomy","cat14":"category number 14 of the incident taxonomy","cat15":"category number 15 of the incident taxonomy","cat16":"category number 16 of the incident taxonomy","cat17":"category number 17 of the incident taxonomy","cat18":"category number 18 of the incident taxonomy","cat19":"category number 19 of the incident taxonomy"}}},"answers":[{"id":"category","ids":[449,334,390,350,60,34,69,110,284,450,452,274,272,50,452,274,272,50,452,274,272,331,452,274,272,50,452,274,272,50,452,274,272,50,452,274,272,50,452,274,272,50,452,274,272,50,452,274,272,50,452,274,272,51,452,274,272,51,452,274,272,51,452,274,272,51,452,274,272,51,452,274,272,51,452,274,272,51,452,274,272,51,452,274,272,51,452,274,272,51,450,85,103,101,319,316,123,297,101,107,102,280,60,265,275,286,337,424,418,385,353,303,113,123,103,261,361,412,261,335,260,107,116,275,284,406,269,102,117,48,450],"markers":[10,14,18,22,26,30,34,38,42,46,50,54,58,62,66,70,74,78,82,86],"logits":[1.6276103258132935,1.271726131439209,1.158445119857788,1.2097952365875244,1.2095370292663574,1.2767925262451172,1.3562877178192139,1.3325998783111572,1.2972207069396973,1.2697169780731201,1.1271255016326904,1.0182697772979736,0.9797287583351135,0.9335014820098877,1.0000560283660889,1.0177512168884277,1.023991584777832,1.1210556030273438,1.1499793529510498,1.5215578079223633],"probs":[0.11101242899894714,0.05448218435049057,0.04343700408935547,0.04813511669635773,0.04811026155948639,0.05503704771399498,0.06452140212059021,0.06153593212366104,0.05733222886919975,0.05426369607448578,0.04079960659146309,0.032817475497722626,0.030382871627807617,0.027699779719114304,0.03164352476596832,0.03278345242142677,0.03319517895579338,0.04030730202794075,0.042707737535238266,0.08979570120573044],"temperature":0.5,"act_probability":0.003958355635404587,"state_tokens":39,"state_dropped":0}],"predict":{"model":"laya-rl-agent","answers":{"category":{"type":"choice","choice":"cat00","probabilities":{"cat00":0.111,"cat01":0.0545,"cat02":0.0434,"cat03":0.0481,"cat04":0.0481,"cat05":0.055,"cat06":0.0645,"cat07":0.0615,"cat08":0.0573,"cat09":0.0543,"cat10":0.0408,"cat11":0.0328,"cat12":0.0304,"cat13":0.0277,"cat14":0.0316,"cat15":0.0328,"cat16":0.0332,"cat17":0.0403,"cat18":0.0427,"cat19":0.0898},"confidence":0.024,"answer_confidence":0.111,"action":{"act_probability":0.004}}},"usage":{"input_tokens":131,"output_tokens":0,"state_tokens":39,"state_tokens_dropped":0,"truncated":false,"truncated_questions":[],"options":{"category":{"total":20,"distinct":3,"tokens_per_option":4}}}}},+ {"name":"too_many_options","state":"Too many options for the head budget.","questions":{"bucket":{"type":"choice","instructions":"Pick one.","criteria":{"option0":"the description of option 0","option1":"the description of option 1","option2":"the description of option 2","option3":"the description of option 3","option4":"the description of option 4","option5":"the description of option 5","option6":"the description of option 6","option7":"the description of option 7","option8":"the description of option 8","option9":"the description of option 9","option10":"the description of option 10","option11":"the description of option 11","option12":"the description of option 12","option13":"the description of option 13","option14":"the description of option 14","option15":"the description of option 15","option16":"the description of option 16","option17":"the description of option 17","option18":"the description of option 18","option19":"the description of option 19","option20":"the description of option 20","option21":"the description of option 21","option22":"the description of option 22","option23":"the description of option 23","option24":"the description of option 24","option25":"the description of option 25","option26":"the description of option 26","option27":"the description of option 27","option28":"the description of option 28","option29":"the description of option 29","option30":"the description of option 30","option31":"the description of option 31","option32":"the description of option 32","option33":"the description of option 33","option34":"the description of option 34","option35":"the description of option 35","option36":"the description of option 36","option37":"the description of option 37","option38":"the description of option 38","option39":"the description of option 39"}}},"error":"question 'bucket': only 38 of its 40 option markers fit in max_len=160 with head_max_len=64 spent on the question; lower head_max_len, raise max_len, or use fewer options"},+ {"name":"unicode","state":"Buongiorno, la fattura è sbagliata. Grüße aus München, ευχαριστώ. 我们的账户被锁了 🙂👍🏽 Å Ω","questions":{"language":{"type":"choice","instructions":"Which language comes first?","criteria":{"italian":"italiano","german":"deutsch","chinese":"中文"}},"polite":{"type":"noul","instructions":"Is the message polite?"}},"answers":[{"id":"language","ids":[449,334,390,350,60,34,89,106,264,106,290,312,105,119,369,103,274,343,261,321,107,116,404,65,450,452,34,316,310,107,312,60,34,316,310,107,312,113,452,34,105,266,111,312,60,352,119,118,117,334,452,274,106,262,261,103,60,34,230,186,175,232,152,137,450,68,119,281,105,107,269,112,113,46,290,99,321,272,118,319,99,34,197,170,263,100,369,110,107,272,99,48,34,73,116,197,190,197,161,103,265,304,34,79,197,190,112,101,259,112,46,34,208,183,209,135,209,137,208,179,209,131,208,187,209,133,209,134,209,144,48,34,232,138,147,230,189,174,233,156,134,234,182,168,232,138,185,234,164,173,235,150,131,230,188,136,34,242,161,450],"markers":[25,38,50],"logits":[1.644952416419983,0.7264605164527893,1.1558306217193604],"probs":[0.42538365721702576,0.25243809819221497,0.32217833399772644],"temperature":1.7601518630981445,"act_probability":0.00099587079603225,"state_tokens":110,"state_dropped":16},{"id":"polite","ids":[449,112,277,110,350,60,411,117,260,34,111,261,117,369,103,275,113,110,316,103,65,450,452,321,310,403,60,428,46,260,432,282,342,326,426,452,258,116,119,103,60,415,261,46,260,432,426,117,450,68,119,281,105,107,269,112,113,46,290,99,321,272,118,319,99,34,197,170,263,100,369,110,107,272,99,48,34,73,116,197,190,197,161,103,265,304,34,79,197,190,112,101,259,112,46,34,208,183,209,135,209,137,208,179,209,131,208,187,209,133,209,134,209,144,48,34,232,138,147,230,189,174,233,156,134,234,182,168,232,138,185,234,164,173,235,150,131,230,188,136,34,242,161,155,132,242,161,147,143,242,161,145,191,34,197,135,34,208,171,450],"markers":[22,35],"logits":[0.4593088626861572,-0.0906318873167038],"probs":[0.5688772201538086,0.4311228096485138],"temperature":1.983399510383606,"act_probability":0.7027542591094971,"state_tokens":110,"state_dropped":0}],"predict":{"model":"laya-rl-agent","answers":{"language":{"type":"choice","choice":"italian","probabilities":{"italian":0.4254,"german":0.2524,"chinese":0.3222},"confidence":0.0206,"answer_confidence":0.4254,"action":{"act_probability":0.001}},"polite":{"type":"noul","noul":0.4311,"confidence":0.5689,"answer_confidence":0.5689,"action":{"act_probability":0.7028}}},"usage":{"input_tokens":320,"output_tokens":0,"state_tokens":110,"state_tokens_dropped":16,"truncated":true,"truncated_questions":["language"]}}}+ ]+}diff --git a/c/laya_tiny/rl_agent_config.json b/c/laya_tiny/rl_agent_config.jsonnew file mode 100644index 00000000..57f630bd--- /dev/null+++ b/c/laya_tiny/rl_agent_config.json@@ -0,0 +1,26 @@+{+ "encoder": "answerdotai/ModernBERT-large",+ "head_layers": 2,+ "max_len": 160,+ "head_max_len": 64,+ "max_prefixes": 6,+ "act_costs": {+ "escalate": 0.5+ },+ "cost_wrong_act": 3.0,+ "amp_dtype": "bf16",+ "model_name": "laya-tiny",+ "temperature": [+ 1.6369030475616455,+ 1.2514300346374512,+ 1.983399510383606+ ],+ "temperature_by_options": {+ "choice:3-5": 1.7601518630981445,+ "choice:6-10": 1.0000158548355103,+ "score:3-5": 1.2514300346374512,+ "noul:2": 1.983399510383606,+ "choice:11+": 0.10058280825614929,+ "choice:2": 1.9063563346862793+ }+}diff --git a/c/laya_tiny/tokenizer/tokenizer.json b/c/laya_tiny/tokenizer/tokenizer.jsonnew file mode 100644index 00000000..ea3c036d--- /dev/null+++ b/c/laya_tiny/tokenizer/tokenizer.json@@ -0,0 +1,1477 @@+{+ "version": "1.0",+ "truncation": null,+ "padding": null,+ "added_tokens": [+ {+ "id": 0,+ "content": "|||IP_ADDRESS|||",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 1,+ "content": "<|padding|>",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": false,+ "special": true+ },+ {+ "id": 438,+ "content": " ",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 439,+ "content": " ",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 440,+ "content": " ",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 441,+ "content": " ",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 442,+ "content": " ",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 443,+ "content": " ",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 444,+ "content": " ",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 445,+ "content": "|||EMAIL_ADDRESS|||",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 446,+ "content": "|||PHONE_NUMBER|||",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 447,+ "content": "<|endoftext|>",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": false,+ "special": true+ },+ {+ "id": 448,+ "content": "[UNK]",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": false,+ "special": true+ },+ {+ "id": 449,+ "content": "[CLS]",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": false,+ "special": true+ },+ {+ "id": 450,+ "content": "[SEP]",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": false,+ "special": true+ },+ {+ "id": 451,+ "content": "[PAD]",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": false,+ "special": true+ },+ {+ "id": 452,+ "content": "[MASK]",+ "single_word": false,+ "lstrip": true,+ "rstrip": false,+ "normalized": false,+ "special": true+ },+ {+ "id": 453,+ "content": "[unused0]",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ },+ {+ "id": 454,+ "content": "[unused1]",+ "single_word": false,+ "lstrip": false,+ "rstrip": false,+ "normalized": true,+ "special": false+ }+ ],+ "normalizer": {+ "type": "NFC"+ },+ "pre_tokenizer": {+ "type": "ByteLevel",+ "add_prefix_space": false,+ "trim_offsets": true,+ "use_regex": true+ },+ "post_processor": {+ "type": "TemplateProcessing",+ "single": [+ {+ "SpecialToken": {+ "id": "[CLS]",+ "type_id": 0+ }+ },+ {+ "Sequence": {+ "id": "A",+ "type_id": 0+ }+ },+ {+ "SpecialToken": {+ "id": "[SEP]",+ "type_id": 0+ }+ }+ ],+ "pair": [+ {+ "SpecialToken": {+ "id": "[CLS]",+ "type_id": 0+ }+ },+ {+ "Sequence": {+ "id": "A",+ "type_id": 0+ }+ },+ {+ "SpecialToken": {+ "id": "[SEP]",+ "type_id": 0+ }+ },+ {+ "Sequence": {+ "id": "B",+ "type_id": 0+ }+ },+ {+ "SpecialToken": {+ "id": "[SEP]",+ "type_id": 0+ }+ }+ ],+ "special_tokens": {+ "[UNK]": {+ "id": "[UNK]",+ "ids": [+ 448+ ],+ "tokens": [+ "[UNK]"+ ]+ },+ "[CLS]": {+ "id": "[CLS]",+ "ids": [+ 449+ ],+ "tokens": [+ "[CLS]"+ ]+ },+ "[SEP]": {+ "id": "[SEP]",+ "ids": [+ 450+ ],+ "tokens": [+ "[SEP]"+ ]+ },+ "[PAD]": {+ "id": "[PAD]",+ "ids": [+ 451+ ],+ "tokens": [+ "[PAD]"+ ]+ },+ "[MASK]": {+ "id": "[MASK]",+ "ids": [+ 452+ ],+ "tokens": [+ "[MASK]"+ ]+ }+ }+ },+ "decoder": {+ "type": "ByteLevel",+ "add_prefix_space": false,+ "trim_offsets": true,+ "use_regex": true+ },+ "model": {+ "type": "BPE",+ "dropout": null,+ "unk_token": null,+ "continuing_subword_prefix": null,+ "end_of_word_suffix": null,+ "fuse_unk": false,+ "byte_fallback": false,+ "ignore_merges": false,+ "vocab": {+ "|||IP_ADDRESS|||": 0,+ "<|padding|>": 1,+ "Ā": 2,+ "ā": 3,+ "Ă": 4,+ "ă": 5,+ "Ą": 6,+ "ą": 7,+ "Ć": 8,+ "ć": 9,+ "Ĉ": 10,+ "ĉ": 11,+ "Ċ": 12,+ "ċ": 13,+ "Č": 14,+ "č": 15,+ "Ď": 16,+ "ď": 17,+ "Đ": 18,+ "đ": 19,+ "Ē": 20,+ "ē": 21,+ "Ĕ": 22,+ "ĕ": 23,+ "Ė": 24,+ "ė": 25,+ "Ę": 26,+ "ę": 27,+ "Ě": 28,+ "ě": 29,+ "Ĝ": 30,+ "ĝ": 31,+ "Ğ": 32,+ "ğ": 33,+ "Ġ": 34,+ "!": 35,+ "\"": 36,+ "#": 37,+ "$": 38,+ "%": 39,+ "&": 40,+ "'": 41,+ "(": 42,+ ")": 43,+ "*": 44,+ "+": 45,+ ",": 46,+ "-": 47,+ ".": 48,+ "/": 49,+ "0": 50,+ "1": 51,+ "2": 52,+ "3": 53,+ "4": 54,+ "5": 55,+ "6": 56,+ "7": 57,+ "8": 58,+ "9": 59,+ ":": 60,+ ";": 61,+ "<": 62,+ "=": 63,+ ">": 64,+ "?": 65,+ "@": 66,+ "A": 67,+ "B": 68,+ "C": 69,+ "D": 70,+ "E": 71,+ "F": 72,+ "G": 73,+ "H": 74,+ "I": 75,+ "J": 76,+ "K": 77,+ "L": 78,+ "M": 79,+ "N": 80,+ "O": 81,+ "P": 82,+ "Q": 83,+ "R": 84,+ "S": 85,+ "T": 86,+ "U": 87,+ "V": 88,+ "W": 89,+ "X": 90,+ "Y": 91,+ "Z": 92,+ "[": 93,+ "\\": 94,+ "]": 95,+ "^": 96,+ "_": 97,+ "`": 98,+ "a": 99,+ "b": 100,+ "c": 101,+ "d": 102,+ "e": 103,+ "f": 104,+ "g": 105,+ "h": 106,+ "i": 107,+ "j": 108,+ "k": 109,+ "l": 110,+ "m": 111,+ "n": 112,+ "o": 113,+ "p": 114,+ "q": 115,+ "r": 116,+ "s": 117,+ "t": 118,+ "u": 119,+ "v": 120,+ "w": 121,+ "x": 122,+ "y": 123,+ "z": 124,+ "{": 125,+ "|": 126,+ "}": 127,+ "~": 128,+ "ġ": 129,+ "Ģ": 130,+ "ģ": 131,+ "Ĥ": 132,+ "ĥ": 133,+ "Ħ": 134,+ "ħ": 135,+ "Ĩ": 136,+ "ĩ": 137,+ "Ī": 138,+ "ī": 139,+ "Ĭ": 140,+ "ĭ": 141,+ "Į": 142,+ "į": 143,+ "İ": 144,+ "ı": 145,+ "IJ": 146,+ "ij": 147,+ "Ĵ": 148,+ "ĵ": 149,+ "Ķ": 150,+ "ķ": 151,+ "ĸ": 152,+ "Ĺ": 153,+ "ĺ": 154,+ "Ļ": 155,+ "ļ": 156,+ "Ľ": 157,+ "ľ": 158,+ "Ŀ": 159,+ "ŀ": 160,+ "Ł": 161,+ "ł": 162,+ "¡": 163,+ "¢": 164,+ "£": 165,+ "¤": 166,+ "¥": 167,+ "¦": 168,+ "§": 169,+ "¨": 170,+ "©": 171,+ "ª": 172,+ "«": 173,+ "¬": 174,+ "Ń": 175,+ "®": 176,+ "¯": 177,+ "°": 178,+ "±": 179,+ "²": 180,+ "³": 181,+ "´": 182,+ "µ": 183,+ "¶": 184,+ "·": 185,+ "¸": 186,+ "¹": 187,+ "º": 188,+ "»": 189,+ "¼": 190,+ "½": 191,+ "¾": 192,+ "¿": 193,+ "À": 194,+ "Á": 195,+ "Â": 196,+ "Ã": 197,+ "Ä": 198,+ "Å": 199,+ "Æ": 200,+ "Ç": 201,+ "È": 202,+ "É": 203,+ "Ê": 204,+ "Ë": 205,+ "Ì": 206,+ "Í": 207,+ "Î": 208,+ "Ï": 209,+ "Ð": 210,+ "Ñ": 211,+ "Ò": 212,+ "Ó": 213,+ "Ô": 214,+ "Õ": 215,+ "Ö": 216,+ "×": 217,+ "Ø": 218,+ "Ù": 219,+ "Ú": 220,+ "Û": 221,+ "Ü": 222,+ "Ý": 223,+ "Þ": 224,+ "ß": 225,+ "à": 226,+ "á": 227,+ "â": 228,+ "ã": 229,+ "ä": 230,+ "å": 231,+ "æ": 232,+ "ç": 233,+ "è": 234,+ "é": 235,+ "ê": 236,+ "ë": 237,+ "ì": 238,+ "í": 239,+ "î": 240,+ "ï": 241,+ "ð": 242,+ "ñ": 243,+ "ò": 244,+ "ó": 245,+ "ô": 246,+ "õ": 247,+ "ö": 248,+ "÷": 249,+ "ø": 250,+ "ù": 251,+ "ú": 252,+ "û": 253,+ "ü": 254,+ "ý": 255,+ "þ": 256,+ "ÿ": 257,+ "Ġt": 258,+ "he": 259,+ "Ġthe": 260,+ "es": 261,+ "in": 262,+ "Ġs": 263,+ "ic": 264,+ "Ġa": 265,+ "er": 266,+ "Ġw": 267,+ "le": 268,+ "or": 269,+ "en": 270,+ "is": 271,+ "at": 272,+ "ing": 273,+ "Ġc": 274,+ "Ġp": 275,+ "nd": 276,+ "ou": 277,+ "re": 278,+ "Ġn": 279,+ "ent": 280,+ "on": 281,+ "Ġd": 282,+ "Ġe": 283,+ "as": 284,+ "est": 285,+ "his": 286,+ "ot": 287,+ "qu": 288,+ "ve": 289,+ "Ġl": 290,+ "Ġre": 291,+ "ay": 292,+ "il": 293,+ "ke": 294,+ "quest": 295,+ "Ġb": 296,+ "Ġin": 297,+ "ct": 298,+ "ion": 299,+ "ice": 300,+ "ment": 301,+ "od": 302,+ "pl": 303,+ "us": 304,+ "ver": 305,+ "Ġ\"": 306,+ "Ġle": 307,+ "Ġand": 308,+ "The": 309,+ "al": 310,+ "am": 311,+ "an": 312,+ "ar": 313,+ "ate": 314,+ "fu": 315,+ "it": 316,+ "ld": 317,+ "plic": 318,+ "ur": 319,+ "vel": 320,+ "Ġf": 321,+ "Ġh": 322,+ "Ġis": 323,+ "Ġou": 324,+ "Ġlevel": 325,+ "Ġnot": 326,+ "Ġpr": 327,+ "Ġthis": 328,+ "Ġwe": 329,+ "\":": 330,+ "02": 331,+ "ase": 332,+ "ce": 333,+ "ch": 334,+ "et": 335,+ "fund": 336,+ "hing": 337,+ "ike": 338,+ "ill": 339,+ "lease": 340,+ "oc": 341,+ "oes": 342,+ "om": 343,+ "question": 344,+ "user": 345,+ "very": 346,+ "Ġ2": 347,+ "Ġo": 348,+ "Ġor": 349,+ "Ġquestion": 350,+ "Ġcan": 351,+ "Ġde": 352,+ "Ġem": 353,+ "Ġlike": 354,+ "Ġour": 355,+ "Ġrefund": 356,+ "Ġrequest": 357,+ "Ġst": 358,+ "Ġsu": 359,+ "Ġte": 360,+ "Ġto": 361,+ "Ġwor": 362,+ "\",": 363,+ "026": 364,+ "Does": 365,+ "Please": 366,+ "This": 367,+ "act": 368,+ "ag": 369,+ "ail": 370,+ "ave": 371,+ "atement": 372,+ "ayment": 373,+ "bod": 374,+ "body": 375,+ "cel": 376,+ "ed": 377,+ "ew": 378,+ "ex": 379,+ "ers": 380,+ "gent": 381,+ "ical": 382,+ "king": 383,+ "ks": 384,+ "ked": 385,+ "loc": 386,+ "locking": 387,+ "me": 388,+ "og": 389,+ "oice": 390,+ "old": 391,+ "ow": 392,+ "odu": 393,+ "oduct": 394,+ "ont": 395,+ "ontr": 396,+ "ontract": 397,+ "ould": 398,+ "pp": 399,+ "ps": 400,+ "plicate": 401,+ "reat": 402,+ "se": 403,+ "st": 404,+ "su": 405,+ "sw": 406,+ "um": 407,+ "uplicate": 408,+ "urgent": 409,+ "Ġ3": 410,+ "ĠI": 411,+ "Ġr": 412,+ "Ġurgent": 413,+ "Ġuser": 414,+ "Ġy": 415,+ "Ġ2026": 416,+ "Ġan": 417,+ "Ġas": 418,+ "Ġblocking": 419,+ "Ġcontract": 420,+ "Ġcr": 421,+ "Ġcancel": 422,+ "Ġevery": 423,+ "Ġemail": 424,+ "Ġfor": 425,+ "Ġhold": 426,+ "Ġinv": 427,+ "Ġno": 428,+ "Ġpayment": 429,+ "Ġproduct": 430,+ "Ġsc": 431,+ "Ġstatement": 432,+ "Ġth": 433,+ "Ġteam": 434,+ "Ġwas": 435,+ "Ġwill": 436,+ "Ġworks": 437,+ " ": 438,+ " ": 439,+ " ": 440,+ " ": 441,+ " ": 442,+ " ": 443,+ " ": 444,+ "|||EMAIL_ADDRESS|||": 445,+ "|||PHONE_NUMBER|||": 446,+ "<|endoftext|>": 447+ },+ "merges": [+ [+ "Ġ",+ "t"+ ],+ [+ "h",+ "e"+ ],+ [+ "Ġt",+ "he"+ ],+ [+ "e",+ "s"+ ],+ [+ "i",+ "n"+ ],+ [+ "Ġ",+ "s"+ ],+ [+ "i",+ "c"+ ],+ [+ "Ġ",+ "a"+ ],+ [+ "e",+ "r"+ ],+ [+ "Ġ",+ "w"+ ],+ [+ "l",+ "e"+ ],+ [+ "o",+ "r"+ ],+ [+ "e",+ "n"+ ],+ [+ "i",+ "s"+ ],+ [+ "a",+ "t"+ ],+ [+ "in",+ "g"+ ],+ [+ "Ġ",+ "c"+ ],+ [+ "Ġ",+ "p"+ ],+ [+ "n",+ "d"+ ],+ [+ "o",+ "u"+ ],+ [+ "r",+ "e"+ ],+ [+ "Ġ",+ "n"+ ],+ [+ "en",+ "t"+ ],+ [+ "o",+ "n"+ ],+ [+ "Ġ",+ "d"+ ],+ [+ "Ġ",+ "e"+ ],+ [+ "a",+ "s"+ ],+ [+ "es",+ "t"+ ],+ [+ "h",+ "is"+ ],+ [+ "o",+ "t"+ ],+ [+ "q",+ "u"+ ],+ [+ "v",+ "e"+ ],+ [+ "Ġ",+ "l"+ ],+ [+ "Ġ",+ "re"+ ],+ [+ "a",+ "y"+ ],+ [+ "i",+ "l"+ ],+ [+ "k",+ "e"+ ],+ [+ "qu",+ "est"+ ],+ [+ "Ġ",+ "b"+ ],+ [+ "Ġ",+ "in"+ ],+ [+ "c",+ "t"+ ],+ [+ "i",+ "on"+ ],+ [+ "ic",+ "e"+ ],+ [+ "m",+ "ent"+ ],+ [+ "o",+ "d"+ ],+ [+ "p",+ "l"+ ],+ [+ "u",+ "s"+ ],+ [+ "v",+ "er"+ ],+ [+ "Ġ",+ "\""+ ],+ [+ "Ġ",+ "le"+ ],+ [+ "Ġa",+ "nd"+ ],+ [+ "T",+ "he"+ ],+ [+ "a",+ "l"+ ],+ [+ "a",+ "m"+ ],+ [+ "a",+ "n"+ ],+ [+ "a",+ "r"+ ],+ [+ "at",+ "e"+ ],+ [+ "f",+ "u"+ ],+ [+ "i",+ "t"+ ],+ [+ "l",+ "d"+ ],+ [+ "pl",+ "ic"+ ],+ [+ "u",+ "r"+ ],+ [+ "ve",+ "l"+ ],+ [+ "Ġ",+ "f"+ ],+ [+ "Ġ",+ "h"+ ],+ [+ "Ġ",+ "is"+ ],+ [+ "Ġ",+ "ou"+ ],+ [+ "Ġle",+ "vel"+ ],+ [+ "Ġn",+ "ot"+ ],+ [+ "Ġp",+ "r"+ ],+ [+ "Ġt",+ "his"+ ],+ [+ "Ġw",+ "e"+ ],+ [+ "\"",+ ":"+ ],+ [+ "0",+ "2"+ ],+ [+ "as",+ "e"+ ],+ [+ "c",+ "e"+ ],+ [+ "c",+ "h"+ ],+ [+ "e",+ "t"+ ],+ [+ "fu",+ "nd"+ ],+ [+ "h",+ "ing"+ ],+ [+ "i",+ "ke"+ ],+ [+ "il",+ "l"+ ],+ [+ "le",+ "ase"+ ],+ [+ "o",+ "c"+ ],+ [+ "o",+ "es"+ ],+ [+ "o",+ "m"+ ],+ [+ "quest",+ "ion"+ ],+ [+ "us",+ "er"+ ],+ [+ "ver",+ "y"+ ],+ [+ "Ġ",+ "2"+ ],+ [+ "Ġ",+ "o"+ ],+ [+ "Ġ",+ "or"+ ],+ [+ "Ġ",+ "question"+ ],+ [+ "Ġc",+ "an"+ ],+ [+ "Ġd",+ "e"+ ],+ [+ "Ġe",+ "m"+ ],+ [+ "Ġl",+ "ike"+ ],+ [+ "Ġou",+ "r"+ ],+ [+ "Ġre",+ "fund"+ ],+ [+ "Ġre",+ "quest"+ ],+ [+ "Ġs",+ "t"+ ],+ [+ "Ġs",+ "u"+ ],+ [+ "Ġt",+ "e"+ ],+ [+ "Ġt",+ "o"+ ],+ [+ "Ġw",+ "or"+ ],+ [+ "\"",+ ","+ ],+ [+ "02",+ "6"+ ],+ [+ "D",+ "oes"+ ],+ [+ "P",+ "lease"+ ],+ [+ "T",+ "his"+ ],+ [+ "a",+ "ct"+ ],+ [+ "a",+ "g"+ ],+ [+ "a",+ "il"+ ],+ [+ "a",+ "ve"+ ],+ [+ "ate",+ "ment"+ ],+ [+ "ay",+ "ment"+ ],+ [+ "b",+ "od"+ ],+ [+ "bod",+ "y"+ ],+ [+ "ce",+ "l"+ ],+ [+ "e",+ "d"+ ],+ [+ "e",+ "w"+ ],+ [+ "e",+ "x"+ ],+ [+ "er",+ "s"+ ],+ [+ "g",+ "ent"+ ],+ [+ "ic",+ "al"+ ],+ [+ "k",+ "ing"+ ],+ [+ "k",+ "s"+ ],+ [+ "ke",+ "d"+ ],+ [+ "l",+ "oc"+ ],+ [+ "loc",+ "king"+ ],+ [+ "m",+ "e"+ ],+ [+ "o",+ "g"+ ],+ [+ "o",+ "ice"+ ],+ [+ "o",+ "ld"+ ],+ [+ "o",+ "w"+ ],+ [+ "od",+ "u"+ ],+ [+ "odu",+ "ct"+ ],+ [+ "on",+ "t"+ ],+ [+ "ont",+ "r"+ ],+ [+ "ontr",+ "act"+ ],+ [+ "ou",+ "ld"+ ],+ [+ "p",+ "p"+ ],+ [+ "p",+ "s"+ ],+ [+ "plic",+ "ate"+ ],+ [+ "re",+ "at"+ ],+ [+ "s",+ "e"+ ],+ [+ "s",+ "t"+ ],+ [+ "s",+ "u"+ ],+ [+ "s",+ "w"+ ],+ [+ "u",+ "m"+ ],+ [+ "u",+ "plicate"+ ],+ [+ "ur",+ "gent"+ ],+ [+ "Ġ",+ "3"+ ],+ [+ "Ġ",+ "I"+ ],+ [+ "Ġ",+ "r"+ ],+ [+ "Ġ",+ "urgent"+ ],+ [+ "Ġ",+ "user"+ ],+ [+ "Ġ",+ "y"+ ],+ [+ "Ġ2",+ "026"+ ],+ [+ "Ġa",+ "n"+ ],+ [+ "Ġa",+ "s"+ ],+ [+ "Ġb",+ "locking"+ ],+ [+ "Ġc",+ "ontract"+ ],+ [+ "Ġc",+ "r"+ ],+ [+ "Ġcan",+ "cel"+ ],+ [+ "Ġe",+ "very"+ ],+ [+ "Ġem",+ "ail"+ ],+ [+ "Ġf",+ "or"+ ],+ [+ "Ġh",+ "old"+ ],+ [+ "Ġin",+ "v"+ ],+ [+ "Ġn",+ "o"+ ],+ [+ "Ġp",+ "ayment"+ ],+ [+ "Ġpr",+ "oduct"+ ],+ [+ "Ġs",+ "c"+ ],+ [+ "Ġst",+ "atement"+ ],+ [+ "Ġt",+ "h"+ ],+ [+ "Ġte",+ "am"+ ],+ [+ "Ġw",+ "as"+ ],+ [+ "Ġw",+ "ill"+ ],+ [+ "Ġwor",+ "ks"+ ]+ ]+ }+}diff --git a/c/laya_tiny/tokenizer/tokenizer_config.json b/c/laya_tiny/tokenizer/tokenizer_config.jsonnew file mode 100644index 00000000..051649f3--- /dev/null+++ b/c/laya_tiny/tokenizer/tokenizer_config.json@@ -0,0 +1,14 @@+{+ "clean_up_tokenization_spaces": true,+ "cls_token": "[CLS]",+ "mask_token": "[MASK]",+ "model_input_names": [+ "input_ids",+ "attention_mask"+ ],+ "model_max_length": 512,+ "pad_token": "[PAD]",+ "sep_token": "[SEP]",+ "tokenizer_class": "PreTrainedTokenizerFast",+ "unk_token": "[UNK]"+}diff --git a/c/tests/test_decision_serve.py b/c/tests/test_decision_serve.pynew file mode 100644index 00000000..86d2fcc4--- /dev/null+++ b/c/tests/test_decision_serve.py@@ -0,0 +1,394 @@+"""Decision engines behind POST /v1/systemone: the DECIDE command, end to end.++A decision engine (Laya) does not generate. It announces `CAPS decide=1+chat=0`, and /v1/systemone hands it the request as one DECIDE record instead+of scoring options through the logprob channel. Three layers are pinned here:++ DecideProtocol the wire, against a fake engine process: the DECIDE frame+ byte for byte, the DECISION + DONE reply, a refusal+ (DECIDE_INVALID) as the caller's 422, the CAPS flags.+ DecisionGateway the HTTP route against a fake decision engine: the reply+ has exactly the shape the LLM path returns, the record+ carries the state and the options as the reference reads+ them, the generating endpoints answer 400 with a pointer,+ validation stays the route's own.+ LayaTinyEndToEnd the real laya engine on the tiny fixture behind the real+ gateway: the probabilities that come back over HTTP are+ the `laya` package's (laya_tiny/ref.json), and the+ latency is split into gateway and engine time.++LayaTinyEndToEnd is skipped without the fixture and the binary+(`make laya-tiny-check` builds both and turns the skip into a failure).+"""+import json+import math+import os+import sys+import threading+import time+import unittest+from pathlib import Path+from unittest.mock import patch+from urllib.error import HTTPError+from urllib.request import Request, urlopen++HERE = Path(__file__).resolve().parent.parent+sys.path.insert(0, str(HERE))+sys.path.insert(0, str(HERE / "tests"))+import openai_server # noqa: E402+from family_registry import family_by_id # noqa: E402+from openai_server import APIError, APIServer, Engine, READY # noqa: E402+from test_openai_server import BlockingStream, FakeProcess # noqa: E402++FIXTURE = HERE / "laya_tiny"+BINARY = HERE / ("laya.exe" if os.name == "nt" else "laya")+REQUIRED = os.environ.get("LAYA_TINY_REQUIRED") == "1"++TICKET = {"from": "[email protected]", "subject": "Duplicate charge on invoice #4411",+ "body": "Hi, we were billed twice for March. Please refund the duplicate today or we "+ "will cancel our plan."}+QUESTIONS = {+ "department": {"type": "choice", "instructions": "Which department should handle this request?",+ "criteria": {"billing": "invoices, payments, refunds",+ "technical": "bugs, outages, system errors",+ "sales": "pricing, new contracts", "other": "everything else"}},+ "urgency": {"type": "score", "instructions": "How urgent is this request?",+ "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},+ "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},+ "refund_requested": {"type": "noul", "instructions": "Does the user explicitly request a refund?"},+}+++def decision_process(on_decide):+ """A FakeProcess whose handshake announces a decision engine."""+ process = FakeProcess(on_decide)+ process.stdout = BlockingStream(READY + b"CAPS decide=1 chat=0 max_len=512\n"+ b"STAT 0 0.0 0.0 1.00 0 0\n")+ return process+++class DecideProtocol(unittest.TestCase):+ def start(self, respond):+ process = decision_process(respond)+ with patch("openai_server.subprocess.Popen", return_value=process):+ engine = Engine("laya", "model", family=family_by_id("laya"))+ self.addCleanup(engine.close)+ return engine, process++ def test_caps_say_decide_and_no_chat(self):+ engine, _ = self.start(lambda process, frame: None)+ self.assertTrue(engine.decides)+ self.assertFalse(engine.chats)+ self.assertTrue(openai_server.engine_decides(engine))+ self.assertFalse(openai_server.engine_chats(engine))++ def test_round_trip_is_byte_exact(self):+ record = {"state": "sé", "state_type": "string", "questions": [+ {"id": "q", "type": "noul", "instructions": "ok?",+ "options": [{"label": "false", "text": None}, {"label": "true", "text": None}]}]}+ payload = json.dumps(record, ensure_ascii=False, separators=(",", ":")).encode()+ expected = f"DECIDE 1 0 {len(payload)}\n".encode() + payload + b"\n"+ reply = json.dumps({"answers": [{"id": "q", "logits": [0.0, 1.0], "probs": [0.25, 0.75],+ "temperature": 2.0, "tokens": 9}],+ "input_tokens": 9, "engine_ms": 4.5}).encode()++ def respond(process, frame):+ self.assertEqual(frame, expected)+ process.stdout.feed(f"DECISION 1 {len(reply)}\n".encode() + reply ++ b"\nDONE 1 STAT 0 2000.000 0.0 1.00 9 0\n")++ engine, process = self.start(respond)+ decision, stats = engine.decide(record)+ self.assertEqual(process.writes, [expected])+ self.assertEqual(decision["answers"][0]["probs"], [0.25, 0.75])+ self.assertEqual(stats["prompt_tokens"], 9)+ self.assertEqual(stats["completion_tokens"], 0)++ def test_a_refused_record_is_the_callers_422(self):+ def respond(process, frame):+ process.stdout.feed(b"ERROR 1 DECIDE_INVALID questions.bucket: only 38 of its 40 "+ b"option markers fit in max_len=160\n")++ engine, _ = self.start(respond)+ with self.assertRaises(APIError) as caught:+ engine.decide({"state": "x", "state_type": "string", "questions": []})+ self.assertEqual(caught.exception.status, 422)+ self.assertEqual(caught.exception.param, "questions.bucket")+ self.assertIn("only 38 of its 40", caught.exception.message)++ def test_an_engine_failure_is_not_the_callers(self):+ def respond(process, frame):+ process.stdout.feed(b"ERROR 1 DECIDE_FAILED out of memory\n")++ engine, _ = self.start(respond)+ with self.assertRaises(RuntimeError):+ engine.decide({"state": "x", "state_type": "string", "questions": []})++ def test_a_nul_never_reaches_the_engine(self):+ engine, process = self.start(lambda process, frame: self.fail("written"))+ with self.assertRaises(APIError) as caught:+ engine.decide({"state": "a\0b", "state_type": "string", "questions": []})+ self.assertEqual(caught.exception.status, 422)+ self.assertEqual(process.writes, [])+++class FakeDecisionEngine:+ """A decision engine that answers from a table: the probabilities of each+ question by id, in the record's option order."""++ decides = True+ chats = False+ kv_slots = 1++ def __init__(self, table, refuse=None):+ self.table = table+ self.refuse = refuse+ self.records = []++ def decide(self, record, cache_slot=0, cancelled=None):+ self.records.append(record)+ if self.refuse:+ raise APIError(422, self.refuse, "questions.q", "invalid_question")+ answers = [{"id": q["id"], "logits": [math.log(p) for p in self.table[q["id"]]],+ "probs": self.table[q["id"]], "temperature": 1.0, "tokens": 10}+ for q in record["questions"]]+ return {"answers": answers, "input_tokens": 10 * len(answers), "engine_ms": 12.5}, {}++ def generate(self, *args, **kwargs):+ raise AssertionError("a decision engine is never asked to generate")++ def close(self):+ pass+++class DecisionGateway(unittest.TestCase):+ def serve(self, engine):+ self.engine = engine+ self.server = APIServer(("127.0.0.1", 0), engine, "laya")+ self.addCleanup(self.server.server_close)+ self.addCleanup(self.server.shutdown)+ threading.Thread(target=self.server.serve_forever, daemon=True).start()+ self.base = f"http://127.0.0.1:{self.server.server_port}"++ def post(self, path, body):+ request = Request(self.base + path, data=json.dumps(body).encode(),+ headers={"Content-Type": "application/json"})+ with urlopen(request, timeout=10) as response:+ return json.loads(response.read()), dict(response.headers)++ def post_error(self, path, body):+ try:+ self.post(path, body)+ except HTTPError as error:+ return error.code, json.loads(error.read())+ self.fail("expected an error")++ def table(self):+ return {"department": [0.7, 0.1, 0.1, 0.1], "urgency": [0.2, 0.3, 0.5],+ "churn_risk": [0.25, 0.75], "refund_requested": [0.9, 0.1]}++ def test_reply_has_the_shape_the_llm_path_returns(self):+ self.serve(FakeDecisionEngine(self.table()))+ out, headers = self.post("/v1/systemone", {"model": "jev-latest", "state": TICKET,+ "questions": QUESTIONS})+ self.assertEqual(out["model"], "laya")+ self.assertEqual(list(out["answers"]), list(QUESTIONS))+ self.assertEqual(out["answers"]["churn_risk"], {"type": "noul", "noul": 0.75})+ self.assertEqual(out["answers"]["refund_requested"], {"type": "noul", "noul": 0.1})+ department = out["answers"]["department"]+ self.assertEqual(set(department), {"type", "choice", "probabilities", "confidence"})+ self.assertEqual(department["choice"], "billing")+ self.assertEqual(list(department["probabilities"]), ["billing", "technical", "sales", "other"])+ self.assertAlmostEqual(department["confidence"], (4 * 0.7 - 1) / 3, places=6)+ urgency = out["answers"]["urgency"]+ self.assertEqual(set(urgency), {"type", "score", "legend", "probabilities", "confidence"})+ self.assertEqual(urgency["legend"], {"1": "not urgent", "2": "soon",+ "3": "critical deadline or blocking issue"})+ self.assertEqual(list(urgency["probabilities"]), ["1", "2", "3"])+ self.assertAlmostEqual(urgency["score"], 1 * 0.2 + 2 * 0.3 + 3 * 0.5, places=6)+ self.assertEqual(out["usage"], {"input_tokens": 40, "output_tokens": 0})+ self.assertEqual(headers["x-colibri-engine-ms"], "12.5")++ def test_the_record_carries_what_the_reference_reads(self):+ self.serve(FakeDecisionEngine({"q": [0.5, 0.5], "c": [0.6, 0.4], "s": [0.5, 0.5],+ "d": [0.5, 0.5]}))+ self.post("/v1/systemone", {"model": "jev-latest",+ "state": {"ticket": 4711, "text": "café", "ok": True},+ "questions": {+ "q": {"type": "noul", "instructions": {"ask": "money?"},+ "criteria": {"true": "about a payment"}},+ "c": {"type": "choice", "instructions": " Which? ",+ "criteria": {"a": None, "b": {"desc": "structured"}}},+ "s": {"type": "score", "criteria": ["low", {"n": 2}]},+ "d": {"type": "noul"}}})+ record = self.engine.records[0]+ # a JSON state as json.dumps(ensure_ascii=False) writes it, and its type+ self.assertEqual(record["state"], '{"ticket": 4711, "text": "café", "ok": true}')+ self.assertEqual(record["state_type"], "object")+ q, c, s, d = record["questions"]+ self.assertEqual(q["instructions"], '{"ask": "money?"}')+ self.assertEqual(q["options"], [{"label": "false", "text": None},+ {"label": "true", "text": "about a payment"}])+ self.assertEqual(c["instructions"], " Which? ") # exactly as sent+ self.assertEqual(c["options"], [{"label": "a", "text": None},+ {"label": "b", "text": '{"desc": "structured"}'}])+ self.assertEqual(s["options"], [{"label": "1", "text": "low"},+ {"label": "2", "text": '{"n": 2}'}])+ self.assertEqual(s["instructions"], "Rate this on the scale below.")+ self.assertEqual(d["instructions"], "Is this true?")++ def test_a_list_state_is_a_conversation(self):+ self.serve(FakeDecisionEngine({"q": [0.5, 0.5]}))+ turns = [{"role": "user", "content": "hi"}, {"role": "assistant", "content": "hello"}]+ self.post("/v1/systemone", {"state": turns, "questions": {"q": {"type": "noul",+ "instructions": "?"}}})+ self.assertEqual(self.engine.records[0]["state_type"], "array")+ self.assertEqual(json.loads(self.engine.records[0]["state"]), turns)++ def test_generating_endpoints_point_to_systemone(self):+ self.serve(FakeDecisionEngine({}))+ for path, body in (("/v1/chat/completions", {"messages": [{"role": "user", "content": "hi"}]}),+ ("/v1/completions", {"prompt": "hi"}),+ ("/v1/messages", {"max_tokens": 5, "messages": [{"role": "user", "content": "hi"}]}),+ ("/v1/brio", {"state": "x", "options": ["a", "b"]})):+ with self.subTest(path=path):+ status, error = self.post_error(path, dict(body, model="laya"))+ self.assertEqual(status, 400)+ if path != "/v1/messages": # Anthropic's envelope has no code+ self.assertEqual(error["error"]["code"], "unsupported_endpoint")+ self.assertIn("POST /v1/systemone", error["error"]["message"])++ def test_models_card_and_health_say_decision(self):+ self.serve(FakeDecisionEngine({}))+ with urlopen(self.base + "/v1/models", timeout=10) as response:+ card = json.loads(response.read())["data"][0]+ self.assertEqual(card["capabilities"], ["decision"])+ with urlopen(self.base + "/health", timeout=10) as response:+ self.assertEqual(json.loads(response.read())["capabilities"], ["decision"])++ def test_validation_is_the_routes_own(self):+ self.serve(FakeDecisionEngine({}))+ for body, param in (({"questions": {"q": {"type": "noul"}}}, "state"),+ ({"state": "x", "questions": {}}, "questions"),+ ({"state": "x", "questions": {"q": {"type": "maybe"}}}, "questions.q.type"),+ ({"state": "x", "questions": {"q": {"type": "score", "criteria": ["one"]}}},+ "questions.q.criteria")):+ with self.subTest(param=param):+ status, error = self.post_error("/v1/systemone", body)+ self.assertEqual(status, 422)+ self.assertEqual(error["error"]["param"], param)+ self.assertEqual(self.engine.records, [])++ def test_an_engine_refusal_reaches_the_client_as_422(self):+ self.serve(FakeDecisionEngine({}, refuse="questions.q: only 3 of its 30 option markers fit"))+ status, error = self.post_error("/v1/systemone", {"state": "x", "questions": {+ "q": {"type": "choice", "criteria": {"a": "x", "b": "y"}}}})+ self.assertEqual(status, 422)+ self.assertIn("only 3 of its 30", error["error"]["message"])+++def _laya_ready():+ return (FIXTURE / "model.safetensors").exists() and BINARY.exists()+++class LayaTinyEndToEnd(unittest.TestCase):+ """The real engine behind the real gateway, on the tiny fixture."""++ @classmethod+ def setUpClass(cls):+ if not _laya_ready():+ message = "laya or laya_tiny missing: run make laya-tiny-check"+ if REQUIRED:+ raise AssertionError(message)+ raise unittest.SkipTest(message)+ cls.ref = {case["name"]: case for case in+ json.loads((FIXTURE / "ref.json").read_text(encoding="utf-8"))["cases"]}++ def setUp(self):+ engine = Engine(BINARY, FIXTURE, env=dict(os.environ, OMP_NUM_THREADS="2"),+ family=family_by_id("laya"))+ self.addCleanup(engine.process.stdout.close)+ self.addCleanup(engine.close)+ self.server = APIServer(("127.0.0.1", 0), engine, "laya-tiny")+ self.addCleanup(self.server.server_close)+ self.addCleanup(self.server.shutdown)+ threading.Thread(target=self.server.serve_forever, daemon=True).start()+ self.base = f"http://127.0.0.1:{self.server.server_port}"++ def post(self, path, body):+ request = Request(self.base + path, data=json.dumps(body).encode(),+ headers={"Content-Type": "application/json"})+ with urlopen(request, timeout=120) as response:+ return json.loads(response.read()), dict(response.headers)++ def test_systemone_answers_with_the_reference_probabilities(self):+ # the cases the route accepts as Jev requests (it refuses a bare number as a+ # score level, which the engine-level oracle still covers)+ for name in ("readme_ticket", "plain_two_options", "conversation_truncated_left",+ "json_instructions_and_state", "special_text", "unicode"):+ case = self.ref[name]+ with self.subTest(case=name):+ out, headers = self.post("/v1/systemone", {"model": "jev-latest",+ "state": case["state"],+ "questions": case["questions"]})+ for want in case["answers"]:+ got = out["answers"][want["id"]]+ probs = want["probs"]+ if got["type"] == "noul":+ self.assertAlmostEqual(got["noul"], probs[1], delta=2e-5)+ continue+ values = list(got["probabilities"].values())+ self.assertEqual(len(values), len(probs))+ for a, b in zip(values, probs):+ self.assertAlmostEqual(a, b, delta=2e-5)+ if got["type"] == "choice":+ labels = list(case["questions"][want["id"]]["criteria"])+ self.assertEqual(got["choice"], labels[probs.index(max(probs))])+ self.assertEqual(out["usage"]["input_tokens"],+ sum(len(a["ids"]) for a in case["answers"]))+ self.assertEqual(out["usage"]["output_tokens"], 0)+ self.assertIn("x-colibri-engine-ms", headers)++ def test_a_question_whose_options_do_not_fit_is_a_422(self):+ case = self.ref["too_many_options"]+ request = Request(self.base + "/v1/systemone",+ data=json.dumps({"state": case["state"], "questions": case["questions"]}).encode(),+ headers={"Content-Type": "application/json"})+ with self.assertRaises(HTTPError) as caught:+ urlopen(request, timeout=60)+ self.assertEqual(caught.exception.code, 422)+ error = json.loads(caught.exception.read())["error"]+ self.assertEqual(error["param"], "questions.bucket")+ self.assertIn("only 38 of its 40 option markers fit", error["message"])++ def test_chat_is_refused(self):+ request = Request(self.base + "/v1/chat/completions",+ data=json.dumps({"model": "laya-tiny",+ "messages": [{"role": "user", "content": "hi"}]}).encode(),+ headers={"Content-Type": "application/json"})+ with self.assertRaises(HTTPError) as caught:+ urlopen(request, timeout=10)+ self.assertEqual(caught.exception.code, 400)++ def test_latency_gateway_and_engine(self):+ """Not a gate: the numbers the report quotes, measured the same way every+ time. Wall time at the client, the engine's own compute time, and the rest+ (HTTP, JSON, the pipe) as the gateway's share."""+ case = self.ref["readme_ticket"]+ body = {"model": "jev-latest", "state": case["state"], "questions": case["questions"]}+ self.post("/v1/systemone", body) # warm+ walls, engines = [], []+ for _ in range(10):+ started = time.perf_counter()+ _, headers = self.post("/v1/systemone", body)+ walls.append((time.perf_counter() - started) * 1e3)+ engines.append(float(headers["x-colibri-engine-ms"]))+ walls.sort(), engines.sort()+ wall, engine = walls[len(walls) // 2], engines[len(engines) // 2]+ sys.stderr.write(f"\n[laya tiny e2e] /v1/systemone, 4 questions: median {wall:.1f} ms at the "+ f"client, {engine:.1f} ms in the engine, {wall - engine:.1f} ms gateway\n")+ self.assertGreater(wall, engine)+++if __name__ == "__main__":+ unittest.main()diff --git a/c/tests/test_family_registry.py b/c/tests/test_family_registry.pyindex d5d42148..1223a363 100644--- a/c/tests/test_family_registry.py+++ b/c/tests/test_family_registry.py@@ -1054,6 +1054,8 @@ class FamilyRegistryTest(unittest.TestCase):# An image model has no chat template and nothing to tune; the# template is the identity and `coli tune` refuses the modality."qwen_image": "hello {world}",+ # A decision model answers questions; it has no prompt to replay.+ "laya": "hello {world}",}self.assertEqual({family.id: tuning_replay_prompt(family, prompt) for family in FAMILIES},diff --git a/c/tests/test_laya_tiny.py b/c/tests/test_laya_tiny.pynew file mode 100644index 00000000..8a546a9e--- /dev/null+++ b/c/tests/test_laya_tiny.py@@ -0,0 +1,134 @@+"""laya.c against the `laya` package, on the tiny fixture.++laya_tiny/ref.json holds what Convai Innovations' own runtime (laya.Agent,+pinned version) answered on the checkpoint tools/make_laya_tiny.py writes:+per question the token ids, the [MASK] marker positions, the raw scorer+logits, the calibrated probabilities and the act head, plus the refusal for a+question whose options do not fit. tools/make_laya_ref.py wrote it, offline;+this test needs neither torch nor the package.++Each case goes through the gateway's own record builder+(openai_server.systemone_decision_record) and the engine's record parser, the+path a POST /v1/systemone takes, and must come back with:++ - the same token ids and markers, exactly (tokenizer, sequence assembly,+ truncation of long states and conversations, the option budget);+ - logits within 2e-4 and probabilities within 2e-5 of the reference, the+ tolerance ref.json states (f32 against f32, different summation order);+ - the same decision (argmax) on every question;+ - the same refusal text when the reference refused.++The tokenizer is also checked alone on strings chosen to hit NFC, added tokens+matched before BPE (runs of spaces, the |||...||| placeholders, specials),+contractions, digits, non-Latin scripts and emoji.++Skipped when the fixture or the binary is missing (`make laya-tiny-check`+builds both and sets LAYA_TINY_REQUIRED=1, which turns the skip into a failure).+"""+import hashlib+import json+import os+import subprocess+import sys+import tempfile+import unittest+from pathlib import Path++HERE = Path(__file__).resolve().parent.parent+sys.path.insert(0, str(HERE))+from openai_server import systemone_decision_record # noqa: E402++FIXTURE = HERE / "laya_tiny"+BINARY = HERE / ("laya.exe" if os.name == "nt" else "laya")+REQUIRED = os.environ.get("LAYA_TINY_REQUIRED") == "1"+++def sha256(path):+ return hashlib.sha256(path.read_bytes()).hexdigest()+++class LayaTinyOracle(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ cls.ref = json.loads((FIXTURE / "ref.json").read_text(encoding="utf-8"))+ missing = [str(p) for p in (FIXTURE / "model.safetensors", BINARY) if not p.exists()]+ if missing:+ message = f"missing {', '.join(missing)}: run make laya-tiny-check"+ if REQUIRED:+ raise AssertionError(message)+ raise unittest.SkipTest(message)+ for name, digest in cls.ref["fixture_sha256"].items():+ if sha256(FIXTURE / name) != digest:+ raise AssertionError(f"laya_tiny/{name} is not the file ref.json was computed on; "+ "regenerate it with make laya-tiny-generate (or the reference "+ "with tools/make_laya_ref.py if the generator changed)")++ def run_engine(self, *args):+ result = subprocess.run([str(BINARY), "--model", str(FIXTURE), *args],+ capture_output=True, text=True, encoding="utf-8",+ env=dict(os.environ, OMP_NUM_THREADS="2"), timeout=300)+ self.assertEqual(result.returncode, 0, result.stderr[-2000:])+ return result.stdout++ def test_tokenizer_matches_the_reference(self):+ strings = [entry["text"] for entry in self.ref["tokenizer"]]+ with tempfile.TemporaryDirectory() as tmp:+ path = Path(tmp) / "strings.json"+ path.write_text(json.dumps(strings, ensure_ascii=False), encoding="utf-8")+ got = json.loads(self.run_engine("--tokenize", str(path)))+ for entry, ids in zip(self.ref["tokenizer"], got):+ self.assertEqual(ids, entry["ids"], f"tokenizer: {entry['text']!r}")+ self.assertEqual(len(got), len(strings))++ def test_every_case_matches_the_reference(self):+ cases = self.ref["cases"]+ tol = self.ref["tolerance"]+ records = []+ for case in cases:+ record = systemone_decision_record({"state": case["state"],+ "questions": case["questions"]})+ records.append({"payload": json.dumps(record, ensure_ascii=False)})+ with tempfile.TemporaryDirectory() as tmp:+ path = Path(tmp) / "records.json"+ path.write_text(json.dumps(records, ensure_ascii=False), encoding="utf-8")+ lines = self.run_engine("--records", str(path), "--ids").splitlines()+ self.assertEqual(len(lines), len(cases))+ worst = {"probs": 0.0, "logits": 0.0, "act_probability": 0.0}+ decisions = 0+ for case, line in zip(cases, lines):+ got = json.loads(line)+ name = case["name"]+ if "error" in case:+ self.assertIn("error", got, f"{name}: the reference refused, the engine answered")+ # the engine names the field; the reference names the question+ want = case["error"].split(": ", 1)[1]+ self.assertTrue(got["error"].endswith(want), f"{name}: {got['error']!r} vs {want!r}")+ continue+ self.assertNotIn("error", got, f"{name}: {got.get('error')}")+ self.assertEqual(len(got["answers"]), len(case["answers"]), name)+ for want, answer, seq in zip(case["answers"], got["answers"], got["sequences"]):+ where = f"{name}.{want['id']}"+ self.assertEqual(answer["id"], want["id"], where)+ self.assertEqual(seq["ids"], want["ids"], f"{where}: token ids")+ self.assertEqual(seq["markers"], want["markers"], f"{where}: markers")+ self.assertEqual(answer["tokens"], len(want["ids"]), where)+ self.assertEqual(answer["state_tokens"], want["state_tokens"], where)+ self.assertEqual(answer["state_dropped"], want["state_dropped"], where)+ self.assertAlmostEqual(answer["temperature"], want["temperature"], places=6, msg=where)+ for key in ("logits", "probs"):+ diff = max(abs(a - b) for a, b in zip(answer[key], want[key]))+ worst[key] = max(worst[key], diff)+ self.assertLessEqual(diff, tol[key], f"{where}: {key} {answer[key]} vs {want[key]}")+ act = answer["actions"]["act"]+ worst["act_probability"] = max(worst["act_probability"], abs(act - want["act_probability"]))+ self.assertLessEqual(abs(act - want["act_probability"]), tol["act_probability"], where)+ pick = max(range(len(answer["probs"])), key=lambda i: (answer["probs"][i], -i))+ ref_pick = max(range(len(want["probs"])), key=lambda i: (want["probs"][i], -i))+ self.assertEqual(pick, ref_pick, f"{where}: a different decision")+ decisions += 1+ sys.stderr.write(f"\n[laya tiny] {decisions} decisions identical; max |dp| {worst['probs']:.2e}, "+ f"max |dlogit| {worst['logits']:.2e}, max |dact| {worst['act_probability']:.2e}\n")+++if __name__ == "__main__":+ unittest.main()diff --git a/c/tools/clean.py b/c/tools/clean.pyindex b58a33c3..34558703 100644--- a/c/tools/clean.py+++ b/c/tools/clean.py@@ -41,6 +41,7 @@ FILES = ["deepseek_v41", "deepseek_v41.exe","mimo", "mimo.exe","qwenimage", "qwenimage.exe",+ "laya", "laya.exe","native_quant.o", "native_quant_parallel.o", "native_quant_dual.o","native_quant_batch_avx512.o", "native_quant_fp4_rows16.o",]diff --git a/c/tools/make_laya_ref.py b/c/tools/make_laya_ref.pynew file mode 100644index 00000000..483d3af0--- /dev/null+++ b/c/tools/make_laya_ref.py@@ -0,0 +1,289 @@+#!/usr/bin/env python3+"""Write laya_tiny/ref.json: what the `laya` package answers on the tiny fixture.++Maintainer-run, offline, once per change of the fixture: it needs torch,+transformers and the pinned `laya` package (pip install laya==0.3.24), which CI+does not. The answers come from Convai Innovations' own runtime, laya.Agent, on+the checkpoint tools/make_laya_tiny.py writes; the SHA-256 of that checkpoint is+recorded so tests/test_laya_tiny.py refuses a regenerated fixture that differs.++For every case it records, per question, what the engine must reproduce:+ ids, markers Agent._encode_state (common.build_sequence)+ logits DecisionModel.forward, the raw scorer output+ probs Agent._decode_answers' calibrated softmax, unrounded+ act_probability the act head's softmax, entry 0+and the package's own rounded answer (Agent.predict), against which the+unrounded numbers are checked here before anything is written. A case the+package refuses records the refusal instead. Every decision is checked for a+near-tie, so a float-level difference can never flip an argmax in the test.++ python tools/make_laya_tiny.py --output ./laya_tiny+ python tools/make_laya_ref.py --fixture ./laya_tiny+"""++import argparse+import hashlib+import json+import sys+from pathlib import Path++import numpy as np++PINNED_LAYA = "0.3.24"+MIN_GAP = 1e-3 # smallest top-1 / top-2 probability gap accepted++LONG_BODY = (+ "Hello team, I am writing again about the invoice we received on the first of the month. "+ "The amount is twice what we agreed in the contract and the payment was already taken from "+ "our card. We asked for a refund last week and nobody answered. Our finance department needs "+ "the corrected invoice before Friday or we will have to escalate this to our lawyers and stop "+ "using the service. Please also check the other invoices of this year, because we think the "+ "same mistake happened in March and in April. Thank you, the accounts team.")++CONVERSATION = [+ {"role": "user", "content": "Hi, I cannot log in to my account since yesterday."},+ {"role": "assistant", "content": "Sorry about that. Did you try to reset the password?"},+ {"role": "user", "content": "Yes, twice, and the reset email never arrives."},+ {"role": "assistant", "content": "Let me check the email address on the account."},+ {"role": "user", "content": "It is the same as always. This is really urgent, I have a "+ "payment due today and I need the invoices from the portal."},+ {"role": "assistant", "content": "I understand. I am escalating it now."},+ {"role": "user", "content": "If it is not fixed today I will cancel the subscription."},+]++DEPARTMENT = {"type": "choice", "instructions": "Which department should handle this request?",+ "criteria": {"billing": "invoices, payments, refunds",+ "technical": "bugs, outages, system errors",+ "sales": "pricing, new contracts", "other": "everything else"}}++CASES = [+ ("readme_ticket",+ {"from": "[email protected]", "subject": "Duplicate charge on invoice #4411",+ "body": "Hi, we were billed twice for March. Please refund the duplicate today or we will "+ "cancel our plan."},+ {"department": DEPARTMENT,+ "urgency": {"type": "score", "instructions": "How urgent is this request?",+ "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},+ "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},+ "refund_requested": {"type": "noul",+ "instructions": "Does the user explicitly request a refund?"}}),+ ("plain_two_options",+ "The application crashes every time I open the settings page.",+ {"is_bug": {"type": "choice", "instructions": "Is this a bug report or a feature request?",+ "criteria": {"bug": "something is broken", "feature": "something new is wanted"}},+ "blocking": {"type": "noul", "instructions": "Is the user blocked?",+ "criteria": {"true": "the user cannot work at all",+ "false": "the user can still work"}},+ "only_true": {"type": "noul", "instructions": "Does it mention settings?",+ "criteria": {"true": "the settings page is named"}}}),+ ("choice_seven_structured",+ "We would like to schedule a demo of the product for our team next week.",+ {"intent": {"type": "choice", "instructions": "What does the sender want?",+ "criteria": {"demo": "a product demonstration", "refund": None, "cancel": "",+ "pricing": {"desc": "price questions", "examples": ["cost", "plan"]},+ "support": 3, "partnership": ["co-marketing", "reselling"],+ "other": "anything else"}}}),+ ("choice_twelve_labels",+ "I love this product, it works great and the support team was very helpful.",+ {"emotion": {"type": "choice", "instructions": "Which emotion does the text express?",+ "criteria": {name: None for name in (+ "joy", "anger", "sadness", "fear", "surprise", "disgust", "trust",+ "anticipation", "neutral", "confusion", "gratitude", "pride")}}}),+ ("scores",+ "This is the worst service I have ever used, nothing works and nobody answers.",+ {"sentiment": {"type": "score", "instructions": "What is the sender's tone?",+ "criteria": ["angry", "negative", "neutral", "positive", "delighted"]},+ "six_levels": {"type": "score", "instructions": "How severe is the impact?",+ "criteria": ["none", "minor", "moderate", "major", "critical", "catastrophic"]},+ "two_levels": {"type": "score", "instructions": "Is any action needed?",+ "criteria": ["no action", "action needed"]},+ "numeric_levels": {"type": "score", "instructions": "Rate the urgency.",+ "criteria": [1, 2, 3]}}),+ ("long_state_truncated",+ {"subject": "Invoice problem again", "body": LONG_BODY},+ {"department": DEPARTMENT,+ "legal_threat": {"type": "noul", "instructions": "Does the sender threaten legal action?"}}),+ ("conversation_truncated_left", CONVERSATION,+ {"churn": {"type": "noul", "instructions": "Does the user threaten to cancel?"},+ "topic": {"type": "choice", "instructions": "What is the conversation about?",+ "criteria": {"login": "access to the account", "billing": "payments",+ "other": "anything else"}}}),+ ("json_instructions_and_state",+ {"ticket": {"id": 4411, "amount": 129.5, "currency": "EUR", "paid": True,+ "refund": None, "ratio": 1e-05, "big": 1e21, "tags": ["billing", "urgent"],+ "nested": {"a": [1, 2.0, {"b": "c"}]}}},+ {"action": {"type": "choice",+ "instructions": {"task": "pick the next action", "context": "support desk",+ "steps": ["read", "decide"]},+ "criteria": {"refund": "give the money back", "escalate": "a human decides",+ "close": "nothing to do"}},+ "paid": {"type": "noul", "instructions": ["Was", "the", "ticket", "paid?"]}}),+ ("special_text",+ "Order [MASK] arrived broken, see\tphoto.\r\n\r\nThe café [SEP] said |||IP_ADDRESS||| "+ "is blocked <|endoftext|> and [unused0] support said 'we're sorry'.",+ {"damaged": {"type": "noul", "instructions": "Was the item [MASK] damaged?"},+ "channel": {"type": "choice", "instructions": "Which channel should answer?",+ "criteria": {"email": "write an email [MASK] back",+ "phone": "call\tthe customer", "none": None}}}),+ ("long_option",+ "The meeting notes say that the contract will be renewed if the price stays the same.",+ {"renewal": {"type": "choice", "instructions": "Will the contract be renewed?",+ "criteria": {"yes": "the contract will be renewed " + "under the same terms " * 12,+ "no": "the contract ends", "unclear": "the notes do not say"}}}),+ ("many_options_shrunk",+ "Security incident: a phishing email asked employees to reset their passwords.",+ {"category": {"type": "choice",+ "instructions": "Classify the incident into exactly one of the categories "+ "below, considering the attack vector, the target and the "+ "likely impact on the organisation and its customers.",+ "criteria": {f"cat{i:02d}": f"category number {i} of the incident taxonomy"+ for i in range(20)}}}),+ ("too_many_options",+ "Too many options for the head budget.",+ {"bucket": {"type": "choice", "instructions": "Pick one.",+ "criteria": {f"option{i}": f"the description of option {i}" for i in range(40)}}}),+ ("unicode",+ "Buongiorno, la fattura è sbagliata. Grüße aus München, ευχαριστώ. "+ "我们的账户被锁了 \U0001F642\U0001F44D\U0001F3FD Å Ω",+ {"language": {"type": "choice", "instructions": "Which language comes first?",+ "criteria": {"italian": "italiano", "german": "deutsch", "chinese": "中文"}},+ "polite": {"type": "noul", "instructions": "Is the message polite?"}}),+]++TOKENIZER_STRINGS = [+ "", " ", " ", " ", "\n\n", "\t", "a b c d e", "Hello world",+ "don't I'll we've they're she'd 'S 'T", "2026-10-02 3.14 1000000 12345678901",+ "café café Å Ω Å", "日本語のテキスト",+ "\U0001F642\U0001F44D\U0001F3FD", "[SEP] in [CLS] text [PAD][UNK]",+ "|||IP_ADDRESS||| and |||EMAIL_ADDRESS|||, |||PHONE_NUMBER|||", "<|endoftext|><|padding|>",+ "[unused0]x[unused1]", "tab\tand\r\nnewline\n", " leading spaces", "trailing spaces ",+ "mixed ,.;:!? punctuation!!! ... ???", "über straße Ελληνικά "+ "русский عربي", "x" * 300,+ "{\"subject\": \"Duplicate charge\", \"body\": \"Please fix this\"}",+ "level 0: not urgent", " choice question: Which department?",+]+++def compact_json(ref):+ """One key per line at the top, one case or tokenizer string per line below+ it: diffable, without a line per token id."""+ def dump(value):+ return json.dumps(value, ensure_ascii=False, separators=(",", ":"))+ lines = ["{"]+ keys = list(ref)+ for i, key in enumerate(keys):+ tail = "," if i + 1 < len(keys) else ""+ value = ref[key]+ if isinstance(value, list):+ lines.append(f" {dump(key)}:[")+ lines.extend(f" {dump(item)}{',' if j + 1 < len(value) else ''}"+ for j, item in enumerate(value))+ lines.append(f" ]{tail}")+ else:+ lines.append(f" {dump(key)}:{dump(value)}{tail}")+ lines.append("}")+ return "\n".join(lines) + "\n"+++def sha256(path):+ digest = hashlib.sha256()+ with open(path, "rb") as handle:+ for block in iter(lambda: handle.read(1 << 20), b""):+ digest.update(block)+ return digest.hexdigest()+++def main():+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])+ parser.add_argument("--fixture", default="./laya_tiny")+ parser.add_argument("--allow-version", action="store_true",+ help=f"run with an installed laya other than {PINNED_LAYA}")+ args = parser.parse_args()++ import torch+ import transformers+ import laya+ from laya.common import QTYPES, collate_items, temp_bucket++ if laya.__version__ != PINNED_LAYA and not args.allow_version:+ sys.exit(f"laya {laya.__version__} is installed; the reference is pinned to {PINNED_LAYA}")+ torch.manual_seed(0)+ fixture = Path(args.fixture)+ agent = laya.load(str(fixture), device="cpu")+ pad = agent.tok.pad_token_id++ cases = []+ for name, state, questions in CASES:+ case = {"name": name, "state": state, "questions": questions}+ ids = list(questions)+ try:+ for qid in ids:+ agent._check_question(qid, questions[qid])+ internal = {qid: agent._to_internal(questions[qid]) for qid in ids}+ items = agent._encode_state(state, ids, internal)+ except ValueError as error:+ case["error"] = str(error)+ cases.append(case)+ print(f"{name}: refused ({error})", file=sys.stderr)+ continue+ batch = collate_items([items], pad)+ with torch.no_grad():+ logits, act = agent._forward(batch)+ predicted = agent.predict(state, questions)+ answers = []+ for j, qid in enumerate(ids):+ q = internal[qid]+ k = len(items[j]["markers"])+ qt = QTYPES[q["t"]]+ # Agent._decode_answers, unrounded+ t_scale = agent.temperature_by_options.get(temp_bucket(qt, k), agent.temperature[qt])+ z = logits[j, :k] / t_scale+ p = np.exp(z - z.max())+ p = p / p.sum()+ order = np.sort(p)[::-1]+ gap = float(order[0] - order[1]) if k > 1 else 1.0+ if gap < MIN_GAP:+ sys.exit(f"{name}.{qid}: top-2 gap {gap:.2e} is a near-tie; change the case or SEED")+ got = predicted["answers"][qid]+ reference = (got["probabilities"].values() if q["t"] != "noul" else [1 - got["noul"], got["noul"]])+ if q["t"] != "noul" and any(abs(round(float(a), 4) - b) > 1e-12 for a, b in zip(p, reference)):+ sys.exit(f"{name}.{qid}: unrounded probabilities disagree with Agent.predict")+ if q["t"] == "noul" and abs(round(float(p[1]), 4) - got["noul"]) > 1e-12:+ sys.exit(f"{name}.{qid}: unrounded noul disagrees with Agent.predict")+ answers.append({+ "id": qid, "ids": [int(x) for x in items[j]["ids"]],+ "markers": [int(x) for x in items[j]["markers"]],+ "logits": [float(x) for x in logits[j, :k]],+ "probs": [float(x) for x in p], "temperature": float(t_scale),+ "act_probability": float(act[j, 0]),+ "state_tokens": items[j]["state_stats"]["state_tokens"],+ "state_dropped": items[j]["state_stats"]["state_tokens_dropped"],+ })+ case["answers"] = answers+ case["predict"] = predicted+ cases.append(case)+ print(f"{name}: {len(answers)} question(s), {sum(len(a['ids']) for a in answers)} tokens",+ file=sys.stderr)++ tok = agent.tok+ tokenizer = [{"text": text, "ids": tok(text, add_special_tokens=False)["input_ids"]}+ for text in TOKENIZER_STRINGS]+ ref = {+ "generator": "tools/make_laya_ref.py",+ "reference": {"laya": laya.__version__, "torch": torch.__version__,+ "transformers": transformers.__version__},+ "fixture_sha256": {name: sha256(fixture / name) for name in (+ "model.safetensors", "tokenizer/tokenizer.json", "encoder/config.json",+ "rl_agent_config.json")},+ "tolerance": {"probs": 2e-5, "logits": 2e-4, "act_probability": 2e-4},+ "tokenizer": tokenizer,+ "cases": cases,+ }+ (fixture / "ref.json").write_text(compact_json(ref), encoding="utf-8")+ print(f"wrote {fixture / 'ref.json'}: {len(cases)} cases, {len(tokenizer)} tokenizer strings",+ file=sys.stderr)+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/c/tools/make_laya_tiny.py b/c/tools/make_laya_tiny.pynew file mode 100644index 00000000..29492327--- /dev/null+++ b/c/tools/make_laya_tiny.py@@ -0,0 +1,336 @@+#!/usr/bin/env python3+"""Generate the tiny Laya checkpoint used by tests/test_laya_tiny.py.++numpy-only and deterministic: CI regenerates it from SEED and gets byte-identical+files on every platform. The reference probabilities in laya_tiny/ref.json are+NOT produced here: they come from the `laya` package itself (Convai Innovations'+reference implementation), run offline by tools/make_laya_ref.py, so the engine+and its oracle never share an author.++The fixture has the layout of the release (convaiinnovations/laya):++ rl_agent_config.json head_layers, max_len, head_max_len, act_costs,+ temperature and temperature_by_options (the+ release's values, so choice:11+ is clamped)+ encoder/config.json ModernBERT, transformers 5 field names+ tokenizer/tokenizer.json byte-level BPE, NFC, GPT-2 regex, added tokens+ tokenizer/tokenizer_config.json+ model.safetensors F16 like the release, temperature in F32++and a geometry that crosses every path of laya.c: a global layer every three+(0 and 3) with RoPE theta 160000 and sliding layers with theta 10000 and a+window of +-8 that every sequence outgrows; two decision-head layers with two+heads (hidden 128 = 2 x 64, the d // 64 rule); max_len 160 and head_max_len 64,+so long states are cut and many options are shrunk.++The tokenizer is trained here, by a small deterministic BPE over the corpus+below, and carries the added tokens of the release: the [CLS]/[SEP]/[MASK]+family, runs of spaces (matched before BPE), and the |||...||| placeholders.+"""++import argparse+import json+import re+import struct+import sys+from pathlib import Path++import numpy as np++SEED = 20261002++HIDDEN = 128+LAYERS = 4+HEADS = 4+INTER = 160+LOCAL_ATTENTION = 16 # the window is local_attention // 2 = 8 each side+MAX_POS = 512+VOCAB = 640 # embedding rows; the tokenizer uses fewer+HEAD_LAYERS = 2+ACT_HIDDEN = 256+MERGES = 320++RL_AGENT_CONFIG = {+ "encoder": "answerdotai/ModernBERT-large",+ "head_layers": HEAD_LAYERS,+ "max_len": 160,+ "head_max_len": 64,+ "max_prefixes": 6,+ "act_costs": {"escalate": 0.5},+ "cost_wrong_act": 3.0,+ "amp_dtype": "bf16",+ "model_name": "laya-tiny",+ # the release's calibration: choice:11+ is 0.1006, which the reference+ # clamps to 0.5; score:6-10 and score:2 are absent and fall back to+ # temperature[1]+ "temperature": [1.6369030475616455, 1.2514300346374512, 1.983399510383606],+ "temperature_by_options": {+ "choice:3-5": 1.7601518630981445,+ "choice:6-10": 1.0000158548355103,+ "score:3-5": 1.2514300346374512,+ "noul:2": 1.983399510383606,+ "choice:11+": 0.10058280825614929,+ "choice:2": 1.9063563346862793,+ },+}++CORPUS = """+Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan.+The application crashes every time I open the settings page. This is blocking our release.+Which department should handle this request? How urgent is this request?+Does the user threaten to cancel or leave? Does the user explicitly request a refund?+choice question: score question: noul question: level 0: level 1: level 2: level 3:+false: no, the statement does not hold. true: yes, the statement holds.+billing invoices payments refunds technical bugs outages system errors sales pricing new contracts+other everything else not urgent soon critical deadline or blocking issue+The customer wrote an email about the invoice, the payment and the subscription.+We would like to schedule a demo of the product for our team next week.+Our account was locked and we cannot log in since yesterday morning.+I love this product, it works great and the support team was very helpful.+This is the worst service I have ever used, nothing works and nobody answers.+The quick brown fox jumps over the lazy dog while the cat sleeps in the sun.+Numbers like 2026, 42 and 3.14 appear in tickets, as do dates like 2026-10-02.+{"subject": "Duplicate charge", "body": "Please fix this", "from": "[email protected]"}+The meeting notes say that the contract will be renewed if the price stays the same.+Security incident: a phishing email asked employees to reset their passwords.+"""++# GPT-2's map from bytes to printable code points (the ByteLevel alphabet)+def bytes_to_unicode():+ bs = (list(range(ord("!"), ord("~") + 1)) + list(range(ord("\xa1"), ord("\xac") + 1)) ++ list(range(ord("\xae"), ord("\xff") + 1)))+ cs = bs[:]+ n = 0+ for b in range(256):+ if b not in bs:+ bs.append(b)+ cs.append(256 + n)+ n += 1+ return dict(zip(bs, [chr(c) for c in cs]))+++def train_bpe(corpus, merges):+ """A plain BPE: count adjacent pairs over the words, merge the most frequent,+ ties broken by the pair's text so every platform learns the same merges."""+ byte_map = bytes_to_unicode()+ words = {}+ for piece in re.findall(r" ?[A-Za-z]+| ?[0-9]+| ?[^\sA-Za-z0-9]+|\s+", corpus):+ symbols = tuple(byte_map[b] for b in piece.encode("utf-8"))+ words[symbols] = words.get(symbols, 0) + 1+ learned = []+ for _ in range(merges):+ counts = {}+ for symbols, freq in words.items():+ for pair in zip(symbols, symbols[1:]):+ counts[pair] = counts.get(pair, 0) + freq+ if not counts:+ break+ best = min(counts, key=lambda pair: (-counts[pair], pair))+ if counts[best] < 2:+ break+ learned.append(best)+ merged = {}+ for symbols, freq in words.items():+ out, i = [], 0+ while i < len(symbols):+ if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == best:+ out.append(symbols[i] + symbols[i + 1])+ i += 2+ else:+ out.append(symbols[i])+ i += 1+ merged[tuple(out)] = merged.get(tuple(out), 0) + freq+ words = merged+ return byte_map, learned+++def added(content, token_id, special, normalized, lstrip=False):+ return {"id": token_id, "content": content, "single_word": False, "lstrip": lstrip,+ "rstrip": False, "normalized": normalized, "special": special}+++def build_tokenizer():+ byte_map, merges = train_bpe(CORPUS, MERGES)+ vocab = {"|||IP_ADDRESS|||": 0, "<|padding|>": 1}+ for b in range(256):+ vocab[byte_map[b]] = len(vocab)+ for left, right in merges:+ if left + right not in vocab:+ vocab[left + right] = len(vocab)+ runs = [" " * n for n in range(8, 1, -1)] # longest first, as the release+ added_tokens = [added("|||IP_ADDRESS|||", 0, False, True), added("<|padding|>", 1, True, False)]+ for run in runs:+ vocab[run] = len(vocab)+ added_tokens.append(added(run, vocab[run], False, True))+ for name in ("|||EMAIL_ADDRESS|||", "|||PHONE_NUMBER|||"):+ vocab[name] = len(vocab)+ added_tokens.append(added(name, vocab[name], False, True))+ vocab["<|endoftext|>"] = len(vocab)+ added_tokens.append(added("<|endoftext|>", vocab["<|endoftext|>"], True, False))+ next_id = len(vocab)+ specials = {}+ for name in ("[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"):+ specials[name] = next_id+ added_tokens.append(added(name, next_id, True, False, lstrip=name == "[MASK]"))+ next_id += 1+ for i in range(2):+ added_tokens.append(added(f"[unused{i}]", next_id, False, True))+ next_id += 1+ if next_id > VOCAB:+ raise SystemExit(f"tokenizer needs {next_id} ids, the model has {VOCAB}")+ tokenizer = {+ "version": "1.0",+ "truncation": None,+ "padding": None,+ "added_tokens": added_tokens,+ "normalizer": {"type": "NFC"},+ "pre_tokenizer": {"type": "ByteLevel", "add_prefix_space": False, "trim_offsets": True,+ "use_regex": True},+ "post_processor": {+ "type": "TemplateProcessing",+ "single": [{"SpecialToken": {"id": "[CLS]", "type_id": 0}},+ {"Sequence": {"id": "A", "type_id": 0}},+ {"SpecialToken": {"id": "[SEP]", "type_id": 0}}],+ "pair": [{"SpecialToken": {"id": "[CLS]", "type_id": 0}},+ {"Sequence": {"id": "A", "type_id": 0}},+ {"SpecialToken": {"id": "[SEP]", "type_id": 0}},+ {"Sequence": {"id": "B", "type_id": 0}},+ {"SpecialToken": {"id": "[SEP]", "type_id": 0}}],+ "special_tokens": {name: {"id": name, "ids": [token], "tokens": [name]}+ for name, token in specials.items()},+ },+ "decoder": {"type": "ByteLevel", "add_prefix_space": False, "trim_offsets": True,+ "use_regex": True},+ "model": {"type": "BPE", "dropout": None, "unk_token": None,+ "continuing_subword_prefix": None, "end_of_word_suffix": None,+ "fuse_unk": False, "byte_fallback": False, "ignore_merges": False,+ "vocab": vocab, "merges": [[left, right] for left, right in merges]},+ }+ config = {"clean_up_tokenization_spaces": True, "cls_token": "[CLS]", "mask_token": "[MASK]",+ "model_input_names": ["input_ids", "attention_mask"], "model_max_length": MAX_POS,+ "pad_token": "[PAD]", "sep_token": "[SEP]",+ "tokenizer_class": "PreTrainedTokenizerFast", "unk_token": "[UNK]"}+ return tokenizer, config, specials+++def encoder_config(specials):+ layer_types = ["full_attention" if i % 3 == 0 else "sliding_attention" for i in range(LAYERS)]+ return {+ "architectures": ["ModernBertForMaskedLM"],+ "attention_bias": False, "attention_dropout": 0.0,+ "bos_token_id": specials["[CLS]"], "cls_token_id": specials["[CLS]"],+ "eos_token_id": specials["[SEP]"], "sep_token_id": specials["[SEP]"],+ "pad_token_id": specials["[PAD]"],+ "classifier_activation": "gelu", "classifier_bias": False, "classifier_dropout": 0.0,+ "classifier_pooling": "mean", "decoder_bias": True, "deterministic_flash_attn": False,+ "dtype": "float32", "embedding_dropout": 0.0, "global_attn_every_n_layers": 3,+ "gradient_checkpointing": False, "hidden_activation": "gelu", "hidden_size": HIDDEN,+ "initializer_cutoff_factor": 2.0, "initializer_range": 0.02,+ "intermediate_size": INTER, "layer_norm_eps": 1e-05, "layer_types": layer_types,+ "local_attention": LOCAL_ATTENTION, "max_position_embeddings": MAX_POS,+ "mlp_bias": False, "mlp_dropout": 0.0, "model_type": "modernbert", "norm_bias": False,+ "norm_eps": 1e-05, "num_attention_heads": HEADS, "num_hidden_layers": LAYERS,+ "position_embedding_type": "absolute", "repad_logits_with_grad": False,+ "rope_parameters": {"full_attention": {"rope_theta": 160000.0, "rope_type": "default"},+ "sliding_attention": {"rope_theta": 10000.0, "rope_type": "default"}},+ "sparse_pred_ignore_index": -100, "sparse_prediction": False,+ "tie_word_embeddings": True, "transformers_version": "5.0.0", "vocab_size": VOCAB,+ }+++def build_tensors(rng):+ """Random weights at scales that keep every layer contributing and the logits+ of a question a few units apart, so the decisions are not near-ties."""+ d, I = HIDDEN, INTER+ tensors = {}++ def normal(shape, scale):+ return (rng.standard_normal(shape) * scale).astype(np.float32)++ def norm_weight(n):+ return (1.0 + 0.1 * rng.standard_normal(n)).astype(np.float32)++ def put(name, array):+ tensors[name] = array.astype(np.float16)++ put("encoder.embeddings.tok_embeddings.weight", normal((VOCAB, d), 1.0))+ put("encoder.embeddings.norm.weight", norm_weight(d))+ for layer in range(LAYERS):+ prefix = f"encoder.layers.{layer}."+ if layer:+ put(prefix + "attn_norm.weight", norm_weight(d))+ put(prefix + "attn.Wqkv.weight", normal((3 * d, d), 1.0 / np.sqrt(d)))+ put(prefix + "attn.Wo.weight", normal((d, d), 0.5 / np.sqrt(d)))+ put(prefix + "mlp_norm.weight", norm_weight(d))+ put(prefix + "mlp.Wi.weight", normal((2 * I, d), 1.0 / np.sqrt(d)))+ put(prefix + "mlp.Wo.weight", normal((d, I), 0.5 / np.sqrt(I)))+ put("encoder.final_norm.weight", norm_weight(d))+ for layer in range(HEAD_LAYERS):+ prefix = f"head.layers.{layer}."+ put(prefix + "self_attn.in_proj_weight", normal((3 * d, d), 1.0 / np.sqrt(d)))+ put(prefix + "self_attn.in_proj_bias", normal(3 * d, 0.05))+ put(prefix + "self_attn.out_proj.weight", normal((d, d), 0.5 / np.sqrt(d)))+ put(prefix + "self_attn.out_proj.bias", normal(d, 0.05))+ put(prefix + "linear1.weight", normal((4 * d, d), 1.0 / np.sqrt(d)))+ put(prefix + "linear1.bias", normal(4 * d, 0.05))+ put(prefix + "linear2.weight", normal((d, 4 * d), 0.5 / np.sqrt(4 * d)))+ put(prefix + "linear2.bias", normal(d, 0.05))+ put(prefix + "norm1.weight", norm_weight(d))+ put(prefix + "norm1.bias", normal(d, 0.1))+ put(prefix + "norm2.weight", norm_weight(d))+ put(prefix + "norm2.bias", normal(d, 0.1))+ put("type_emb.weight", normal((3, d), 0.5))+ put("scorer.0.weight", norm_weight(d))+ put("scorer.0.bias", normal(d, 0.1))+ put("scorer.1.weight", normal((d, d), 1.0 / np.sqrt(d)))+ put("scorer.1.bias", normal(d, 0.1))+ put("scorer.3.weight", normal((1, d), 3.0 / np.sqrt(d)))+ put("scorer.3.bias", normal(1, 0.1))+ put("act_head.0.weight", normal((ACT_HIDDEN, d + 4), 1.0 / np.sqrt(d + 4)))+ put("act_head.0.bias", normal(ACT_HIDDEN, 0.1))+ put("act_head.2.weight", normal((2, ACT_HIDDEN), 3.0 / np.sqrt(ACT_HIDDEN)))+ put("act_head.2.bias", normal(2, 0.1))+ tensors["temperature"] = np.ones(3, dtype=np.float32)+ return tensors+++def write_safetensors(path, tensors):+ header, offset = {}, 0+ for name, array in tensors.items():+ dtype = {"float16": "F16", "float32": "F32"}[array.dtype.name]+ header[name] = {"dtype": dtype, "shape": list(array.shape),+ "data_offsets": [offset, offset + array.nbytes]}+ offset += array.nbytes+ blob = json.dumps(header, separators=(",", ":")).encode()+ blob += b" " * ((-len(blob)) % 8)+ with open(path, "wb") as handle:+ handle.write(struct.pack("<Q", len(blob)))+ handle.write(blob)+ for array in tensors.values():+ handle.write(np.ascontiguousarray(array).view(np.uint8).tobytes())+++def write_json(path, value):+ path.write_text(json.dumps(value, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")+++def main():+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])+ parser.add_argument("--output", default="./laya_tiny")+ args = parser.parse_args()+ out = Path(args.output)+ (out / "encoder").mkdir(parents=True, exist_ok=True)+ (out / "tokenizer").mkdir(parents=True, exist_ok=True)+ tokenizer, tokenizer_config, specials = build_tokenizer()+ write_json(out / "tokenizer" / "tokenizer.json", tokenizer)+ write_json(out / "tokenizer" / "tokenizer_config.json", tokenizer_config)+ write_json(out / "encoder" / "config.json", encoder_config(specials))+ write_json(out / "rl_agent_config.json", RL_AGENT_CONFIG)+ write_safetensors(out / "model.safetensors", build_tensors(np.random.default_rng(SEED)))+ print(f"wrote {out}: {len(tokenizer['model']['vocab'])} vocab entries, "+ f"{len(tokenizer['model']['merges'])} merges", file=sys.stderr)+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/site/index.html b/site/index.htmlindex d4cbd579..7bfe3983 100644--- a/site/index.html+++ b/site/index.html@@ -436,7 +436,8 @@ footer{border-top:1px solid var(--line)}GLM-5.3-Flash, GLM-5.2/5.3, Inkling, Kimi K3, OLMoE, Qwen3.6-35B-A3B (which also runsQwen3-Coder-30B-A3B and the dense Qwen3.8-27B), Qwen3.8-Flash-Next, DeepSeek V4 Flash,DeepSeek V4.1 Flash, MiMo-V2.6 Flash (and Pro, on the same engine).- And for images, Qwen-Image-2.1: text to picture, drawn inside the terminal.</p>+ And for images, Qwen-Image-2.1: text to picture, drawn inside the terminal.+ And for decisions, Laya: typed questions in, calibrated probabilities out.</p><a class="more" href="#models">Browse all models →</a></div>