Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

23.5 KB
Raw Blame

laya-ts

TypeScript inference for Laya (Agent.predict, Router, lang, email, presets, shortlist, hooks) on Node and the browser via split ONNX (encoder.onnx + head.onnx). ESM-only ("type": "module"); no CJS build — import from ESM or bundle.

Export weights (once per checkpoint)

python laya-ts/scripts/export_onnx.py --model-dir <ckpt> --out-dir ./model
# writes encoder.onnx, head.onnx + copies tokenizer.json, rl_agent_config.json
# verifies torch vs ONNX match within 1e-4 (skip with --no-verify)

Node (CPU/CUDA)

import { Agent, Router } from "laya-ts";

const agent = await Agent.load("./model"); // local dir, or ("convaiinnovations/laya", { subfolder: "multilingual" })
const router = new Router({ default: "english" }); // this example attaches only the English checkpoint
router.attach("english", agent);
const out = await router.predict({ body: "charged twice, refund please" }, {
  intent: { type: "choice", instructions: "What does the customer want?", criteria: { refund: "money back", other: "anything else" } },
});
console.log(out.answers.intent);

CUDA: Agent.load("./model", { device: "cuda" }) (falls back to CPU with a warning).

new Router() uses the multilingual checkpoint for text whose language the detector cannot identify, including text with no letters, matching Python 0.4.0. Identified English still routes to the English checkpoint, and detected non-Latin scripts route to multilingual. Set new Router({ default: "english" }) to restore the earlier fallback; explicit model and lang options still take precedence.

Browser (WebGPU → WASM fallback)

import { Agent } from "laya-ts";

const agent = await Agent.load("https://example.com/models/laya"); // serves encoder.onnx, head.onnx, tokenizer.json, rl_agent_config.json
const out = await agent.predict("charged twice", {
  d: { type: "choice", instructions: "pick", criteria: { refund: "money back", other: "rest" } },
});

onnxruntime-node / onnxruntime-web are optional peer deps, imported lazily behind the provider you use.

Long documents (predictLong)

Agent.predictLong and Router.predictLong scan overlapping token windows locally, in Node or the browser. This lets a question consider a long ticket or conversation beyond the single window used by predict. The Router chooses and loads a checkpoint once for the document.

const scanned = await router.predictLong(longTicket, {
  refund: { type: "noul", instructions: "Does the customer request a refund?" },
  action: { type: "choice", instructions: "Choose the next action", criteria: ["refund", "investigate"] },
}, { window: 256, stride: 128, batchSize: 4 });
console.log(scanned.answers.refund, scanned.usage.windows, scanned.routing);

The only aggregation mode is "auto": noul takes the largest value; choice and score take the answer with the highest answer_confidence. Ties retain the first result. The winner keeps its metadata. Its confidence is a window score, not calibrated confidence for the whole document; scanning more windows can change which answer wins.

The window is capped by the space left by the questions in the checkpoint's configured sequence budget. Its default is max(64, max_len - head_max_len - 8), then capped; stride defaults to half that effective window. An explicit stride cannot leave gaps. window, stride, and batchSize must be positive safe integers when supplied; null/undefined use defaults. maxLen, headMaxLen, and minConfidence are deliberately absent from the long options. Window sizing and aggregation have different meanings from single-pass truncation and confidence gating. Ordinary prediction options and result types remain unchanged.

Windows follow Python 0.4.18's decode → batch prediction → re-encode path. The official English and multilingual tokenizers support decoding. A custom encode-only tokenizer still supports ordinary predictions and the short-input fast path; splitting requires decode and throws if it is unavailable. Decoding and re-encoding can change the token count, particularly for multilingual text, so normal truncation can still occur. An answer's optional window.token_start / token_end identifies its original candidate token slice, not a guarantee that exactly those tokens reached the model.

Agent hooks wrap one window batch; Router hooks wrap one document. Process defaults run once. Reordering result objects preserves identifiable origins, while rewriting/reordering states, replacing an answer object, or editing prediction fields removes window attribution. In-place reliability/action annotations preserve it. An end hook may uniformly remove its own temporary questions, but cannot silently remove a caller's question. Inconsistent answer schemas or window counts fail. A skipping hook must return exactly one document result; the result and usage are copied, a warning is emitted, and usage.windows is zero. The warning also covers short-input and Router cache hits, as agreed in #1316; Python 0.4.18 itself only warns on the multi-window Agent skip path. Hook results are validated against the TypeScript return contract, which also rejects malformed payloads Python accepts.

If a start hook changes questions or token budgets so planned windows no longer fit, the scan is refused after inference. This check is outside hook error suppression and hook timeouts; the forward-pass cost has already been incurred. The short path checks this too. LongResult / RoutedLongResult have separate usage types: truncated is a boolean on the short path and a numeric count on the multi-window path. Numeric usage is summed, dictionaries are shallow-merged, other values come from the last result, and output_tokens stays zero.

Deterministic parity fixtures are recorded directly from Python ONNXAgent.predict_long at v0.4.18, with source hashes, and replayed by TypeScript tests. The generator's --check detects later reference behavior changes; these checks establish control-flow parity, not model quality or a speed improvement.

To repeat the optional real-inference comparison with an English checkpoint and its split ONNX export (weights are not downloaded by these scripts):

# From laya-ts/, with numpy, torch, transformers and onnxruntime installed in Python.
python scripts/predict_long_onnx_parity.py --python-root /path/to/laya-v0.4.18 \
  --checkpoint /path/to/english-checkpoint --bundle /path/to/split-onnx --output python-long.json
npm run build
node scripts/predict_long_onnx_parity.mjs python-long.json comparison.json

The Python harness calls ONNXAgent.predict_long with an adapter joining the same encoder and head graphs used by TypeScript. It records source/artifact hashes and checks all three answer types on short, overlapping long, and structured inputs. The Node harness verifies artifact digests and compares actual token/marker/mask inputs, decoded answers, attribution, and usage at absolute tolerance 1e-4, excluding the existing runtime model-name difference. It does not measure retrieval quality, accuracy, or performance, and does not cover browser GPU numerical differences.

Hooks (observe or shape every decision)

Port of the Python laya.hooks lifecycle. A hook is a (ctx) => void for onPredictStart / onPredictEnd, or an object implementing any subset of onPredictStart, onPredictEnd, onRoute, onLoad, onEvict, onError. A hook may be async: it is awaited, in order, before the call continues, and a rejection follows hooksRaise like a thrown error. onRoute runs inside the synchronous route(), so it is not awaited and a rejection there is only logged:

const tracer = {
  onPredictStart(ctx) { console.time(ctx.runId); },
  onPredictEnd(ctx) { console.timeEnd(ctx.runId); console.log(ctx.model, ctx.usage, ctx.elapsedMs); },
};
const router = new Router({ hooks: [tracer], hooksRaise: false }); // telemetry must not fail a request
await router.withHooks([auditHook], () => router.predict(state, questions)); // scoped install

// a start hook may rewrite ctx.states / ctx.questions, or serve a cached result:
const cache = { onPredictStart(ctx) { const hit = lookup(ctx.states[0]); if (hit) ctx.skip([hit]); } };
// an onRoute hook may replace ctx.decision (e.g. pin a checkpoint)

// subclass BaseHook to override only the events you need:
class MetricsHook extends BaseHook {
  onPredictEnd(ctx) { record(ctx.usage); }
}

// process-wide defaults run before installed and per-call hooks for every Agent/Router,
// so a tracer or metrics hook does not have to be threaded through every construction:
setDefaultHooks([new MetricsHook()]);  // addDefaultHook(...) appends; clearDefaultHooks() resets

Option stability

OptionStabilityHook adds opt-in reliability diagnostics to choice answers. It expands the questions with reorder and letter-rename probes and evaluates them through the existing batch path. The original answer stays in place; internal probe answers and their usage keys are removed before the result returns. Extra rows cost inference work even within one call.

import { Agent, OptionStabilityHook, calibrateReliability, decideReliability } from "laya-ts";

const agent = await Agent.load("./dist/laya");
agent.addHook(new OptionStabilityHook({
  accept: null, escalate: null, variants: "fast", seed: "evaluation-v1",
}));
const result = await agent.predict("Please approve the request", {
  action: { type: "choice", instructions: "Choose action", criteria: ["approve", "reject"] },
});
if (result.answers.action.type === "choice") {
  console.log(result.answers.action.reliability);
}

// Fit on your labelled examples, using only probed numeric soft_stability values.
// Keep these scores and cutoffs as numbers; round only a separate display string.
function fitAndReplay(scores: number[], correct: boolean[], savedScore: number) {
  const [accept, escalate] = calibrateReliability(scores, correct, { targetAccuracy: 0.9 });
  return decideReliability(savedScore, accept, escalate);
}

Both thresholds are required. accept: null disables categorical decisions; escalate: null disables only ESCALATE. No calibrated defaults are recommended. ACCEPT uses soft_stability >= accept; ESCALATE uses soft_stability < escalate; otherwise the decision is VERIFY. An unprobed question has null signals. The report keeps the unrounded mean used by the comparator, matching the Python precision correction in #1289. Other diagnostic fractions retain Python's four-decimal rounding. Historical rounded Python reports cannot recover the lost precision.

Calibration requires equally sized, nonempty score/boolean-label arrays, finite scores in [0, 1], and accuracy targets in [0, 1]. Invalid evidence is rejected rather than silently dropping unmatched rows. The helpers match Python on valid inputs; they do not copy its permissive handling of invalid calibration inputs.

Options use camelCase (nShuffles, renameMaxOptions); result fields match Python's snake_case fields. variants accepts "fast", "full", or an array of variant names; probes accepts "auto", "reorder", "rename", "all", or a probe-name array. Renaming retains original label text for missing descriptions, including blank strings, as in #1286, and preserves nonblank descriptions and caller option_order.

Explicit string seeds and the same ordered labels produce the same permutations as Python's CRC32-seeded shuffle. JavaScript enumerates integer-like object keys numerically, so preserve the intended label order when comparing inputs across languages. They reproduce probe selection, not model outputs. The default seed uses the single string state, or "batch" for multiple states. A single structured state uses the existing TS state serializer; Python uses repr, so supply the same explicit seed for cross-language replay. The shared fixtures record their Python version and source hashes. Numerical comparison is for those decoded inputs; different inference backends or Python floating-point accumulation versions can still produce different scores near a threshold.

The hook reads agent.cfg.option_layout to choose probes automatically: a declared parallel layout selects rename-only probes; sequential selects both. This is probe selection, not an implementation of parallel attention. The current local ONNX runtime supports the sequential path; this hook adds no new checkpoint/backend support. It also does not add the feature to the separate HTTP SDK.

Install the hook once at the Agent, Router, or process-default scope. predictBatch and concurrent calls can share one instance; plans are scoped to each prediction context and released on completion or error. A cache hit without probe answers does not gain a fabricated reliability score. The hook's own methods are synchronous and work within the existing async Agent/Router prediction lifecycle.

Structured decisions (decide)

Turn a JSON schema into typed values in one call — the port of Python's laya.structured (#280). Enum properties become choice questions, booleans become noul, bounded integers become scores; anything the fixed-option model cannot answer (free strings, arrays, nested objects, $ref) is rejected with a SchemaError naming the path:

import { Agent, decide } from "laya-ts";

const agent = await Agent.load("./dist/laya");
const values = await agent.decide(ticketText, {
  type: "object",
  properties: {
    department: { type: "string", enum: ["billing", "support", "sales"] },
    urgency: { type: "integer", minimum: 0, maximum: 2 },
    needs_human: { type: "boolean" },
  },
});
// { department: "billing", urgency: 2, needs_human: false }

router.decide(...) works the same way (routing options are forwarded to predict), and the free decide(runner, state, schema, opts) accepts anything with a predict method. Pass { returnDetails: true } for per-field confidence, answerConfidence (calibrated max(p)), and probabilities, or { questions } instead of a schema to get raw answers.

For high throughput over multiple inputs, agent.decideBatch(states, schema) (and router.decideBatch, or the free decideBatch(runner, states, schema, opts)) plans the schema once and evaluates every state through shared forward passes on predictBatch (mirroring Python's laya.decide_batch).

Zod/TypeBox users can pass z.toJSONSchema(Model) — any object with a toJSONSchema() method is accepted. planFromJsonSchema, questionsFromJsonSchema and answersToJson expose the planning and projection steps.

Truncation reporting

The state is clamped to whatever token room a question's head leaves, and that budget moves with max_len, head_max_len and the rendered head — so usage reports it instead of letting you guess from character counts (issue #174; mirrors Python #181):

const out = await agent.predict(longState, questions);
out.usage.truncated;             // true when any state token was dropped
out.usage.state_tokens;          // encoded length of the full state
out.usage.state_tokens_dropped;  // worst case across the questions
out.usage.truncated_questions;   // ids of the questions whose head left too little room

The fields are absent only where no state was ever encoded (empty question schema, or a start hook that supplied the result).

Collapsed options

The same budget also cuts the options themselves: each is capped at 48 tokens, and once they overflow head_max_len all of them are re-capped at max(4, (head_max_len - 16) // n). Two options that share a prefix can come out of that cut as the same token span, so the question can no longer name them apart while still answering normally — and an answer chosen from 42 distinguishable spans of 58 has an accuracy ceiling of 72% that nothing else in the response mentions (issue #538; mirrors Python laya.common.collapsed_options):

out.usage.options;   // absent when every option kept a span of its own
out.usage.options?.intent;  // { total: 58, distinct: 42, tokens_per_option: 4 }

total is what the question defines, not the markers that reached the sequence, so a report cannot read "43 of 43" about a question whose missing options never entered the input at all.

Batching (many states, one call)

Port of Agent.predict_batch / Router.route_batch / Router.predict_batch. The throughput path: states that share a question schema are collated into one shared forward pass (or one per batchSize chunk) instead of one pass per state.

// Agent: same questions over many states; results align with `states` by index.
const results = await agent.predictBatch(states, questions, { batchSize: 32 });

// Router: heterogeneous requests — route first, then each checkpoint scores its requests
// in as few forward passes as possible. Results keep input order and carry `routing`.
const decisions = router.routeBatch(requests);          // validate + route, nothing loads
const routed = await router.predictBatch(requests, 32); // == router.predictMany(...)
// Token budgets belong to each request; omitted/null values use the Agent config.
const budgeted = await router.predictBatch([
  { state: "ticket", questions, maxLen: 64, headMaxLen: 16 },
]);
// Per-call hooks are composed like predict: installed hooks first, then this list,
// then onPredictStart / onPredictEnd. null and [] add nothing. hooksRaise overrides
// the Router's policy for this call. The list runs once per request, including onRoute.
const traced = await router.predictBatch(requests, null, {
  hooks: [tracer],
  onPredictEnd(ctx) { console.log(ctx.usage); },
  hooksRaise: false,
});

// requests routed to the same checkpoint still split into separate batches when their
// question schemas differ (order-sensitively), when their maxLen / headMaxLen
// budgets differ after start hooks (which can replace or clear them), or — for an agent carrying
// lang_temperatures — when their languages differ: each request's effective language (an
// explicit lang, otherwise the detected non-English one) is forwarded to the agent, so
// the batched path scores exactly like predict. Router-level predict hooks run once per
// request: ctx.decision is set, ctx.skip() serves a cached result, and onPredictEnd runs
// per request even when the batch fails.

Python's sort_by_length grouping is not ported yet.

Shortlist (many labels)

import { shortlistChoice, predictShortlist, embedFnFromAgent, cached } from "laya-ts";

const cachedEmbed = cached(embedFn);
const keep = await shortlistChoice(state, bigCriteriaDict, cachedEmbed, 20);
const out = await predictShortlist(agent, state, questions, cachedEmbed, 20);
// out.shortlist[qid] = { labels, scores, k, n, passthrough }
// embedFnFromAgent(agent) mean-pools the loaded encoder; a dedicated bi-encoder usually shortlists better.
// cached(embedFn) keeps an LRU cache across calls; cachedEmbed.cacheInfo() reports size/hits/misses.

Tournament (without an embedder)

For a large tool or intent catalog, predictTournament lets the decision model narrow the options itself. It ports Python's predict_tournament to the local Node/browser runtime:

import { predictTournament } from "laya-ts";

const out = await predictTournament(agent, state, questions);
console.log(out.answers.intent);
console.log(out.tournament.intent); // { labels: [...finalists], n: originalCount, rounds: 1 }

// Pick a group size and forward options to every prediction, including the final one.
const routed = await predictTournament(router, state, questions, 16, { model: "english" });

Each oversized choice is split into balanced groups of at most 16 labels. All groups from all questions share one predict call per round; their winners advance until each choice fits. Up to 256 labels therefore take one elimination call and one final call. Small choices and non-choice questions reach the final call unchanged, and a request needing no elimination makes exactly one call. The helper does not mutate the supplied questions.

The returned tournament metadata covers every choice, including rounds: 0 for a passthrough. Probabilities and confidences describe only the finalists; usage, routing, and other prediction fields come from the final call, not all rounds combined. Hooks run for every underlying call. Calibrate thresholds on the strategy you actually use; a threshold fitted on the full catalog does not automatically transfer to a tournament. Widening the head budget or using a dedicated shortlist embedder can still be a better fit.

A narrowed choice drops its original option_order with one warning per question, matching Python. Passthrough definitions keep it. This does not add option_order handling to the TypeScript Agent, whose prediction path does not currently apply that field.

For a reproducible local comparison, build the package and export a checkpoint as above, then run from the repository root:

python laya-ts/scripts/tournament_parity.py --model-dir CHECKPOINT --onnx-dir ./model \
  --fixture ./tournament-python.json
node laya-ts/examples/tournament.mjs --model-dir ./model \
  --fixture ./tournament-python.json --report ./tournament-node.json

The Python script records fresh predictions for fixed inputs; Node independently runs the same inputs through the split ONNX export. The check compares each round's questions, winners, probabilities and final metadata, including a two-round case. Reports include artifact hashes, call counts and warmed timings. These few cases check runtime parity; they do not establish accuracy gains or production latency. Model files stay local.

Per-language calibration (lang_temperatures)

Port of the Python Agent(lang_temperatures=...) knob. A language override replaces the checkpoint's temperature for matching requests — keys normalise to the base subtag (de-AT → de), an omitted temperature inherits the base one, and temperature_by_options works per option-count bucket as usual:

const agent = await Agent.load("convaiinnovations/laya", {
  lang_temperatures: {
    de: { temperature: [1.2, 1.2, 1.2] },                 // fitted on German evals
    ja: { temperature_by_options: { "choice:11+": 1.4 } }, // buckets only, base temperature kept
  },
});
await agent.systemOne(state, questions, { lang: "de" });   // uses the German temperature
await router.predict(state, questions);                    // Router forwards the detected language

Router.predict forwards an explicit lang verbatim and otherwise the detected language (never "en" — matching Python, where detection only names non-English languages), so an override applies exactly to the requests it was fitted on.

Example (repo root)

node laya-ts/examples/try-ml.mjs   # needs ./model-ml from the export step
node laya-ts/examples/snake.mjs --ticks 50   # autonomous snake demo, headless smoke (live TUI without --ticks)

Web demo (browser, WebGPU → WASM)

No build step — serve the repo root over HTTP and open the page (localhost counts as a secure context for WebGPU):

python -m http.server 8000   # run at the repo root
# open http://localhost:8000/laya-ts/examples/web/

The page loads laya-ts/dist (run npm run build inside laya-ts/ first), pulls onnxruntime-web from a pinned CDN import map, and fetches the model from the URL in the box (default ../../../model-ml, i.e. the exported ./model-ml at the repo root). First load transfers ~1.3GB and is cached in CacheStorage afterwards; the encoder tries WebGPU and falls back to WASM automatically. Chrome/Edge for WebGPU, any modern browser for WASM.

Packaging

ponytail: CJS/browser-field dual build + tsconfig tests-include deferred — Task 7 verified ESM-only; CJS needs second tsc config + export-map change, untested. Add when a CJS consumer or browser-field swap is requested.