Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

Merge PR #1446

Nandakishorb939072879b4
diff --cc laya-ts/src/lang.ts
index 14576726,f4f5c53e..d60e5f9a
--- a/laya-ts/src/lang.ts
+++ b/laya-ts/src/lang.ts
@@@ -213,19 -213,19 +213,32 @@@ function scriptOf(ch: string): string
return null;
}
+/** Python's `len()`: the number of code points, not of UTF-16 units. Python's `lang.py` divides
+ * the diacritic count by `len(lowered)`, and every other port counts code points too (Go
+ * `utf8.RuneCountInString`, Rust `chars().count()`, Java `codePointLength`, C++ `RuneCount`), so
+ * an astral character must count as one here as well. `String.length` counts it as two, which
+ * understated `diacriticRate` on any text carrying emoji -- e.g. `analyse()` on
+ * `"é\uD83D\uDE00\uD83D\uDE00" + "-".repeat(46)` read `isEnglish: true` in this port where
+ * Python answers `false`, routing the text to the English checkpoint. */
+function width(text: string): number {
+ let count = 0;
+ for (const _ch of text) count++;
+ return count;
+}
+
+ /** Python's `text[:count]`: a prefix cut at code points, so a surrogate pair is never split
+ * in half. Python's `lang.py` slices the same budgets by code points (`state_text`,
+ * `_non_english_segment`, the 4000-character leaf samples), and the email port already cuts this
+ * way (`cutCodePoints` in `email.ts`, added when a budget cut an emoji in half). `String.slice`
+ * counts UTF-16 units, so `stateText("A".repeat(3999) + "\uD83D\uDE00" + "...")` used to return a
+ * string ending in the orphaned high half of the emoji -- text Python never produces, and a lone
+ * surrogate the serve side already refuses as malformed (#454, #814). */
+ function cutCodePoints(text: string, count: number): string {
+ const chars = Array.from(text);
+ const end = count < 0 ? Math.max(0, chars.length + count) : Math.min(count, chars.length);
+ return chars.slice(0, end).join("");
+ }
+
/** String leaves of a state. Keys are ignored: they are usually English field names. */
function iterText(state: unknown, depth = 0): string[] {
if (depth > 6 || state == null) return [];