Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

fix(ts): match accented words in latinProfile like Python (WORD_RE was ASCII-only)

divit aggarwalf48080594131
diff --git a/laya-ts/src/lang.ts b/laya-ts/src/lang.ts
index 90f54b65..7d46ff58 100644
--- a/laya-ts/src/lang.ts
+++ b/laya-ts/src/lang.ts
@@ -123,7 +123,8 @@ const SHARED_WORDS: Set<string> = (() => {
return out;
})();
-const WORD_RE = /[^\W\d_]+/gu;
+// JS `\w` is ASCII-only, so Python's `[^\W\d_]` needs the Unicode classes spelled out (Nl/No are in Python's `\w`).
+const WORD_RE = /[\p{L}\p{Nl}\p{No}]+/gu;
// Lookbehind for the same reason as the Python side (see laya/lang.py): without it the
// greedy prefix is retried at every offset inside a run of word characters, which is
// quadratic in the run's length -- 50 000 characters of one token took 1540 ms here.
diff --git a/laya-ts/tests/lang.test.ts b/laya-ts/tests/lang.test.ts
index 120018fe..5a1f8653 100644
--- a/laya-ts/tests/lang.test.ts
+++ b/laya-ts/tests/lang.test.ts
@@ -69,4 +69,15 @@ describe("lang", () => {
expect(analyse("que hora es en australia").language).toBe("es");
expect(analyse("baisse le volume du haut-parleur").language).toBe("fr");
});
+ it.each<[string, string | null, boolean]>([
+ ["Preciso do contrato confidencial assinado até sexta.", "pt", false],
+ ["Gătește-mi o rețetă de sarmale de post pentru mâine.", "ro", false],
+ ["Müşteriden iki kez ücret alındı ve para iadesi istiyor", null, false],
+ ["Khách hàng đã bị thu phí hai lần và muốn được hoàn tiền ngay", null, false],
+ ])("counts accented words whole, like Python: %s", (text, language, english) => {
+ const a = analyse(text);
+ expect(a.language).toBe(language);
+ expect(a.isEnglish).toBe(english);
+ expect(a.languageUndecided).toBe(language === null);
+ });
});