Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
test(ts): pin the two whitespace primitives the recorded email corpus cannot see
Aashish8e615710c66f
diff --git a/laya-ts/tests/email-parity.test.ts b/laya-ts/tests/email-parity.test.tsindex 79abf7b1..eae01d4b 100644--- a/laya-ts/tests/email-parity.test.ts+++ b/laya-ts/tests/email-parity.test.ts@@ -51,3 +51,51 @@ describe("email.json state cases", () => {);});});++// Two of the primitives in `src/email.ts` are invisible to the corpus above: no recorded body or+// subject holds U+FEFF, and no recorded character sits next to an `@` where the difference between+// the two whitespace classes decides a match. Python's `\s` and `str.strip()` are its 29 whitespace+// code points, which hold U+001C-U+001F and U+0085 and not U+FEFF, while JS's hold U+FEFF and not+// U+001C-U+001F. So each answer below is what `laya/email.py` printed for the same input+// (`clean_email_body`, and `email_state(...)["subject"]` for the two subjects), written out here+// instead of added to `email.json` because that corpus is shared with the Java port, whose+// `EmailTest` asserts its exact case counts.+describe("Python's whitespace set, where JS's differs", () => {+ const BOM_CONST = "\uFEFF";+ const FILE_SEP = "\u001C";+ const UNIT_SEP = "\u001F";++ // A paragraph's trailing whitespace is removed by `trimPy`, which reads Python's class, and+ // U+FEFF is not in it. `String.prototype.trim` does read U+FEFF as whitespace, and would delete a+ // character from the text handed to the model.+ it("keeps a U+FEFF that ends a paragraph", () => {+ const body = "I cannot log in" + BOM_CONST + "\n\nPlease reset my password.";+ expect(cleanEmailBody(body)).toBe(body);+ });++ it("keeps a U+FEFF that ends a subject", () => {+ expect(emailState("Refund" + BOM_CONST, "body").subject).toBe("Refund" + BOM_CONST);+ });++ // The other direction of the same difference: JS's `trim` leaves U+001C in the text.+ it("strips a U+001C that ends a line", () => {+ expect(cleanEmailBody("Trace the refund." + FILE_SEP + "\n\nPlease reset it.")).toBe(+ "Trace the refund.\n\nPlease reset it.",+ );+ });++ // U+001F is whitespace to Python, so the `\S` before the `@` in the attribution tail does not+ // match it and the line stays ordinary text; JS's `\S` would match, cut the body there, and drop+ // the sender's request. The next case is that same line with an ordinary letter, where the tail+ // does fire.+ it("does not read a U+001F before an address as an attribution tail", () => {+ const body = "Trace the refund.\n\nBruno " + UNIT_SEP + "@acme.com wrote:";+ expect(cleanEmailBody(body)).toBe(body);+ });++ it("reads an ordinary attribution tail and cuts it", () => {+ expect(cleanEmailBody("Trace the refund.\n\nBruno [email protected] wrote:")).toBe(+ "Trace the refund.",+ );+ });+});