Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

test(ts): pin the two whitespace primitives the recorded email corpus cannot see

Aashish8e615710c66f
diff --git a/laya-ts/tests/email-parity.test.ts b/laya-ts/tests/email-parity.test.ts
index 79abf7b1..eae01d4b 100644
--- a/laya-ts/tests/email-parity.test.ts
+++ b/laya-ts/tests/email-parity.test.ts
@@ -51,3 +51,51 @@ describe("email.json state cases", () => {
);
});
});
+
+// Two of the primitives in `src/email.ts` are invisible to the corpus above: no recorded body or
+// subject holds U+FEFF, and no recorded character sits next to an `@` where the difference between
+// the two whitespace classes decides a match. Python's `\s` and `str.strip()` are its 29 whitespace
+// code points, which hold U+001C-U+001F and U+0085 and not U+FEFF, while JS's hold U+FEFF and not
+// U+001C-U+001F. So each answer below is what `laya/email.py` printed for the same input
+// (`clean_email_body`, and `email_state(...)["subject"]` for the two subjects), written out here
+// instead of added to `email.json` because that corpus is shared with the Java port, whose
+// `EmailTest` asserts its exact case counts.
+describe("Python's whitespace set, where JS's differs", () => {
+ const BOM_CONST = "\uFEFF";
+ const FILE_SEP = "\u001C";
+ const UNIT_SEP = "\u001F";
+
+ // A paragraph's trailing whitespace is removed by `trimPy`, which reads Python's class, and
+ // U+FEFF is not in it. `String.prototype.trim` does read U+FEFF as whitespace, and would delete a
+ // character from the text handed to the model.
+ it("keeps a U+FEFF that ends a paragraph", () => {
+ const body = "I cannot log in" + BOM_CONST + "\n\nPlease reset my password.";
+ expect(cleanEmailBody(body)).toBe(body);
+ });
+
+ it("keeps a U+FEFF that ends a subject", () => {
+ expect(emailState("Refund" + BOM_CONST, "body").subject).toBe("Refund" + BOM_CONST);
+ });
+
+ // The other direction of the same difference: JS's `trim` leaves U+001C in the text.
+ it("strips a U+001C that ends a line", () => {
+ expect(cleanEmailBody("Trace the refund." + FILE_SEP + "\n\nPlease reset it.")).toBe(
+ "Trace the refund.\n\nPlease reset it.",
+ );
+ });
+
+ // U+001F is whitespace to Python, so the `\S` before the `@` in the attribution tail does not
+ // match it and the line stays ordinary text; JS's `\S` would match, cut the body there, and drop
+ // the sender's request. The next case is that same line with an ordinary letter, where the tail
+ // does fire.
+ it("does not read a U+001F before an address as an attribution tail", () => {
+ const body = "Trace the refund.\n\nBruno " + UNIT_SEP + "@acme.com wrote:";
+ expect(cleanEmailBody(body)).toBe(body);
+ });
+
+ it("reads an ordinary attribution tail and cuts it", () => {
+ expect(cleanEmailBody("Trace the refund.\n\nBruno [email protected] wrote:")).toBe(
+ "Trace the refund.",
+ );
+ });
+});