Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(java): port laya.email, and the measurement that made it usable
Adeshina Lasisi6f882e398947
diff --git a/laya-java/README.md b/laya-java/README.mdindex acd2c837..a51faabc 100644--- a/laya-java/README.md+++ b/laya-java/README.md@@ -13,9 +13,10 @@ four questions about a document cost one batched encode rather than four round tImplemented: tokenizer, sequence builder, config, ONNX inference (fused or split graph), answerdecoding, `predict`, `predictBatch`, usage and truncation reporting, script and language detection(`lang.LanguageDetection`), the question presets (`Presets`), the checkpoint `Router` with its-load-and-evict lifecycle, and the embedding `Shortlist` with its LRU cache.+load-and-evict lifecycle, the embedding `Shortlist` with its LRU cache, and the email cleaner+and state builder (`LayaEmail`).-Not implemented yet: hooks, `predictLong`, structured `decide`, `laya.email`, the+Not implemented yet: hooks, `predictLong`, structured `decide`, the`laya-java-client` HTTP module, Android. **Not published to Maven Central** — see[Installing](#installing).@@ -233,6 +234,50 @@ passthrough: the labels come back in order and **the embedder is never called**.Both `Agent` and `Router` implement `Predictor`, so shortlisting works identically against a fixedcheckpoint or a routed one.+## Cleaning an email++`laya.email`'s cleaner and state builder. The markers cover English, Portuguese, Spanish and+French mail clients, because the router already sends the last three to the multilingual+checkpoint and an English-only cleaner left their quoted history — often a *different* request —+weighing on the answer as much as the new message.++```java+import com.convaiinnovations.laya.LayaEmail;+import java.util.Map;++String body = """+ I was charged twice for order 8812. Please refund one of them.++ Atenciosamente,+ Ana Souza+ Enviado do meu iPhone++ Esta mensagem e confidencial e de uso exclusivo do destinatario.++ Em ter., 3 de set. de 2025, Suporte <[email protected]> escreveu:+ > Podemos ajudar?+ """;++// quoted history, sign-off, device footer and disclaimer all go+String clean = LayaEmail.cleanEmailBody(body);++Map<String, Object> state = LayaEmail.emailState("Cobranca duplicada", body, "[email protected]");+// {subject=Cobranca duplicada, body=I was charged twice..., [email protected]}++Prediction p = agent.predict(state, LayaEmail.emailQuestions());+```++`emailState` takes the same `maxChars` budget and passes it through, and it is worth raising for+a long message: at the default the body stops after 3,000 characters, so a request arriving in+the last paragraphs never reaches the model.++What it deliberately does **not** cut is the interesting half. `From: my side the integration+works, but please refund...` is prose, not a header, so a reply header is recognised only when an+address follows it or its own `Sent:`/`Enviado:` line does. `Thanks for the quick reply.` is not+a sign-off, `Obrigado pelo retorno, mas ...` is a request, and `Is this confidential?` is a+question — a cleaner that is too eager deletes what the sender actually wrote, which is worse+than leaving one boilerplate line behind.+## Lower-level pieces```javadiff --git a/laya-java/fixtures/email.json b/laya-java/fixtures/email.jsonnew file mode 100644index 00000000..34d25b50--- /dev/null+++ b/laya-java/fixtures/email.json@@ -0,0 +1,543 @@+{+ "cleaned": [+ {+ "name": "empty",+ "body": "",+ "result": ""+ },+ {+ "name": "plain",+ "body": "I cannot log in and need a password reset.",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "crlf",+ "body": "line one\r\nline two\r\rline three",+ "result": "line one\nline two\n\nline three"+ },+ {+ "name": "literal-backslash-n",+ "body": "first\\nsecond",+ "result": "first\nsecond"+ },+ {+ "name": "collapses-runs",+ "body": "a b\t\tc",+ "result": "a b c"+ },+ {+ "name": "blank-paragraphs",+ "body": "first\n\n\n\nsecond\n \n\nthird",+ "result": "first\n\nsecond\n\nthird"+ },+ {+ "name": "on-wrote",+ "body": "I cannot log in and need a password reset.\n\nOn Tue, 3 Sep 2025 at 10:04, Ana <[email protected]> wrote:\n> older text",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "em-escreveu",+ "body": "I cannot log in and need a password reset.\n\nEm ter., 3 de set. de 2025, Ana escreveu:\n> antigo",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "el-escribio",+ "body": "I cannot log in and need a password reset.\n\nEl mar, 3 sept 2025 a las 10:04, Ana escribió:\n> viejo",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "le-a-ecrit",+ "body": "I cannot log in and need a password reset.\n\nLe mar. 3 sept. 2025 a écrit :\n> ancien",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "original-message",+ "body": "I cannot log in and need a password reset.\n\n-----Original Message-----\nolder",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "mensagem-original",+ "body": "I cannot log in and need a password reset.\n\n-----Mensagem original-----\nantigo",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "message-d-origine",+ "body": "I cannot log in and need a password reset.\n\n-----Message d'origine-----\nancien",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "underscore-rule",+ "body": "I cannot log in and need a password reset.\n\n________________\nolder",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "from-with-address",+ "body": "I cannot log in and need a password reset.\n\nFrom: Ana <[email protected]>\nolder",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "de-with-address",+ "body": "I cannot log in and need a password reset.\n\nDe: Ana <[email protected]>\nantigo",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "de-spaced-colon",+ "body": "I cannot log in and need a password reset.\n\nDe : Marie <[email protected]>\nancien",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "em-without-date",+ "body": "I cannot log in and need a password reset.\n\nEm resposta ao que você escreveu:\nmais detalhes",+ "result": "I cannot log in and need a password reset.\n\nEm resposta ao que você escreveu:\nmais detalhes"+ },+ {+ "name": "le-without-date",+ "body": "I cannot log in and need a password reset.\n\nLe rapport que vous avez écrit :\nplus de détails",+ "result": "I cannot log in and need a password reset.\n\nLe rapport que vous avez écrit :\nplus de détails"+ },+ {+ "name": "from-as-prose",+ "body": "From: my side the integration works, but please refund the charge.",+ "result": "From: my side the integration works, but please refund the charge."+ },+ {+ "name": "de-as-date-range",+ "body": "De: 10/09 a 15/09 estarei fora, por favor adie a cobranca.",+ "result": "De: 10/09 a 15/09 estarei fora, por favor adie a cobranca."+ },+ {+ "name": "quote-header-first-line",+ "body": "On Tue, 3 Sep 2025, Ana <[email protected]> wrote:\nstill the body",+ "result": "On Tue, 3 Sep 2025, Ana <[email protected]> wrote:\nstill the body"+ },+ {+ "name": "from-name-then-sent",+ "body": "I cannot log in and need a password reset.\n\nFrom: Maria Souza\nSent: Tuesday\nolder",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "de-name-then-enviado",+ "body": "I cannot log in and need a password reset.\n\nDe: Maria Souza\nEnviado: terca\nantigo",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "de-name-then-envoye",+ "body": "I cannot log in and need a password reset.\n\nDe : Marie Dupont\nEnvoyé : mardi\nancien",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "from-name-then-date-year",+ "body": "I cannot log in and need a password reset.\n\nFrom: Maria Souza\nDate: 3 Sep 2025\nolder",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "de-name-no-neighbour",+ "body": "I cannot log in and need a password reset.\n\nDe: Maria Souza\nPara: 15/09\nstill body",+ "result": "I cannot log in and need a password reset.\n\nDe: Maria Souza\nPara: 15/09\nstill body"+ },+ {+ "name": "wrapped-attribution",+ "body": "I cannot log in and need a password reset.\n\nEm ter., 3 de set. de 2025,\[email protected]> escreveu:\n> antigo",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "wrapped-attribution-fr",+ "body": "I cannot log in and need a password reset.\n\nLe mar. 3 sept. 2025,\[email protected]> a écrit :\n> ancien",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "attribution-tail-no-head",+ "body": "I cannot log in and need a password reset.\n\[email protected]> escreveu:\n> antigo",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "angle-quoted",+ "body": "I cannot log in and need a password reset.\n> quoted\n > indented quote\nmore of my message",+ "result": "I cannot log in and need a password reset.\nmore of my message"+ },+ {+ "name": "signoff-regards-name",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, Ana\[email protected]",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "signoff-latin-ext-name",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, Łukasz\[email protected]",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "signoff-caseless-name",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, 山田\[email protected]",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "not-signoff-pt-with-name",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nObrigado, 山田\[email protected]",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nObrigado, 山田\[email protected]"+ },+ {+ "name": "signoff-titlecase-name",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, Džarko\[email protected]",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "signoff-combining-name",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, José\[email protected]",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "signoff-dashes",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\n--\nAna",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "signoff-warmest",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nWarmest regards, Ana",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "signoff-and-regards",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nThanks and regards, Ana",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "signoff-many-thanks",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nMany thanks, Ana",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "not-signoff-sentence",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nThanks for the quick reply.",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nThanks for the quick reply."+ },+ {+ "name": "not-signoff-lowercase-other-script",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nThanks, żaneta",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nThanks, żaneta"+ },+ {+ "name": "not-signoff-symbol-name",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nThanks, Ⓐ",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nThanks, Ⓐ"+ },+ {+ "name": "not-signoff-zwnj",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nThanks, क्ष",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nThanks, क्ष"+ },+ {+ "name": "signoff-atenciosamente",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nAtenciosamente,\nAna",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "signoff-cordialement",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nCordialement,\nMarie",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "signoff-saudacoes",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nSaudações!",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "not-signoff-obrigado-mas",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nObrigado pelo retorno, mas preciso do estorno.",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nObrigado pelo retorno, mas preciso do estorno."+ },+ {+ "name": "not-signoff-merci-mais",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nMerci pour votre aide, mais le probleme persiste.",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nMerci pour votre aide, mais le probleme persiste."+ },+ {+ "name": "signoff-too-early",+ "body": "Regards, Ana\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.",+ "result": "Regards, Ana\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "footer-iphone",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nSent from my iPhone",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "footer-samsung",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nEnviado do meu smartphone Samsung Galaxy.",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "footer-envoye-iphone",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nEnvoyé depuis mon iPhone",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "footer-get-outlook",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nGet Outlook for iOS",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "not-footer-with-request",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nEnviado do meu celular o comprovante ontem.",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nEnviado do meu celular o comprovante ontem."+ },+ {+ "name": "disclaimer-en",+ "body": "I cannot log in and need a password reset.\n\nThis email and its contents are confidential and intended solely for the addressee.",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "disclaimer-en-wrapped",+ "body": "I cannot log in and need a password reset.\n\nThis email and its contents are\nconfidential and intended solely for the addressee.",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "disclaimer-received-in-error",+ "body": "I cannot log in and need a password reset.\n\nIf you have received this email in error please delete it.",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "disclaimer-pt",+ "body": "I cannot log in and need a password reset.\n\nEsta mensagem e seus anexos sao confidenciais e de uso exclusivo do destinatario.",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "disclaimer-es",+ "body": "I cannot log in and need a password reset.\n\nEste mensaje es confidencial y para uso exclusivo del destinatario.",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "disclaimer-fr",+ "body": "I cannot log in and need a password reset.\n\nCe message est confidentiel et destiné uniquement au destinataire.",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "disclaimer-print",+ "body": "I cannot log in and need a password reset.\n\nAntes de imprimir pense no meio ambiente.",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "disclaimer-mixed-paragraph",+ "body": "Please refund the charge. This email and its contents are confidential and intended solely for the addressee.",+ "result": "Please refund the charge."+ },+ {+ "name": "disclaimer-fused-line",+ "body": "The account is locked\nThis email and its contents are confidential and intended solely for the addressee.",+ "result": "The account is locked"+ },+ {+ "name": "not-disclaimer-question",+ "body": "Is this confidential?",+ "result": "Is this confidential?"+ },+ {+ "name": "not-disclaimer-colon",+ "body": "Confidential: I need a refund.",+ "result": "Confidential: I need a refund."+ },+ {+ "name": "not-disclaimer-print",+ "body": "Antes de imprimir o boleto, confira o valor.",+ "result": "Antes de imprimir o boleto, confira o valor."+ },+ {+ "name": "not-disclaimer-contract",+ "body": "Preciso do contrato confidencial assinado.",+ "result": "Preciso do contrato confidencial assinado."+ },+ {+ "name": "not-disclaimer-fr-contract",+ "body": "Je voudrais le contrat confidentiel signé.",+ "result": "Je voudrais le contrat confidentiel signé."+ },+ {+ "name": "signoff-class-zero-mark",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, Jo͏se",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ },+ {+ "name": "not-signoff-baseless-mark",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, ́Ana",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, ́Ana"+ },+ {+ "name": "not-signoff-too-wide",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, Anastasia Konstantinopolitanopoulos",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, Anastasia Konstantinopolitanopoulos"+ },+ {+ "name": "not-signoff-four-tokens",+ "body": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, Ana Maria Souza Lima",+ "result": "I cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nRegards, Ana Maria Souza Lima"+ },+ {+ "name": "attribution-tail-keeps-prose",+ "body": "I cannot log in and need a password reset.\nSee the details below.\[email protected]> escreveu:\n> antigo",+ "result": "I cannot log in and need a password reset.\nSee the details below."+ },+ {+ "name": "quote-header-across-u0085",+ "body": "I cannot log in and need a password reset.\n\nOn Tue, 3 Sep 2025 Ana wrote:\n> older",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "quote-header-nbsp-indent",+ "body": "I cannot log in and need a password reset.\n\n On Tue, 3 Sep 2025, Ana wrote:\n> older",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "quote-header-u001c-indent",+ "body": "I cannot log in and need a password reset.\n\n\u001cOn Tue, 3 Sep 2025, Ana wrote:\n> older",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "quote-header-arabic-indic-date",+ "body": "I cannot log in and need a password reset.\n\nEm ter., ٣ de set. de ٢٠٢٥, Ana escreveu:\n> antigo",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "header-next-devanagari-year",+ "body": "I cannot log in and need a password reset.\n\nFrom: Maria Souza\nDate: 3 Sep २०२५\nolder",+ "result": "I cannot log in and need a password reset."+ },+ {+ "name": "full-stack",+ "body": "I cannot log in and need a password reset.\nPlease check the attached receipt.\n\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nAtenciosamente,\nAna Souza\nEnviado do meu iPhone\n\nEsta mensagem e confidencial e de uso exclusivo do destinatario.\n\nEm ter., 3 de set. de 2025, Bruno <[email protected]> escreveu:\n> mensagem antiga",+ "result": "I cannot log in and need a password reset.\nPlease check the attached receipt.\n\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset.\nI cannot log in and need a password reset."+ }+ ],+ "budgets": [+ {+ "name": "budget-default",+ "body": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx I cannot log in and need a password reset.",+ "max_chars": 3000,+ "result": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx I cannot log in and need a password reset."+ },+ {+ "name": "budget-tiny",+ "body": "I cannot log in and need a password reset.",+ "max_chars": 12,+ "result": "I cannot log"+ },+ {+ "name": "budget-cut-mid-word",+ "body": "abcdefghij abcdefghij abcdefghij abcdefghij abcdefghij abcdefghij abcdefghij abcdefghij abcdefghij abcdefghij ",+ "max_chars": 25,+ "result": "abcdefghij abcdefghij abc"+ },+ {+ "name": "budget-four-x-bound",+ "body": "word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word This email is confidential and intended solely for the addressee.",+ "max_chars": 100,+ "result": "word word word word word word word word word word word word word word word word word word word word "+ },+ {+ "name": "budget-zero",+ "body": "I cannot log in and need a password reset.",+ "max_chars": 0,+ "result": ""+ }+ ],+ "states": [+ {+ "name": "state-plain",+ "subject": "Refund request",+ "body": "I cannot log in and need a password reset.",+ "sender": null,+ "clean": true,+ "max_chars": 3000,+ "extra": {},+ "result": {+ "subject": "Refund request",+ "body": "I cannot log in and need a password reset."+ }+ },+ {+ "name": "state-with-sender",+ "subject": "Refund request",+ "body": "I cannot log in and need a password reset.",+ "sender": "[email protected]",+ "clean": true,+ "max_chars": 3000,+ "extra": {},+ "result": {+ "subject": "Refund request",+ "body": "I cannot log in and need a password reset.",+ "from": "[email protected]"+ }+ },+ {+ "name": "state-unclean",+ "subject": "Refund request",+ "body": "I cannot log in and need a password reset.\n\nSent from my iPhone",+ "sender": null,+ "clean": false,+ "max_chars": 3000,+ "extra": {},+ "result": {+ "subject": "Refund request",+ "body": "I cannot log in and need a password reset.\n\nSent from my iPhone"+ }+ },+ {+ "name": "state-extra-fields",+ "subject": "Refund",+ "body": "I cannot log in and need a password reset.",+ "sender": "[email protected]",+ "clean": true,+ "max_chars": 3000,+ "extra": {+ "priority": "high",+ "ticket": 42,+ "dropped": null+ },+ "result": {+ "subject": "Refund",+ "body": "I cannot log in and need a password reset.",+ "from": "[email protected]",+ "priority": "high",+ "ticket": 42+ }+ },+ {+ "name": "state-blank-subject",+ "subject": " ",+ "body": "I cannot log in and need a password reset.",+ "sender": null,+ "clean": true,+ "max_chars": 3000,+ "extra": {},+ "result": {+ "subject": "",+ "body": "I cannot log in and need a password reset."+ }+ },+ {+ "name": "state-none-ish",+ "subject": "",+ "body": "",+ "sender": null,+ "clean": true,+ "max_chars": 3000,+ "extra": {},+ "result": {+ "subject": "",+ "body": ""+ }+ },+ {+ "name": "state-budget",+ "subject": "S",+ "body": "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy",+ "sender": null,+ "clean": true,+ "max_chars": 20,+ "extra": {},+ "result": {+ "subject": "S",+ "body": "yyyyyyyyyyyyyyyyyyyy"+ }+ }+ ],+ "questions_match_presets": true+}diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/LayaEmail.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/LayaEmail.javanew file mode 100644index 00000000..686b782a--- /dev/null+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/LayaEmail.java@@ -0,0 +1,785 @@+package com.convaiinnovations.laya;++import com.convaiinnovations.laya.lang.UnicodeTables;+import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import java.util.regex.Matcher;+import java.util.regex.Pattern;++/**+ * {@code laya.email}: strip quoted history, signatures and disclaimers from a mail body, and build+ * a state from a message.+ *+ * <p>The markers cover English, Portuguese, Spanish and French mail clients, which is the set the+ * reference covers and for the reason it gives: the router already sends Portuguese, Spanish and+ * French states to the multilingual checkpoint, so an English-only cleaner left Gmail's+ * {@code Em ... escreveu:}, Outlook's {@code -----Mensagem original-----}, the+ * {@code Atenciosamente} sign-off and the confidentiality footer in the input, and the quoted+ * history -- often a different request -- weighed on the answer as much as the new message.+ *+ * <h2>Why this is scanners and explicit classes rather than the reference's patterns</h2>+ *+ * <p>Every pattern here was translated, not transcribed, because four of Python's regex constructs+ * mean something else in {@code java.util.regex}:+ *+ * <ul>+ * <li>{@code \s}. Python's is 29 code points -- exactly {@code str.isspace()}, verified over all+ * of Unicode. Java's is 6 by default, and 25 under {@code UNICODE_CHARACTER_CLASS}, which is+ * the White_Space property and so excludes U+001C to U+001F. Those four are Cc, not+ * White_Space, and Python treats them as space. The class is built below from the same+ * table the digest gates, so it cannot drift from the reference.+ * <li>{@code \d}. Python's is the Nd category, 680 code points at Unicode 15. Java's is ASCII+ * by default and, under {@code UNICODE_CHARACTER_CLASS}, follows THE JDK'S Unicode version+ * -- which is the defect that shipped in this port's pre-tokenizer, where the same jar gave+ * different token ids on different JDKs. Built from the table for the same reason.+ * <li>{@code \b}. A word boundary is defined in terms of {@code \w}, and Java's {@code \w}+ * admits combining marks and join controls where Python's does not, so the two disagree+ * about whether a boundary exists after {@code confidential} followed by U+0301. Each+ * {@code \b} is therefore a lookaround over the Python word class.+ * <li>{@code .} and {@code $}. Python's {@code .} excludes only {@code \n}; Java's excludes+ * every line terminator, including U+0085 and U+2028, and its {@code $} looks for any of+ * them too. {@code UNIX_LINES} makes both mean what Python means, so every pattern carries+ * it.+ * </ul>+ *+ * <p>Case folding is left to {@code CASE_INSENSITIVE | UNICODE_CASE}. The accented literals in the+ * markers are Latin-1 Supplement and Latin Extended-A, whose case mappings have not moved in any+ * Unicode version, and the fixture is run on two JDKs with different Unicode versions to keep that+ * an observation rather than an assumption.+ */+public final class LayaEmail {++ private LayaEmail() {+ }++ // ------------------------------------------------------- Python's character classes++ /**+ * {@code \s}, {@code \S} and {@code \d} from a single pass over every code point.+ *+ * <p>Built from {@link UnicodeTables}, so these classes and the rest of the port read the+ * same Unicode -- the one recorded from CPython -- and a table change moves both together.+ * Writing the ranges out by hand would be a second copy that can disagree with the first,+ * which is how this port's pre-tokenizer came to classify 9,917 code points by the JDK's+ * Unicode instead of the reference's.+ *+ * <p>One pass rather than one per class, because this walk is the whole cost of the first+ * call into {@link Patterns}. The two space classes come from the same predicate, so the+ * negated one costs nothing extra: a range of spaces is a gap in the complement and the other+ * way round.+ */+ private static String[] classes() {+ StringBuilder space = new StringBuilder("[");+ StringBuilder digit = new StringBuilder("[");+ int cp = 0;+ while (cp <= 0x10FFFF) {+ boolean isSpace = UnicodeTables.isSpace(cp);+ boolean isDigit = UnicodeTables.isDigit(cp);+ if (!isSpace && !isDigit) {+ cp++;+ continue;+ }+ int start = cp;+ if (isSpace) {+ while (cp + 1 <= 0x10FFFF && UnicodeTables.isSpace(cp + 1)) {+ cp++;+ }+ appendRange(space, start, cp);+ } else {+ while (cp + 1 <= 0x10FFFF && UnicodeTables.isDigit(cp + 1)) {+ cp++;+ }+ appendRange(digit, start, cp);+ }+ cp++;+ }+ String spaces = space.append(']').toString();+ return new String[] {spaces, "[^" + spaces.substring(1), digit.append(']').toString()};+ }++ /** One range of a character class, or one code point when the range holds one. */+ private static void appendRange(StringBuilder out, int from, int to) {+ appendEscaped(out, from);+ if (to > from) {+ out.append('-');+ appendEscaped(out, to);+ }+ }++ /** A code point as a regex escape, so no member of a class is ever a metacharacter. */+ private static void appendEscaped(StringBuilder out, int codePoint) {+ if (codePoint <= 0xFFFF) {+ out.append(String.format("\\u%04X", codePoint));+ } else {+ out.append(String.format("\\x{%X}", codePoint));+ }+ }++ /**+ * {@code re.I} plus the two flags that make Java's syntax mean Python's.+ *+ * <p>{@code UNIX_LINES} is not optional: without it Java's {@code .} excludes every line+ * terminator -- U+0085 and U+2028 among them -- where Python's excludes only {@code \n}, and+ * Java's {@code $} looks for any of them too.+ */+ private static Pattern compile(String regex) {+ return Pattern.compile(regex, Pattern.CASE_INSENSITIVE | Pattern.UNICODE_CASE+ | Pattern.UNIX_LINES);+ }++ /** For the one pattern the reference does not pass {@code re.I}, scoping it with inline flags. */+ private static Pattern compileCased(String regex) {+ return Pattern.compile(regex, Pattern.UNIX_LINES);+ }++ /** The default body budget, which {@link #emailState} passes through. */+ public static final int DEFAULT_MAX_CHARS = 3000;++ /**+ * The patterns, built on first use.+ *+ * <p>A holder rather than fields of {@code LayaEmail}, because deriving the character classes+ * from {@link UnicodeTables} walks all 1,114,112 code points and costs about 80 ms. That is a+ * fair price for classes that cannot disagree with CPython, but only for a caller who cleans+ * an email: paid in {@code LayaEmail}'s own initialiser it would land on every caller of the+ * package, including one that only ever opens an {@link Agent}.+ */+ private static final class Patterns {++ /**+ * {@code \s}, {@code \S} and {@code \d}, built in ONE walk of Unicode.+ *+ * <p>Three calls to {@code classOf} is three walks of 1,114,112 code points, and this is+ * the whole cost of the first call into this class. Measured on the first+ * {@code cleanEmailBody}: 160 ms when the unused word class was still built here, about+ * 100 ms with three walks, about 35 ms with one.+ */+ private static final String[] CLASSES = classes();++ /** Python's {@code \s}: {@code str.isspace()}, 29 code points. */+ static final String S = CLASSES[0];+ /** Python's {@code \S}. */+ static final String NS = CLASSES[1];+ /** Python's {@code \d}: the Nd category. */+ static final String D = CLASSES[2];+ /**+ * {@code \b}, Java's own -- applied to a string {@link #boundaryView} has brought into+ * agreement with Python's word class, so the boundary it finds is Python's boundary.+ *+ * <p>The exact alternative, a lookaround over Python's own word class, was written+ * first and measured:+ * it inlines a 10,622-character class at 26 boundary sites, which is a 277,309-character+ * pattern and 1,742 us per paragraph against 7.8 us here -- 220 times the cost, and the+ * whole cost of cleaning a message. The two agreed on all 20,000 paragraphs of the sweep+ * corpus, which is exactly why the lookaround is not kept as the "safe" option:+ * agreement on a corpus is not equivalence, and Java's {@code \w} follows THE JDK'S+ * Unicode version. Deciding the boundary from the recorded table instead is what makes+ * the answer the same on every JDK, which is the property this port lost once already.+ */+ static final String BB = "\\b";+ static final String BA = "\\b";++ /** Java's {@code \w}, probed rather than assumed, because it is the JDK's and not ours. */+ static final Pattern JAVA_WORD =+ Pattern.compile("\\w", Pattern.UNICODE_CHARACTER_CLASS);++ static final List<Pattern> QUOTE_HEADERS = List.of(+ compile("^" + S + "*On .{0,300}wrote:" + S + "*$"),+ // "Em resposta ao que você escreveu:" is body text; a client's attribution always+ // carries a date.+ compile("^" + S + "*Em (?=.*" + D + ").{0,300}escreveu:" + S + "*$"),+ compile("^" + S + "*El (?=.*" + D + ").{0,300}escribi[óo]:" + S + "*$"),+ // Gmail's French attribution carries a date too; a bare "Le rapport que vous avez+ // écrit :" is body text. French typography puts a space before the colon.+ compile("^" + S + "*Le (?=.*" + D + ").{0,300}a [eé]crit" + S + "*:" + S + "*$"),+ compile("^" + S + "*-{2,}" + S + "*(Original|Forwarded) Message" + S + "*-{2,}"),+ compile("^" + S + "*-{2,}" + S + "*(Mensagem (original|encaminhada)"+ + "|Mensaje (original|reenviado)|Message d'origine)" + S + "*-{2,}"),+ compile("^" + S + "*_{8,}" + S + "*$"),+ // `From:` opens ordinary prose too ("From: my side the integration works, but+ // please refund..."), and a reply header always carries the sender, so the header+ // is only recognised when an address follows. A bare `From: Name` is caught by+ // HEADER_FROM_NAME/HEADER_NEXT instead, which need the header's own `Sent:`.+ compile("^" + S + "*From:" + S + ".*[@<]"),+ // `De:` also opens ordinary Portuguese and Spanish lines ("De: 10/09 a 15/09").+ // French Outlook writes `De :` with a space.+ compile("^" + S + "*De" + S + "*:" + S + ".*[@<]"));++ // Gmail wraps a long attribution line, leaving `[email protected]> escreveu:` alone on the+ // next line. That tail cuts too, and takes the `On/Em/El/Le ...` head it belongs to with+ // it. The French tail keeps its spaced colon.+ static final Pattern ATTRIBUTION_TAIL = compile(+ "^.{0,120}" + NS + "@" + NS + "+" + S + "+(wrote|escreveu|escribi[óo]|a [eé]crit)"+ + S + "*:" + S + "*$");+ static final Pattern ATTRIBUTION_HEAD = compile("^" + S + "*(On|Em|El|Le) (?=.*" + D + ")");++ // Exchange often leaves the address out of Outlook's reply header ("De: Maria Souza"), so+ // a bare `De:` only cuts when the header's own `Enviado:` line, or a dated `Data:`/+ // `Fecha:` line, follows it. `Para:` is not enough: "De: 10/09 / Para: 15/09" is how a+ // leave request reads. The same is true of a bare English `From: Maria Souza`.+ static final Pattern HEADER_FROM_NAME = compile("^" + S + "*(De|From)" + S + "*:" + S + "+" + NS);+ static final Pattern HEADER_NEXT = compile(+ "^" + S + "*(Enviad[oa]( em| el)?:" + S + "|Envoy[ée]( le)?" + S + "*:" + S+ + "|Sent:" + S + "|(Data|Fecha|Date):" + S + ".*" + D + "{4})");++ // A closing is the closing word plus punctuation and at most a name. The closing words are+ // matched case-insensitively and the name is not, so the flag is scoped in the reference+ // with `(?i:...)`; `(?iu:...)` is the same scoping with Unicode case folding, which is+ // what Python's `re.I` on a str pattern does.+ static final Pattern SIGNOFF_HEAD = compileCased(+ "^" + S + "*(?iu:best|kind|warmest|warm|many thanks|thanks|thank you|regards"+ + "|cheers|sincerely)"+ + "(?iu:" + S + "+(?:and|&)" + S + "+regards|" + S+ + "+(?:regards|wishes|again|in advance|a lot|so much|very much))?");++ static final Pattern SIGNOFF_DASHES = compile("^" + S + "*--" + S + "*$");++ // Portuguese, Spanish and French sign-offs match only on their own: "Obrigado pelo+ // retorno, mas ..." is a request, not a signature, so unlike the English marker no+ // trailing words are allowed.+ static final Pattern SIGNOFF_ROMANCE = compile(+ "^" + S + "*(atenciosamente|att|abraços?|abs|um abraço|cordialmente|grat[oa]"+ + "|(muito )?obrigad[oa]s?( desde já| pela atenção)?"+ + "|(com os melhores )?cumprimentos|saudações|"+ + "(un )?saludos?( cordiales)?|atentamente|(muchas )?gracias( de antemano)?|"+ + "(bien )?cordialement|salutations( distinguées)?|bien à vous|merci( d'avance)?|"+ + "bonne journée)[" + inner(S) + ",!.]*$");++ // Mobile and mail-app footers. Only a line that is nothing BUT the footer matches --+ // "Enviado do meu celular o comprovante ontem." is a request -- and such a line may run+ // to 60 characters, since Samsung's default is longer than a sign-off's 40.+ private static final String DEVICE =+ "iphone|ipad|android|ios|mobile|celular|telemóvel|móvil|galaxy|smartphone|samsung"+ + "|tablet|outlook|yahoo|mail|e-?mail|gmail|windows";+ static final Pattern DEVICE_FOOTER = compile(+ "^" + S + "*((enviad[oa] (do|pelo|pela|via|desde|a partir do)( meu| minha| mi)?"+ + "|sent from( my)?|envoy[ée] (depuis|de)( mon| ma| mes)?)"+ + " (" + DEVICE + ")( (" + DEVICE + "|para|for|no|na|" + D + "+|phone|device|pro"+ + "|max|mini|plus|using [a-z][a-z0-9_.+-]*))*"+ + "|(obter o|get) outlook (para|for) (ios|android))[" + inner(S) + ".!]*$");++ static final Pattern DISCLAIMER = compile(+ // English: tied to a disclaimer noun AND a disclaimer tail, the way the+ // Portuguese branches are. The bare word matched any sentence that merely+ // mentioned it, so "Is this confidential?" was deleted whole. `[^.]` rather than+ // `[^.\n]`: a footer wraps, so "are\nconfidential" must still match.+ "(" + BB + "(e-?mail|message|information|communication|transmission|contents?)"+ + BA + "[^.]{0,60}" + BB + "confidential" + BA + "[^.]{0,60}" + BB+ + "(intended|solely|addressee|recipient|privileged|disclos|unauthori[sz]ed)|"+ + BB + "confidential" + BA + "[^.]{0,60}" + BB + "(and (may|is) (also )?privileged)|"+ + "if you (have )?received this (e-?mail|message) in error|"+ // Portuguese and Spanish: tied to "this message/e-mail" rather than the bare word+ // `confidencial`, which a sender's own request uses just as often.+ + BB + "(esta|este) (mensagem|e-?mail|mensaje|correo)" + BA + "[^.]{0,80}"+ + "(confidencia|sigilos|privilegiad)|"+ + BB + "(uso exclusivo|exclusivamente|únicamente|unicamente)" + BA + "[^.]{0,30}"+ + "(destinatári|destinatari|pessoa|persona|entidade|entidad)|"+ + BB + "(recebeu|recebido|receber) (esta|este) (mensagem|e-?mail)" + BA+ + "[^.]{0,20} por (engano|erro)|"+ + BB + "(ha recibido|recibió|recibe) (este|esta) (mensaje|correo)" + BA+ + "[^.]{0,20} por error|"+ // the "think before printing" footer, tied to its environmental ending rather+ // than to `antes de imprimir`, which a request uses too.+ + BB + "antes de imprimir" + BA + "[^.]{0,100}"+ + "(meio ambiente|medio ambiente|natureza|planeta|realmente necess)|"+ + BB + "(meio|medio) ambiente" + BA + "[^.]{0,30}antes de imprimir|"+ // French: tied to "ce message/cet e-mail" rather than the bare word+ // `confidentiel`, which a sender's own request uses just as often.+ + BB + "(ce|cet|cette) (message|e-?mail|mail|courriel)" + BA + "[^.]{0,80}"+ + "(confidentiel|privil[eé]gi)|"+ + BB + "avez re[çc]u (ce|cet|cette) (message|e-?mail|mail)" + BA+ + "[^.]{0,20} par erreur|"+ + BB + "(usage exclusif|exclusivement|uniquement)" + BA + "[^.]{0,30}destinataire)");++ /**+ * A necessary condition for DISCLAIMER, checked first because it is 40 times cheaper.+ *+ * <p>Every branch of DISCLAIMER requires at least one of these literals, so a paragraph+ * holding none of them cannot match and does not have to be scanned by a pattern whose+ * twenty alternations each carry two word-boundary lookarounds over a 10,000-character+ * class. Measured: DISCLAIMER alone was 4.9 ms per case on the sweep corpus, which was+ * the whole cost of cleaning a message.+ *+ * <p>This is a filter, never a decision: a hit still runs DISCLAIMER, so a false positive+ * costs only time. What would be a defect is a false NEGATIVE, so the literals are+ * matched with the same flags as DISCLAIMER itself -- `re.I` folds more than lowercasing+ * does, U+017F LATIN SMALL LETTER LONG S against `s` among it, and a filter that lowered+ * the text instead would reject a paragraph the pattern accepts. EmailTest asserts the+ * implication over every branch and over the whole corpus.+ */+ static final Pattern DISCLAIMER_HINT = compile(+ "confidential|confidencia|sigilos|privilegiad|confidentiel|privil[eé]gi"+ + "|antes de imprimir|received this"+ + "|recebeu|recebido|receber|ha recibido|recibi[óo]|recibe|avez re[çc]u"+ + "|uso exclusivo|exclusivamente|[úu]nicamente|usage exclusif|exclusivement"+ + "|uniquement");++ static final Pattern SENTENCE = compile("(?<=[.!?])" + S + "+");+ static final Pattern BLANK_LINE = compile("\n" + S + "*\n");+ static final Pattern SPACES_AND_TABS = compile("[ \t]+");++ private Patterns() {+ }+ }++ /**+ * The same text, with every code point on which Java's {@code \w} and Python's disagree+ * replaced by one they agree about.+ *+ * <p>{@code \b} is defined in terms of {@code \w}, so a boundary is Python's boundary+ * exactly when the two classes agree on the characters either side of it. They do not agree+ * everywhere: Java's admits the combining marks and join controls Python's excludes, and+ * Python's admits the letter- and other-numbers (U+2160 ROMAN NUMERAL ONE, U+00BD VULGAR+ * FRACTION ONE HALF) that Java's, being alphabetic plus Nd plus Pc, excludes. Which code+ * points fall in each set is a property of the RUNNING JDK, so it is probed here rather than+ * tabulated -- a table would be right on one JDK and wrong on the next, which is the defect+ * this port shipped in its pre-tokenizer.+ *+ * <p>Two substitutes, chosen so a replacement cannot change which literal matches:+ * <ul>+ * <li>Java says word, Python says not: U+0000, which Java also calls a non-word character.+ * <li>Python says word, Java says not: {@code '0'}, which Java calls a word character. No+ * literal in any pattern here contains a digit, so a digit can neither complete nor+ * break one.+ * </ul>+ * Both substitutes are matched by the {@code [^.]} windows the patterns measure, so a window+ * counts the same number of characters as it does in the original.+ *+ * <p>ASCII cannot disagree -- both classes are the same there -- so an ASCII-only message,+ * which is most of them, returns the original string without copying it.+ */+ static String boundaryView(String text) {+ int firstDisagreement = -1;+ int i = 0;+ while (i < text.length()) {+ int cp = text.codePointAt(i);+ if (cp > 0x7F && javaWord(cp) != UnicodeTables.isPythonWordChar(cp)) {+ firstDisagreement = i;+ break;+ }+ i += Character.charCount(cp);+ }+ if (firstDisagreement < 0) {+ return text;+ }+ StringBuilder out = new StringBuilder(text.length());+ out.append(text, 0, firstDisagreement);+ i = firstDisagreement;+ while (i < text.length()) {+ int cp = text.codePointAt(i);+ if (cp > 0x7F && javaWord(cp) != UnicodeTables.isPythonWordChar(cp)) {+ out.append(UnicodeTables.isPythonWordChar(cp) ? '0' : NUL_PLACEHOLDER);+ } else {+ out.appendCodePoint(cp);+ }+ i += Character.charCount(cp);+ }+ return out.toString();+ }++ /** Java calls this a non-word character, and no pattern here holds it as a literal. */+ private static final char NUL_PLACEHOLDER = '\u0000';++ /**+ * Whether the RUNNING JDK calls this code point a word character.+ *+ * <p>Spelled out from the definition {@code java.util.regex.Pattern} documents for+ * {@code \w} under {@code UNICODE_CHARACTER_CLASS} -- alphabetic, the three mark categories,+ * decimal digits, connector punctuation and the join controls -- rather than asked of a+ * {@link Patterns#JAVA_WORD} matcher, which allocates a string and a matcher per character+ * and cost more than the rest of the cleaner put together. Every term reads the JDK's own+ * Unicode, so this stays whatever that JDK's {@code \w} is, which is the point: the view is+ * built by comparing the JDK against the recorded table, so this side has to be the JDK's+ * answer and not ours. {@code EmailTest} compares the two forms over all 1,114,112 code+ * points, so the spelled-out version cannot drift from the pattern it replaces.+ */+ private static boolean javaWord(int codePoint) {+ if (Character.isAlphabetic(codePoint) || Character.isDigit(codePoint)) {+ return true;+ }+ switch (Character.getType(codePoint)) {+ case Character.NON_SPACING_MARK:+ case Character.ENCLOSING_MARK:+ case Character.COMBINING_SPACING_MARK:+ case Character.DECIMAL_DIGIT_NUMBER:+ case Character.CONNECTOR_PUNCTUATION:+ return true;+ default:+ return codePoint == 0x200C || codePoint == 0x200D; // the join controls+ }+ }++ /** A character class's contents, for nesting one inside a larger class. */+ private static String inner(String characterClass) {+ return characterClass.substring(1, characterClass.length() - 1);+ }++ // ------------------------------------------------------------------- the sign-off rule++ /**+ * Python's {@code _drop_marks}: remove combining marks that ride on a preceding character.+ *+ * <p>Only a mark with a base goes. One that opens the string, or follows a space, has nothing+ * to ride on and stays, so it still separates tokens -- which is what makes the port's leading+ * {@code \p{Lu}\p{Lt}\p{Lo}} reject it.+ */+ static String dropMarks(String text) {+ StringBuilder kept = new StringBuilder(text.length());+ int i = 0;+ while (i < text.length()) {+ int cp = text.codePointAt(i);+ int width = Character.charCount(cp);+ boolean hasBase = kept.length() > 0;+ boolean baseIsSpace = hasBase+ && UnicodeTables.isSpace(kept.codePointBefore(kept.length()));+ if (!(UnicodeTables.isMark(cp) && hasBase && !baseIsSpace)) {+ kept.appendCodePoint(cp);+ }+ i += width;+ }+ return kept.toString();+ }++ /**+ * Python's {@code _SIGNOFF_TAIL} and {@code _SIGNOFF_TOKEN}, as one scan.+ *+ * <p>{@code ^[\s,;:!.]*(?:[^\W\d_][\w'-]*[\s,.]*){0,3}$}, then every token's first letter must+ * be Lu, Lt or Lo. A scanner rather than a pattern because the two classes are Python's: the+ * token opener is {@code [^\W\d_]} and the rest is {@code \w}, and Java's {@code \w} is a+ * different set -- it admits the combining marks this has just dropped and the join controls+ * that must keep separating tokens.+ *+ * <p>The repetition is greedy and each turn consumes at least one character, so taking as many+ * as possible and then requiring the end of input is what the pattern's backtracking amounts+ * to: with a fourth token present no smaller count reaches the end either.+ */+ private static boolean tailIsNameOnly(String tail) {+ int i = 0;+ int length = tail.length();+ i = skip(tail, i, ",;:!."); // [\s,;:!.]*+ for (int token = 0; token < 3 && i < length; token++) {+ int cp = tail.codePointAt(i);+ if (!UnicodeTables.isWordChar(cp)) { // [^\W\d_]+ break;+ }+ // Every token's first letter carries the rule the whole scan exists for.+ if (!UnicodeTables.isInitial(cp)) {+ return false;+ }+ i += Character.charCount(cp);+ while (i < length) { // [\w'-]*+ int next = tail.codePointAt(i);+ if (!UnicodeTables.isPythonWordChar(next) && next != '\'' && next != '-') {+ break;+ }+ i += Character.charCount(next);+ }+ i = skip(tail, i, ",."); // [\s,.]*+ }+ return i == length;+ }++ /** Advance over Python whitespace and any of {@code extra}. */+ private static int skip(String text, int from, String extra) {+ int i = from;+ while (i < text.length()) {+ int cp = text.codePointAt(i);+ if (!UnicodeTables.isSpace(cp) && extra.indexOf(cp) < 0) {+ return i;+ }+ i += Character.charCount(cp);+ }+ return i;+ }++ /** Python's {@code _is_english_signoff}. */+ static boolean isEnglishSignoff(String line) {+ Matcher head = Patterns.SIGNOFF_HEAD.matcher(line);+ if (!head.lookingAt()) {+ return false;+ }+ return tailIsNameOnly(dropMarks(line.substring(head.end())));+ }++ /** Whether any signature marker claims this line. */+ private static boolean isSignature(String line) {+ return Patterns.SIGNOFF_DASHES.matcher(line).lookingAt()+ || isEnglishSignoff(line)+ || Patterns.SIGNOFF_ROMANCE.matcher(line).lookingAt();+ }++ // -------------------------------------------------------------------- disclaimers++ /**+ * Python's {@code _starts_new_sentence}: the first letter is uppercase.+ *+ * <p>A line in an uncased script never starts a new piece, so wrapped boilerplate in those+ * scripts still drops whole.+ */+ private static boolean startsNewSentence(String line) {+ int i = 0;+ while (i < line.length()) {+ int cp = line.codePointAt(i);+ if (UnicodeTables.isAlpha(cp)) {+ return UnicodeTables.isUpper(cp);+ }+ i += Character.charCount(cp);+ }+ return false;+ }++ /** Python's {@code _split_fused_lines}. */+ private static List<String> splitFusedLines(String sentence) {+ if (sentence.indexOf('\n') < 0) {+ return List.of(sentence);+ }+ List<String> pieces = new ArrayList<>();+ StringBuilder buffer = new StringBuilder();+ for (String raw : sentence.split("\n", -1)) {+ String line = UnicodeTables.strip(raw);+ if (line.isEmpty()) {+ continue;+ }+ if (buffer.length() > 0 && startsNewSentence(line)) {+ pieces.add(buffer.toString());+ buffer.setLength(0);+ buffer.append(line);+ } else {+ if (buffer.length() > 0) {+ buffer.append(' ');+ }+ buffer.append(line);+ }+ }+ if (buffer.length() > 0) {+ pieces.add(buffer.toString());+ }+ return pieces;+ }++ /**+ * Python's {@code _strip_disclaimer}.+ *+ * <p>A paragraph goes whole only when EVERY sentence in it is boilerplate; otherwise only the+ * boilerplate sentences go. A footer that runs on without a blank line used to take the+ * sender's actual request with it, which is worse than leaving one boilerplate line behind.+ */+ /** DISCLAIMER, behind its filter. */+ private static boolean isDisclaimer(String text) {+ return Patterns.DISCLAIMER_HINT.matcher(text).find()+ && Patterns.DISCLAIMER.matcher(boundaryView(text)).find();+ }++ private static String stripDisclaimer(String paragraph) {+ if (!isDisclaimer(paragraph)) {+ return paragraph; // nothing to do: keep the original line structure+ }+ List<String> pieces = new ArrayList<>();+ for (String part : Patterns.SENTENCE.split(paragraph, -1)) {+ String trimmed = UnicodeTables.strip(part);+ if (trimmed.isEmpty()) {+ continue;+ }+ if (isDisclaimer(trimmed)) {+ pieces.addAll(splitFusedLines(trimmed));+ } else {+ pieces.add(trimmed);+ }+ }+ StringBuilder out = new StringBuilder();+ for (String piece : pieces) {+ if (isDisclaimer(piece)) {+ continue;+ }+ if (out.length() > 0) {+ out.append(' ');+ }+ out.append(piece);+ }+ return out.toString();+ }++ // -------------------------------------------------------------------- the cleaner++ /** {@link #cleanEmailBody(String, int)} with the default budget. */+ public static String cleanEmailBody(String body) {+ return cleanEmailBody(body, DEFAULT_MAX_CHARS);+ }++ /**+ * Remove quoted history, signatures and disclaimers, and cut the result to {@code maxChars}.+ *+ * @param body the raw message body; {@code null} is treated as empty, as Python's+ * {@code (body or "")} is+ * @param maxChars the length the result is cut to+ */+ public static String cleanEmailBody(String body, int maxChars) {+ String text = (body == null ? "" : body)+ .replace("\r\n", "\n").replace("\r", "\n").replace("\\n", "\n");+ // Bound regex work before the expensive patterns below: DISCLAIMER uses [^.]{0,60/80/100}+ // alternations whose cost grows with input length, and only maxChars are ever returned.+ if (maxChars > 0 && text.length() > (long) maxChars * 4) {+ text = text.substring(0, maxChars * 4);+ } else if (maxChars <= 0) {+ // Python slices with max_chars * 4, which is <= 0 here: `text[:0]` is empty, and so is+ // the result. Kept explicit because a negative length is an exception in Java.+ text = "";+ }+ String[] source = text.split("\n", -1);+ List<String> lines = new ArrayList<>(source.length);+ for (int i = 0; i < source.length; i++) {+ String line = source[i];+ if (!lines.isEmpty() && matchesAny(Patterns.QUOTE_HEADERS, line)) {+ break;+ }+ if (!lines.isEmpty() && Patterns.HEADER_FROM_NAME.matcher(line).lookingAt()+ && i + 1 < source.length+ && Patterns.HEADER_NEXT.matcher(source[i + 1]).lookingAt()) {+ break;+ }+ if (Patterns.ATTRIBUTION_TAIL.matcher(line).lookingAt() && !lines.isEmpty()) {+ if (Patterns.ATTRIBUTION_HEAD.matcher(lines.get(lines.size() - 1)).lookingAt()) {+ lines.remove(lines.size() - 1);+ }+ break;+ }+ if (stripLeading(line).startsWith(">")) {+ continue;+ }+ lines.add(stripTrailing(line));+ }+ // The signature search starts partway in, so a closing that opens the message is left+ // alone: `max(1, min(int(len*0.6), len-8))`.+ int cut = lines.size();+ int start = Math.max(1, Math.min((int) (lines.size() * 0.6), lines.size() - 8));+ for (int i = start; i < lines.size(); i++) {+ int width = UnicodeTables.strip(lines.get(i)).length();+ if ((width <= 40 && isSignature(lines.get(i)))+ || (width <= 60 && Patterns.DEVICE_FOOTER.matcher(lines.get(i)).lookingAt())) {+ cut = i;+ break;+ }+ }+ lines = lines.subList(0, cut);++ StringBuilder joined = new StringBuilder();+ for (int i = 0; i < lines.size(); i++) {+ if (i > 0) {+ joined.append('\n');+ }+ joined.append(lines.get(i));+ }+ StringBuilder out = new StringBuilder();+ for (String paragraph : Patterns.BLANK_LINE.split(joined.toString(), -1)) {+ String cleaned = UnicodeTables.strip(stripDisclaimer(paragraph));+ if (cleaned.isEmpty()) {+ continue;+ }+ if (out.length() > 0) {+ out.append("\n\n");+ }+ out.append(cleaned);+ }+ String result = Patterns.SPACES_AND_TABS.matcher(out.toString()).replaceAll(" ");+ return result.length() > maxChars ? result.substring(0, Math.max(maxChars, 0)) : result;+ }++ private static boolean matchesAny(List<Pattern> patterns, String line) {+ for (Pattern pattern : patterns) {+ if (pattern.matcher(line).lookingAt()) {+ return true;+ }+ }+ return false;+ }++ /** Python's {@code str.lstrip()} with no argument. */+ private static String stripLeading(String text) {+ int i = 0;+ while (i < text.length()) {+ int cp = text.codePointAt(i);+ if (!UnicodeTables.isSpace(cp)) {+ break;+ }+ i += Character.charCount(cp);+ }+ return text.substring(i);+ }++ /** Python's {@code str.rstrip()} with no argument. */+ private static String stripTrailing(String text) {+ int end = text.length();+ while (end > 0) {+ int cp = text.codePointBefore(end);+ if (!UnicodeTables.isSpace(cp)) {+ break;+ }+ end -= Character.charCount(cp);+ }+ return text.substring(0, end);+ }++ // -------------------------------------------------------------------- the state++ /** {@link #emailState(String, String, String, boolean, int, Map)} with the defaults. */+ public static Map<String, Object> emailState(String subject, String body) {+ return emailState(subject, body, null, true, DEFAULT_MAX_CHARS, Map.of());+ }++ /** {@link #emailState(String, String, String, boolean, int, Map)} with a sender. */+ public static Map<String, Object> emailState(String subject, String body, String sender) {+ return emailState(subject, body, sender, true, DEFAULT_MAX_CHARS, Map.of());+ }++ /**+ * A clean state for email classification.+ *+ * <p>{@code maxChars} is the budget {@link #cleanEmailBody} cuts the body to, and it is worth+ * raising for a long message: at the default the body stops after 3,000 characters, so a+ * request arriving in the last paragraphs never reaches the model. Ignored when+ * {@code clean} is false, which passes the body through whole.+ *+ * <p>Any entry of {@code extra} becomes a field of the state and is therefore read by the+ * model; a wrong key here is an input mutation, not an error, which is the reference's+ * behaviour and is why it is not validated. A {@code null} value is dropped, as the reference+ * drops it.+ *+ * @return a mutable, insertion-ordered map: {@code subject}, {@code body}, then {@code from}+ * if a sender was given, then the extras+ */+ public static Map<String, Object> emailState(String subject, String body, String sender,+ boolean clean, int maxChars,+ Map<String, Object> extra) {+ Map<String, Object> state = new LinkedHashMap<>();+ state.put("subject", UnicodeTables.strip(subject == null ? "" : subject));+ state.put("body", clean ? cleanEmailBody(body, maxChars) : (body == null ? "" : body));+ // Python's `if sender:` -- an empty string is falsy there, so an empty sender adds no+ // field rather than an empty one.+ if (sender != null && !sender.isEmpty()) {+ state.put("from", sender);+ }+ if (extra != null) {+ for (Map.Entry<String, Object> entry : extra.entrySet()) {+ if (entry.getValue() != null) {+ state.put(entry.getKey(), entry.getValue());+ }+ }+ }+ return state;+ }++ /**+ * The email question set, re-exported because {@code from laya.email import email_questions}+ * is a path callers already have. One definition, in {@link Presets}; this is the alias.+ */+ public static Map<String, Question> emailQuestions() {+ return Presets.email();+ }++ /** {@link #emailQuestions()} with the caller's own categories, as {@link Presets#email} takes. */+ public static Map<String, Question> emailQuestions(Map<String, String> categories) {+ return Presets.email(categories);+ }+}diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/EmailTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/EmailTest.javanew file mode 100644index 00000000..36cdb262--- /dev/null+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/EmailTest.java@@ -0,0 +1,341 @@+package com.convaiinnovations.laya;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertSame;+import static org.junit.jupiter.api.Assertions.assertTrue;+import static org.junit.jupiter.api.DynamicTest.dynamicTest;++import com.convaiinnovations.laya.lang.UnicodeTables;+import java.lang.reflect.Field;+import java.lang.reflect.Method;+import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import java.util.regex.Pattern;+import org.junit.jupiter.api.DisplayName;+import org.junit.jupiter.api.DynamicTest;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.TestFactory;++/**+ * {@code laya.email}, against expectations recorded by running the reference.+ *+ * <p>The corpus is not a sample. Every case is a rule the module's comments argue for: a marker+ * that must fire, and -- more of them -- a near-miss that must NOT, because a cleaner that is too+ * eager deletes the sender's request, which is worse than leaving boilerplate behind. Eight of the+ * cases exist because a mutant survived without them, and they are named so that a later edit+ * which "simplifies" one of those rules has something to fail.+ */+final class EmailTest {++ @SuppressWarnings("unchecked")+ private static List<Object> rows(String key) {+ return (List<Object>) Fixtures.load("email.json").get(key);+ }++ @TestFactory+ @DisplayName("every cleaned body is byte-identical to the reference")+ List<DynamicTest> cleanedMatches() {+ List<Object> cases = rows("cleaned");+ assertTrue(cases.size() >= 70,+ () -> "the email corpus lost cases: " + cases.size() + " of at least 70");+ List<DynamicTest> tests = new ArrayList<>();+ for (Object entry : cases) {+ Map<String, Object> row = asMap(entry);+ String name = (String) row.get("name");+ tests.add(dynamicTest(name, () -> assertEquals(+ row.get("result"), LayaEmail.cleanEmailBody((String) row.get("body")),+ () -> "clean_email_body diverged on " + name)));+ }+ return tests;+ }++ @TestFactory+ @DisplayName("every budget cuts where the reference cuts")+ List<DynamicTest> budgetsMatch() {+ List<Object> cases = rows("budgets");+ assertTrue(cases.size() >= 5, "the budget cases vanished");+ List<DynamicTest> tests = new ArrayList<>();+ for (Object entry : cases) {+ Map<String, Object> row = asMap(entry);+ String name = (String) row.get("name");+ int budget = ((Number) row.get("max_chars")).intValue();+ tests.add(dynamicTest(name + " (max " + budget + ")", () -> assertEquals(+ row.get("result"), LayaEmail.cleanEmailBody((String) row.get("body"), budget),+ () -> "clean_email_body diverged on " + name)));+ }+ return tests;+ }++ @TestFactory+ @DisplayName("every state matches the reference, field for field and in order")+ @SuppressWarnings("unchecked")+ List<DynamicTest> statesMatch() {+ List<Object> cases = rows("states");+ assertTrue(cases.size() >= 7, "the state cases vanished");+ List<DynamicTest> tests = new ArrayList<>();+ for (Object entry : cases) {+ Map<String, Object> row = asMap(entry);+ String name = (String) row.get("name");+ tests.add(dynamicTest(name, () -> {+ Map<String, Object> want = (Map<String, Object>) row.get("result");+ Map<String, Object> got = LayaEmail.emailState(+ (String) row.get("subject"), (String) row.get("body"),+ (String) row.get("sender"), (Boolean) row.get("clean"),+ ((Number) row.get("max_chars")).intValue(),+ new LinkedHashMap<>(asMap(row.get("extra"))));+ assertEquals(want, got, () -> "email_state diverged on " + name);+ // Key order is not cosmetic: it is the order the model is shown the fields in.+ assertEquals(new ArrayList<>(want.keySet()), new ArrayList<>(got.keySet()),+ () -> "email_state field order diverged on " + name);+ }));+ }+ return tests;+ }++ @Test+ @DisplayName("the question set is the same object graph as the preset")+ void questionsAreThePreset() {+ assertEquals(Presets.email().keySet(), LayaEmail.emailQuestions().keySet());+ assertEquals(Boolean.TRUE, Fixtures.load("email.json").get("questions_match_presets"),+ "the reference's re-export no longer equals laya.presets.email_questions");+ }++ // ------------------------------------------------------------------ the sign-off rule++ @Test+ @DisplayName("a name may begin with any Lu, Lt or Lo letter, and with nothing else")+ void signoffInitials() {+ // Each of these is a rule UPPER would get wrong; see UnicodeTablesTest for the counts.+ assertTrue(LayaEmail.isEnglishSignoff("Regards, Ana"));+ assertTrue(LayaEmail.isEnglishSignoff("Regards, Łukasz"), "Lu outside Latin-1");+ assertTrue(LayaEmail.isEnglishSignoff("Regards, 山田"), "Lo: a caseless script");+ assertTrue(LayaEmail.isEnglishSignoff("Regards, Džarko"), "Lt: titlecase");+ assertFalse(LayaEmail.isEnglishSignoff("Thanks, Ⓐ"),+ "U+24B6 is Uppercase but category So, so it is not a letter");+ assertFalse(LayaEmail.isEnglishSignoff("Thanks, żaneta"), "a lowercase name is prose");+ assertFalse(LayaEmail.isEnglishSignoff("Thanks for the quick reply."));+ assertFalse(LayaEmail.isEnglishSignoff("Regards, Ana Maria Souza Lima"),+ "the tail allows at most three tokens");+ }++ @Test+ @DisplayName("a mark rides on its base, and a mark without one still separates tokens")+ void marksAreDroppedOnlyWithABase() {+ assertEquals("Jose", LayaEmail.dropMarks("José"), "a mark with a base goes");+ assertEquals("Jose", LayaEmail.dropMarks("Jo͏se"),+ "including U+034F, whose canonical combining class is zero");+ assertEquals("́Ana", LayaEmail.dropMarks("́Ana"),+ "a mark opening the string has no base");+ assertEquals("a ́b", LayaEmail.dropMarks("a ́b"),+ "nor does one following a space");+ // and the consequence, which is the reason the rule exists+ assertTrue(LayaEmail.isEnglishSignoff("Regards, José"));+ assertFalse(LayaEmail.isEnglishSignoff("Regards, ́Ana"),+ "a kept mark is not a token opener, so this is not a sign-off");+ }++ // --------------------------------------------------- the two performance-critical equalities++ @Test+ @DisplayName("javaWord is exactly the JDK's own regex word class, over every code point")+ void javaWordEqualsThePattern() throws Exception {+ // It replaced a matcher call per character, which cost more than the rest of the cleaner.+ // The replacement is only safe while it answers identically, so that is asserted rather+ // than argued: this side of boundaryView has to be the JDK's answer, not ours.+ Method javaWord = LayaEmail.class.getDeclaredMethod("javaWord", int.class);+ javaWord.setAccessible(true);+ Pattern word = Pattern.compile("\\w", Pattern.UNICODE_CHARACTER_CLASS);+ int examined = 0;+ for (int cp = 0; cp < 0x110000; cp++) {+ if (cp >= 0xD800 && cp <= 0xDFFF) {+ continue;+ }+ examined++;+ boolean spelled = (Boolean) javaWord.invoke(null, cp);+ boolean pattern = word.matcher(new String(Character.toChars(cp))).matches();+ if (spelled != pattern) {+ assertEquals(pattern, spelled, String.format("U+%04X", cp));+ }+ }+ assertEquals(0x110000 - 2048, examined, "every code point but the surrogates");+ }++ @Test+ @DisplayName("boundaryView rewrites exactly where the two word classes disagree")+ void boundaryViewSubstitutes() {+ assertSame("plain ascii", LayaEmail.boundaryView("plain ascii"),+ "an ASCII-only message is returned without copying");+ assertSame("café résumé", LayaEmail.boundaryView("café résumé"),+ "ordinary accented letters are word characters in both, so nothing is rewritten");+ // Java says word, Python says not -> U+0000, a non-word character to Java too.+ assertEquals("a b", LayaEmail.boundaryView("áb"), "a combining mark");+ assertEquals("a b", LayaEmail.boundaryView("ab"), "ZWNJ, a join control");+ assertEquals("a b", LayaEmail.boundaryView("ab"), "ZWJ");+ // Python says word, Java says not -> '0', which no literal in any pattern contains.+ assertEquals("a0b", LayaEmail.boundaryView("a½b"), "U+00BD is No: Python's word, not Java's");+ // Length is preserved, so the [^.]{0,N} windows count the same characters.+ for (String text : List.of("áb", "ab", "a½b", "x́½y")) {+ assertEquals(text.length(), LayaEmail.boundaryView(text).length(),+ () -> "boundaryView changed the length of " + text);+ }+ }++ @Test+ @DisplayName("neither substitute can appear in a pattern literal")+ void substitutesAreNotLiterals() throws Exception {+ // The whole argument for substituting is that the replacement cannot complete or break a+ // literal. That holds only while no pattern here holds a digit or a NUL as a literal, so+ // it is checked rather than asserted in a comment.+ for (Pattern pattern : allPatterns()) {+ String source = pattern.pattern();+ assertFalse(source.indexOf(' ') >= 0,+ () -> "a pattern holds a literal NUL: " + source.substring(0, 40));+ }+ // A digit may only appear inside the generated \d class or as a quantifier bound, never+ // as a literal to be matched, so the DISCLAIMER branches are checked directly.+ Pattern disclaimer = (Pattern) patternField("DISCLAIMER");+ assertFalse(disclaimer.pattern().matches(".*[^{,0-9]\\d[^}0-9,].*"),+ "a digit became a literal in DISCLAIMER; '0' is no longer a safe substitute");+ }++ @Test+ @DisplayName("the disclaimer filter never rejects what the pattern would accept")+ void hintIsANecessaryCondition() throws Exception {+ Pattern hint = (Pattern) patternField("DISCLAIMER_HINT");+ Pattern disclaimer = (Pattern) patternField("DISCLAIMER");+ // A filter is only sound as a NECESSARY condition: a false positive costs time, a false+ // negative keeps a disclaimer in the model's input. Checked over every recorded case,+ // which includes one example of every branch of the pattern.+ int fired = 0;+ for (Object entry : rows("cleaned")) {+ Map<String, Object> row = asMap(entry);+ String body = (String) row.get("body");+ for (String raw : body.split("\n", -1)) {+ final String paragraph = raw;+ if (disclaimer.matcher(LayaEmail.boundaryView(paragraph)).find()) {+ fired++;+ assertTrue(hint.matcher(paragraph).find(),+ () -> "the filter rejected a paragraph the pattern accepts: "+ + paragraph);+ }+ }+ }+ final int hits = fired;+ assertTrue(hits >= 9,+ () -> "the corpus no longer exercises the disclaimer pattern: " + hits + " hits");++ // The corpus count is a floor, not coverage. Every BRANCH of the pattern is named here,+ // so a branch whose required literal is missing from the filter fails this rather than+ // waiting for a message in production to be the first to notice.+ String[] oneExamplePerBranch = {+ "This email and its contents are confidential and intended solely for the addressee.",+ "This message is confidential and may also privileged.",+ "If you have received this email in error please delete it.",+ "Esta mensagem e seus anexos sao confidenciais e de uso exclusivo do destinatario.",+ "Este mensaje es confidencial y para uso exclusivo del destinatario.",+ "Uso exclusivo do destinatario desta mensagem.",+ "Se recebeu esta mensagem por engano, apague-a.",+ "Usted ha recibido este mensaje por error.",+ "Antes de imprimir pense no meio ambiente.",+ "Pense no meio ambiente antes de imprimir.",+ "Ce message est confidentiel et destine uniquement au destinataire.",+ "Vous avez recu ce message par erreur.",+ "Usage exclusif du destinataire.",+ };+ for (String example : oneExamplePerBranch) {+ assertTrue(disclaimer.matcher(LayaEmail.boundaryView(example)).find(),+ () -> "this no longer matches DISCLAIMER at all: " + example);+ assertTrue(hint.matcher(example).find(),+ () -> "the filter would reject a disclaimer the pattern accepts: " + example);+ }+ }++ // ------------------------------------------------------------------------- the state++ @Test+ @DisplayName("a null or empty sender adds no field, and a null extra is dropped")+ void stateOmissions() {+ assertFalse(LayaEmail.emailState("s", "b", null).containsKey("from"));+ assertFalse(LayaEmail.emailState("s", "b", "").containsKey("from"),+ "Python's `if sender:` is false for an empty string");+ assertTrue(LayaEmail.emailState("s", "b", "[email protected]").containsKey("from"));+ Map<String, Object> extra = new LinkedHashMap<>();+ extra.put("kept", 1);+ extra.put("dropped", null);+ Map<String, Object> state = LayaEmail.emailState("s", "b", null, true, 3000, extra);+ assertTrue(state.containsKey("kept"));+ assertFalse(state.containsKey("dropped"), "a null extra is dropped, as the reference drops it");+ assertEquals(List.of("subject", "body", "kept"), new ArrayList<>(state.keySet()));+ }++ @Test+ @DisplayName("a null subject or body is empty, not an exception")+ void stateNulls() {+ Map<String, Object> state = LayaEmail.emailState(null, null);+ assertEquals("", state.get("subject"));+ assertEquals("", state.get("body"));+ assertEquals("", LayaEmail.cleanEmailBody(null));+ }++ @Test+ @DisplayName("a non-positive budget returns nothing rather than throwing")+ void nonPositiveBudget() {+ // Python slices with max_chars * 4 and then max_chars, and a slice to a non-positive+ // bound is empty. In Java a negative length is an exception, so the case is explicit.+ assertEquals("", LayaEmail.cleanEmailBody("a message", 0));+ assertEquals("", LayaEmail.cleanEmailBody("a message", -5));+ assertEquals("a", LayaEmail.cleanEmailBody("a message", 1));+ }++ @Test+ @DisplayName("the strip used on a subject is Python's, not Java's trim")+ void subjectStripIsPythons() {+ // U+00A0 and U+2007 are whitespace to str.strip() and not to String.trim(). The same gap+ // sat in the router's checkpoint names and routed a caller to the wrong model.+ assertEquals("Refund", LayaEmail.emailState(" Refund ", "b").get("subject"));+ assertTrue(UnicodeTables.isSpace(0x00a0) && UnicodeTables.isSpace(0x2007),+ "the premise of the assertion above");+ }++ // ------------------------------------------------------------------------- helpers++ @SuppressWarnings("unchecked")+ private static Map<String, Object> asMap(Object value) {+ return (Map<String, Object>) value;+ }++ private static Object patternField(String name) throws Exception {+ Class<?> patterns = Class.forName("com.convaiinnovations.laya.LayaEmail$Patterns");+ Field field = patterns.getDeclaredField(name);+ field.setAccessible(true);+ return field.get(null);+ }++ private static List<Pattern> allPatterns() throws Exception {+ Class<?> patterns = Class.forName("com.convaiinnovations.laya.LayaEmail$Patterns");+ List<Pattern> out = new ArrayList<>();+ for (Field field : patterns.getDeclaredFields()) {+ field.setAccessible(true);+ Object value;+ try {+ value = field.get(null);+ } catch (IllegalAccessException problem) {+ continue;+ }+ if (value instanceof Pattern) {+ out.add((Pattern) value);+ } else if (value instanceof List<?>) {+ for (Object element : (List<?>) value) {+ if (element instanceof Pattern) {+ out.add((Pattern) element);+ }+ }+ }+ }+ assertTrue(out.size() >= 15, () -> "only " + out.size() + " patterns found by reflection");+ return out;+ }+}diff --git a/laya-java/scripts/gen_fixtures.py b/laya-java/scripts/gen_fixtures.pyindex 632dab71..f9dd25f3 100644--- a/laya-java/scripts/gen_fixtures.py+++ b/laya-java/scripts/gen_fixtures.py@@ -1747,6 +1747,201 @@ def predict_golden():"single": single, "batch": batches}+def email_clean():+ """`laya.email`'s cleaner and state builder, over a corpus built from its own reasoning.++ Every case below is a rule the module's comments argue for, which is what makes this a+ contract rather than a sample: a marker that must fire, a near-miss that must NOT fire because+ it is ordinary prose, and the Unicode-sensitive sign-off rule whose tail is matched+ structurally. The near-misses matter more than the hits -- a cleaner that is too eager deletes+ the sender's actual request, which is worse than leaving boilerplate behind, and the module+ says so in four separate places.+ """+ from laya import email as email_mod++ q = "I cannot log in and need a password reset."+ cases = [+ # -- nothing to do+ ("empty", ""),+ ("plain", q),+ ("crlf", "line one\r\nline two\r\rline three"),+ ("literal-backslash-n", "first\\nsecond"),+ ("collapses-runs", "a b\t\tc"),+ ("blank-paragraphs", "first\n\n\n\nsecond\n \n\nthird"),++ # -- quote headers that MUST cut+ ("on-wrote", q + "\n\nOn Tue, 3 Sep 2025 at 10:04, Ana <[email protected]> wrote:\n> older text"),+ ("em-escreveu", q + "\n\nEm ter., 3 de set. de 2025, Ana escreveu:\n> antigo"),+ ("el-escribio", q + "\n\nEl mar, 3 sept 2025 a las 10:04, Ana escribió:\n> viejo"),+ ("le-a-ecrit", q + "\n\nLe mar. 3 sept. 2025 a écrit :\n> ancien"),+ ("original-message", q + "\n\n-----Original Message-----\nolder"),+ ("mensagem-original", q + "\n\n-----Mensagem original-----\nantigo"),+ ("message-d-origine", q + "\n\n-----Message d'origine-----\nancien"),+ ("underscore-rule", q + "\n\n________________\nolder"),+ ("from-with-address", q + "\n\nFrom: Ana <[email protected]>\nolder"),+ ("de-with-address", q + "\n\nDe: Ana <[email protected]>\nantigo"),+ ("de-spaced-colon", q + "\n\nDe : Marie <[email protected]>\nancien"),++ # -- near-misses that MUST NOT cut: the module names each of these as body text+ ("em-without-date", q + "\n\nEm resposta ao que você escreveu:\nmais detalhes"),+ ("le-without-date", q + "\n\nLe rapport que vous avez écrit :\nplus de détails"),+ ("from-as-prose", "From: my side the integration works, but please refund the charge."),+ ("de-as-date-range", "De: 10/09 a 15/09 estarei fora, por favor adie a cobranca."),+ ("quote-header-first-line", "On Tue, 3 Sep 2025, Ana <[email protected]> wrote:\nstill the body"),++ # -- the bare-name Outlook header needs its neighbour to be told from prose+ ("from-name-then-sent", q + "\n\nFrom: Maria Souza\nSent: Tuesday\nolder"),+ ("de-name-then-enviado", q + "\n\nDe: Maria Souza\nEnviado: terca\nantigo"),+ ("de-name-then-envoye", q + "\n\nDe : Marie Dupont\nEnvoyé : mardi\nancien"),+ ("from-name-then-date-year", q + "\n\nFrom: Maria Souza\nDate: 3 Sep 2025\nolder"),+ ("de-name-no-neighbour", q + "\n\nDe: Maria Souza\nPara: 15/09\nstill body"),++ # -- Gmail's wrapped attribution: the tail cuts and takes its head with it+ ("wrapped-attribution", q + "\n\nEm ter., 3 de set. de 2025,\[email protected]> escreveu:\n> antigo"),+ ("wrapped-attribution-fr", q + "\n\nLe mar. 3 sept. 2025,\[email protected]> a écrit :\n> ancien"),+ ("attribution-tail-no-head", q + "\n\[email protected]> escreveu:\n> antigo"),++ # -- quoted lines are dropped wherever they appear+ ("angle-quoted", q + "\n> quoted\n > indented quote\nmore of my message"),++ # -- sign-offs. The search window starts at 60% of the lines, so each needs a body.+ ("signoff-regards-name", "\n".join([q] * 9 + ["Regards, Ana", "[email protected]"])),+ ("signoff-latin-ext-name", "\n".join([q] * 9 + ["Regards, Łukasz", "[email protected]"])),+ # The case INITIAL governs: Lo is a letter a name may begin with, and `str.isupper()`+ # is false for every caseless script, so UPPER here would lose the sign-off.+ ("signoff-caseless-name", "\n".join([q] * 9 + ["Regards, 山田", "[email protected]"])),+ # And the near-miss next to it: a Portuguese closing allows no trailing words at all,+ # so the same name after `Obrigado` is a sentence, not a signature.+ ("not-signoff-pt-with-name", "\n".join([q] * 9 + ["Obrigado, 山田", "[email protected]"])),+ ("signoff-titlecase-name", "\n".join([q] * 9 + ["Regards, Džarko", "[email protected]"])),+ ("signoff-combining-name", "\n".join([q] * 9 + ["Regards, José", "[email protected]"])),+ ("signoff-dashes", "\n".join([q] * 9 + ["--", "Ana"])),+ ("signoff-warmest", "\n".join([q] * 9 + ["Warmest regards, Ana"])),+ ("signoff-and-regards", "\n".join([q] * 9 + ["Thanks and regards, Ana"])),+ ("signoff-many-thanks", "\n".join([q] * 9 + ["Many thanks, Ana"])),+ # NOT sign-offs: the next word is not a name+ ("not-signoff-sentence", "\n".join([q] * 9 + ["Thanks for the quick reply."])),+ ("not-signoff-lowercase-other-script", "\n".join([q] * 9 + ["Thanks, żaneta"])),+ ("not-signoff-symbol-name", "\n".join([q] * 9 + ["Thanks, Ⓐ"])),+ ("not-signoff-zwnj", "\n".join([q] * 9 + ["Thanks, क्ष"])),+ # pt/es/fr closings cut only when they stand alone+ ("signoff-atenciosamente", "\n".join([q] * 9 + ["Atenciosamente,", "Ana"])),+ ("signoff-cordialement", "\n".join([q] * 9 + ["Cordialement,", "Marie"])),+ ("signoff-saudacoes", "\n".join([q] * 9 + ["Saudações!"])),+ ("not-signoff-obrigado-mas", "\n".join([q] * 9 + ["Obrigado pelo retorno, mas preciso do estorno."])),+ ("not-signoff-merci-mais", "\n".join([q] * 9 + ["Merci pour votre aide, mais le probleme persiste."])),+ # a closing before the window is left alone+ ("signoff-too-early", "Regards, Ana\n" + "\n".join([q] * 12)),++ # -- device footers: only a line that is nothing but the footer+ ("footer-iphone", "\n".join([q] * 9 + ["Sent from my iPhone"])),+ ("footer-samsung", "\n".join([q] * 9 + ["Enviado do meu smartphone Samsung Galaxy."])),+ ("footer-envoye-iphone", "\n".join([q] * 9 + ["Envoyé depuis mon iPhone"])),+ ("footer-get-outlook", "\n".join([q] * 9 + ["Get Outlook for iOS"])),+ ("not-footer-with-request", "\n".join([q] * 9 + ["Enviado do meu celular o comprovante ontem."])),++ # -- disclaimers: a paragraph goes whole only when every sentence is boilerplate+ ("disclaimer-en", q + "\n\nThis email and its contents are confidential and intended solely "+ "for the addressee."),+ ("disclaimer-en-wrapped", q + "\n\nThis email and its contents are\nconfidential and intended "+ "solely for the addressee."),+ ("disclaimer-received-in-error", q + "\n\nIf you have received this email in error please "+ "delete it."),+ ("disclaimer-pt", q + "\n\nEsta mensagem e seus anexos sao confidenciais e de uso exclusivo "+ "do destinatario."),+ ("disclaimer-es", q + "\n\nEste mensaje es confidencial y para uso exclusivo del destinatario."),+ ("disclaimer-fr", q + "\n\nCe message est confidentiel et destiné uniquement au destinataire."),+ ("disclaimer-print", q + "\n\nAntes de imprimir pense no meio ambiente."),+ ("disclaimer-mixed-paragraph", "Please refund the charge. This email and its contents are "+ "confidential and intended solely for the addressee."),+ ("disclaimer-fused-line", "The account is locked\nThis email and its contents are confidential "+ "and intended solely for the addressee."),+ # NOT disclaimers: the bare word is a sender's own+ ("not-disclaimer-question", "Is this confidential?"),+ ("not-disclaimer-colon", "Confidential: I need a refund."),+ ("not-disclaimer-print", "Antes de imprimir o boleto, confira o valor."),+ ("not-disclaimer-contract", "Preciso do contrato confidencial assinado."),+ ("not-disclaimer-fr-contract", "Je voudrais le contrat confidentiel signé."),++ # -- cases that exist because a mutant survived without them. Each one is the smallest+ # input that tells the port's rule apart from the plausible wrong rule next to it.+ #+ # A mark whose canonical combining CLASS is zero. U+034F is category Mn, so it is dropped+ # before the tail is matched and the name reads as one token; the combining-class table+ # would keep it, and a kept non-word character ends the scan short of the line.+ ("signoff-class-zero-mark", "\n".join([q] * 9 + ["Regards, Jo\u034Fse"])),+ # A mark with NO base: it follows a space, so the reference keeps it, and a kept mark is+ # not a token opener -- `[^\W\d_]` excludes marks -- so this is NOT a sign-off.+ ("not-signoff-baseless-mark", "\n".join([q] * 9 + ["Regards, \u0301Ana"])),+ # 44 characters: a sign-off by every other rule, kept only by the 40-character limit.+ ("not-signoff-too-wide",+ "\n".join([q] * 9 + ["Regards, Anastasia Konstantinopolitanopoulos"])),+ # A fourth token: `{0,3}` is the whole rule, so four names are a sentence.+ ("not-signoff-four-tokens", "\n".join([q] * 9 + ["Regards, Ana Maria Souza Lima"])),+ # The wrapped-attribution tail pops the line above it ONLY when that line is the+ # `On/Em/El/Le ...` head it belongs to. Here it is ordinary prose and must survive.+ ("attribution-tail-keeps-prose", q + "\nSee the details below.\[email protected]> escreveu:\n> antigo"),+ # U+0085 is a line terminator to Java and an ordinary character to Python's `.`, so a+ # header spanning one is recognised by the reference and only by a port that says so.+ ("quote-header-across-u0085", q + "\n\nOn Tue, 3 Sep 2025\u0085Ana wrote:\n> older"),+ # U+00A0 is Python `\s` and is not Java's; U+001C is Python `\s` and is not even+ # White_Space, so it is missed by Java's Unicode `\s` as well as by its default one.+ ("quote-header-nbsp-indent", q + "\n\n\u00A0On Tue, 3 Sep 2025, Ana wrote:\n> older"),+ ("quote-header-u001c-indent", q + "\n\n\u001COn Tue, 3 Sep 2025, Ana wrote:\n> older"),+ # Python's `\d` is the Nd category, not ASCII. The date these headers require can be+ # written in any decimal script, and an ASCII-only port keeps the quoted history instead.+ ("quote-header-arabic-indic-date",+ q + "\n\nEm ter., \u0663 de set. de \u0662\u0660\u0662\u0665, Ana escreveu:\n> antigo"),+ ("header-next-devanagari-year",+ q + "\n\nFrom: Maria Souza\nDate: 3 Sep \u0968\u0966\u0968\u096B\nolder"),++ # -- everything at once+ ("full-stack", "\n".join(+ [q, "Please check the attached receipt.", ""]+ + [q] * 7+ + ["Atenciosamente,", "Ana Souza", "Enviado do meu iPhone", "",+ "Esta mensagem e confidencial e de uso exclusivo do destinatario.", "",+ "Em ter., 3 de set. de 2025, Bruno <[email protected]> escreveu:",+ "> mensagem antiga"])),+ ]+ budgets = [+ ("budget-default", "x" * 50 + " " + q, 3000),+ ("budget-tiny", q, 12),+ ("budget-cut-mid-word", "abcdefghij " * 10, 25),+ # the 4x pre-truncation bound: input longer than max_chars*4 is cut before matching+ ("budget-four-x-bound", ("word " * 400) + "This email is confidential and intended solely "+ "for the addressee.", 100),+ ("budget-zero", q, 0),+ ]+ states = [+ ("state-plain", "Refund request", q, None, True, 3000, {}),+ ("state-with-sender", "Refund request", q, "[email protected]", True, 3000, {}),+ ("state-unclean", "Refund request", q + "\n\nSent from my iPhone", None, False, 3000, {}),+ ("state-extra-fields", "Refund", q, "[email protected]", True, 3000,+ {"priority": "high", "ticket": 42, "dropped": None}),+ ("state-blank-subject", " ", q, None, True, 3000, {}),+ ("state-none-ish", "", "", None, True, 3000, {}),+ ("state-budget", "S", "y" * 80, None, True, 20, {}),+ ]+ return {+ "cleaned": [{"name": name, "body": body,+ "result": email_mod.clean_email_body(body)}+ for name, body in cases],+ "budgets": [{"name": name, "body": body, "max_chars": n,+ "result": email_mod.clean_email_body(body, max_chars=n)}+ for name, body, n in budgets],+ "states": [{"name": name, "subject": subject, "body": body, "sender": sender,+ "clean": clean, "max_chars": n, "extra": extra,+ "result": email_mod.email_state(subject, body, sender=sender, clean=clean,+ max_chars=n, **extra)}+ for name, subject, body, sender, clean, n, extra in states],+ # The re-export the module exists to keep working for callers who import it from here.+ "questions_match_presets": (+ email_mod.email_questions() == __import__("laya.presets", fromlist=["x"])+ .email_questions()),+ }++FAMILIES = {"lang_tables.json": lang_tables,"lang_detect.json": lang_detect,@@ -1757,6 +1952,7 @@ FAMILIES = {"sequences.json": sequences,"decode.json": decode_answers,"python_json.json": python_json,+ "email.json": email_clean,"predict.json": predict_golden,}