Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(research): non-English fixed states for presentation_checks (#602, part b)
hiroki-abe-5817ee586ecd02
diff --git a/research/eval/README.md b/research/eval/README.mdindex 3ba5f01a..f73f2c06 100644--- a/research/eval/README.md+++ b/research/eval/README.md@@ -285,20 +285,56 @@ conditions were fixed before the 10-state run:The tightest margin is the english first-slot rate, at 0.054 against the 0.05 rule.An order-invariant checkpoint sits at exactly 0.333 on that check.+### Other languages++`--lang` runs both checks on fixed states in Japanese, Korean, Hindi or Turkish (#602).+Each set translates the ten English states one for one, with the level texts in the+same language. Every state routes to `multilingual` under `Router`: Japanese, Korean+and Hindi by script, Turkish by its non-English letters. The run prints the checkpoint+`Router` picks for each language. Korean, Hindi and Turkish are among the languages+whose per-language MASSIVE gains the multilingual model card lists, and they cover+three routing paths (Hangul, Devanagari, Latin with diacritics). Japanese is where+#131 was found. The gates are the English ones. Without `--lang` the run and its report are+unchanged.++```bash+python research/eval/presentation_checks.py --model convaiinnovations/laya \+ --subfolder multilingual --lang ja,ko,hi,tr --out langs.json+```++CPU, fp32, `convaiinnovations/laya@55cf4c4` (subfolder `multilingual`), laya 0.3.21.+Full output: `research/results/presentation_checks_langs.json`. The same run without+`--lang` reproduces the English row above exactly.++| language | `score_slot0_identical` (leave-one-out) | `score_first_slot_permuted` (leave-one-out) | parity max \|Δp\| |+|---|---|---|---|+| `ja` | −0.018 (−0.120 .. +0.031) PASS | **0.000** (0.000 .. 0.000) FAIL | 4.68e-5 |+| `ko` | **−0.570** (−0.622 .. −0.506) FAIL | **0.050** (0.019 .. 0.056) FAIL | 4.97e-5 |+| `hi` | **−0.225** (−0.300 .. −0.182) FAIL | **0.067** (0.037 .. 0.074) FAIL | 4.91e-5 |+| `tr` | +0.139 (+0.068 .. +0.174) PASS | **0.033** (0.019 .. 0.037) FAIL | 4.95e-5 |++The first-slot check fails in all four languages, as it does in English. The+identical-option control passes in Japanese and Turkish, so on those states it would+not catch the prior on its own. That is the case for running both checks in every+language.+### Tests`research/eval/test_presentation_checks.py` runs offline, with scripted logits inplace of a checkpoint:```bash-python research/eval/test_presentation_checks.py # 69 passed, 0 failed+python research/eval/test_presentation_checks.py # 154 passed, 0 failed```It pins the fixed inputs and both gates. It checks that the identical-optionquestions render as `level i: <same text>`, and that every level sits in every slotexactly twice. It also checks the metric arithmetic by hand, the leave-one-outbounds, the one-sided gates, and the exit codes. A scripted slot-0 hole fails both-checks, and an order-invariant model scores exactly 1/3.+checks, and an order-invariant model scores exactly 1/3. For each language it+pins ten distinct states, three levels in every slot twice, and routing to+`multilingual`. It also checks the `--lang` parsing. The default run gives the same+report as `lang="en"`.### Limits@@ -312,8 +348,10 @@ checks, and an order-invariant model scores exactly 1/3.(multilingual: −0.75 / −0.52 / −0.25 and −0.58 / −0.47 / −0.37.)* Passing is not accuracy. A checkpoint can clear both gates and still rank urgencybadly; this checks one known failure, not `score` quality.-* English only, `score` only, 10 states. The states are short support messages, so a- checkpoint's behaviour on long inputs or other languages is not covered here.+* `score` only, 10 states per language. The states are short support messages, so a+ checkpoint's behaviour on long inputs is not covered here. The Korean, Hindi and+ Turkish states and levels were written by a non-native speaker; corrections from+ native speakers are welcome.* Thresholds were set on CPU fp32. On CUDA, `Agent` runs the forward pass underreduced-precision autocast and `score_cases` does not. The parity check reports thatdifference instead of hiding it.diff --git a/research/eval/presentation_checks.py b/research/eval/presentation_checks.pyindex 5afabca4..d74e3762 100644--- a/research/eval/presentation_checks.py+++ b/research/eval/presentation_checks.py@@ -29,11 +29,21 @@ Both metrics read raw marker logits (before temperature) throughuses. ``parity`` checks that path against ``Agent.system_one`` before anything isreported.+Languages+---------+``--lang`` runs the same two checks on fixed states in Japanese, Korean, Hindi or+Turkish (#602): translations of the English states, with the level texts in the same+language. Every one of them routes to ``multilingual`` under ``Router``, so they test+the checkpoint #131 is about on the scripts it serves. The gates are the English ones.+Without ``--lang`` the run and its report are exactly what they were.+Usage-----python research/eval/presentation_checks.py --model convaiinnovations/layapython research/eval/presentation_checks.py --model convaiinnovations/laya \\--subfolder multilingual --out multilingual.json+ python research/eval/presentation_checks.py --model convaiinnovations/laya \\+ --subfolder multilingual --lang ja,ko,hi,tr --out multilingual-langs.jsonExit status: 0 every check passed, 1 a check failed, 2 the harness could not betrusted (parity with ``Agent.system_one`` above ``PARITY_TOL``).@@ -84,6 +94,83 @@ IDENTICAL_TEXTS = ("moderate", "a request")IDENTICAL_KS = (3, 4, 5)LEVELS = ("Not urgent", "Soon", "Work is blocked")+# Fixed states in other languages, for the multilingual checkpoint (#602). Each tuple follows+# STATES item for item, so the urgency spread is the same in every language; every state routes+# to `multilingual` under `Router` (tested offline). The level texts mirror LEVELS. They were+# written for this check by a non-native speaker, except Japanese; corrections are welcome.+LANGUAGES: Dict[str, Dict[str, Any]] = {+ "en": {"states": STATES, "instructions": INSTRUCTIONS, "identical_texts": IDENTICAL_TEXTS,+ "levels": LEVELS},+ "ja": {+ "states": (+ "お手すきの際に、アカウントの請求先住所をハーバー通り12番地に変更していただけますでしょうか。急ぎではございません。",+ "3月分の請求書に、同じシートの料金が二重に計上されております。今週中にご確認いただけますでしょうか。",+ "今朝からCSVの書き出しが失敗しており、復旧するまで経理部門が月次の締めを行えない状況です。",+ "昨日のオンボーディングのお打ち合わせ、ありがとうございました。こちらから特にお願いすることはございません。",+ "来四半期にライセンスを10件追加することを検討しております。ご都合のよいときにお見積もりをお送りいただけますか。",+ "弊社の全ユーザーがログインできない状態です。誰もダッシュボードにアクセスできず、お客様をお待たせしております。",+ "モバイルアプリのレポートで、ときどきタイムゾーンが誤って表示されます。不便ではありますが、回避はできております。",+ "APIキーが誤って無効化され、現在すべての本番リクエストが401エラーを返しております。",+ "明朝に給与計算の実行を控えておりますが、連携機能が依然としてスキーマエラーで従業員ファイルを受け付けません。",+ "いずれ管理画面にダークモードを追加していただくことは可能でしょうか。社内でも何名か希望しております。",+ ),+ "instructions": "この依頼の緊急度は?",+ "identical_texts": ("中程度", "依頼"),+ "levels": ("急がない", "早めに", "業務が止まっている"),+ },+ "ko": {+ "states": (+ "시간 되실 때 저희 계정의 청구 주소를 하버가 12번지로 변경해 주실 수 있을까요? 전혀 급하지 않습니다.",+ "3월 청구서에 같은 좌석 요금이 두 번 청구되어 있습니다. 이번 주 안에 확인해 주실 수 있을까요?",+ "오늘 아침부터 CSV 내보내기가 계속 실패하고 있어서, 해결될 때까지 재무팀이 월말 마감을 할 수 없습니다.",+ "어제 온보딩 통화 감사했습니다. 저희 쪽에서 따로 필요한 것은 없습니다.",+ "다음 분기에 라이선스를 10개 더 추가하려고 합니다. 편하실 때 가격 정보를 보내 주실 수 있나요?",+ "회사 전체 사용자가 로그인할 수 없습니다. 아무도 대시보드에 접속하지 못하고 고객들이 기다리고 있습니다.",+ "모바일 앱 보고서에서 가끔 시간대가 잘못 표시됩니다. 불편하지만 우회할 수는 있습니다.",+ "API 키가 실수로 취소되어 지금 모든 운영 요청이 401 오류를 반환하고 있습니다.",+ "내일 아침에 급여 처리가 실행되는데, 연동 기능이 아직도 스키마 오류로 직원 파일을 거부하고 있습니다.",+ "나중에 관리 화면에 다크 모드를 추가해 주실 수 있을까요? 저희 중 몇 명이 원하고 있습니다.",+ ),+ "instructions": "이 요청은 얼마나 긴급한가요?",+ "identical_texts": ("보통", "요청"),+ "levels": ("급하지 않음", "빨리", "업무가 멈춤"),+ },+ "hi": {+ "states": (+ "जब भी आपके पास समय हो, क्या आप हमारे खाते का बिलिंग पता 12 हार्बर स्ट्रीट कर सकते हैं? कोई जल्दी नहीं है।",+ "मार्च के इनवॉइस में एक ही सीट के लिए दो बार शुल्क लगा है। क्या आप इस हफ़्ते इसे देख सकते हैं?",+ "आज सुबह से CSV एक्सपोर्ट विफल हो रहा है और जब तक यह ठीक नहीं होता, हमारी वित्त टीम महीने का हिसाब बंद नहीं कर सकती।",+ "कल की ऑनबोर्डिंग कॉल के लिए धन्यवाद। आपकी ओर से कुछ भी करने की ज़रूरत नहीं है।",+ "हम अगली तिमाही में दस और लाइसेंस जोड़ने के बारे में सोच रहे हैं। क्या कोई सुविधानुसार कीमतें भेज सकता है?",+ "हमारी कंपनी के सभी उपयोगकर्ताओं के लिए लॉगिन बंद है। कोई भी डैशबोर्ड तक नहीं पहुँच पा रहा और हमारे ग्राहक इंतज़ार कर रहे हैं।",+ "मोबाइल ऐप की रिपोर्ट में कभी-कभी गलत टाइम ज़ोन दिखता है। यह परेशान करता है, पर हम इससे काम चला सकते हैं।",+ "हमारी API कुंजी गलती से रद्द हो गई है और अभी हर प्रोडक्शन अनुरोध 401 त्रुटि दे रहा है।",+ "कल सुबह पेरोल चलना है और इंटीग्रेशन अब भी स्कीमा त्रुटि के कारण हमारी कर्मचारी फ़ाइल को अस्वीकार कर रहा है।",+ "क्या आप कभी एडमिन स्क्रीन में डार्क मोड जोड़ सकते हैं? हम में से कई लोग इसे चाहते हैं।",+ ),+ "instructions": "यह अनुरोध कितना ज़रूरी है?",+ "identical_texts": ("मध्यम", "एक अनुरोध"),+ "levels": ("ज़रूरी नहीं", "जल्द", "काम रुका हुआ है"),+ },+ "tr": {+ "states": (+ "Müsait olduğunuzda hesabımızdaki fatura adresini Liman Sokak No. 12 olarak güncelleyebilir misiniz? Hiç acelesi yok.",+ "Mart faturamızda aynı kullanıcı lisansı için iki kez ücret görünüyor. Bu hafta içinde bakabilir misiniz?",+ "CSV dışa aktarımı bu sabahtan beri başarısız oluyor ve düzelene kadar finans ekibimiz ay sonu kapanışını yapamıyor.",+ "Dünkü tanıtım görüşmesi için teşekkür etmek istedim. Sizden bir şey gerekmiyor.",+ "Gelecek çeyrekte on lisans daha eklemeyi düşünüyoruz. Uygun olduğunuzda fiyat bilgisi gönderebilir misiniz?",+ "Şirketimizdeki tüm kullanıcılar için giriş çalışmıyor. Kimse panele ulaşamıyor ve müşterilerimiz bekliyor.",+ "Mobil uygulama raporlarda bazen yanlış saat dilimi gösteriyor. Can sıkıcı ama idare edebiliyoruz.",+ "API anahtarımız yanlışlıkla iptal edildi ve şu anda tüm üretim istekleri 401 hatası döndürüyor.",+ "Bordro yarın sabah çalışacak ve entegrasyon çalışan dosyamızı hâlâ şema hatasıyla reddediyor.",+ "Bir ara yönetim ekranlarına karanlık mod ekleyebilir misiniz? Birkaçımız bunu istiyor.",+ ),+ "instructions": "Bu talep ne kadar acil?",+ "identical_texts": ("orta", "bir talep"),+ "levels": ("Acil değil", "Yakında", "İş durdu"),+ },+}+# Gates. Set from the shipped checkpoints' leave-one-out ranges (README.md, "Thresholds").SLOT0_MIN = -0.20FIRST_SLOT_MIN = 0.15@@ -92,15 +179,16 @@ PARITY_TOL = 1e-3ScoreFn = Callable[[str, Sequence[Dict[str, Any]]], List[Any]]-def identical_question(text: str, k: int) -> Dict[str, Any]:- return {"type": "score", "instructions": INSTRUCTIONS, "criteria": [text] * k}+def identical_question(text: str, k: int, instructions: str = INSTRUCTIONS) -> Dict[str, Any]:+ return {"type": "score", "instructions": instructions, "criteria": [text] * k}-def permuted_questions() -> List[Tuple[Tuple[int, ...], Dict[str, Any]]]:- """Every order of LEVELS, as (order, question); order[slot] indexes LEVELS."""- return [(order, {"type": "score", "instructions": INSTRUCTIONS,- "criteria": [LEVELS[i] for i in order]})- for order in itertools.permutations(range(len(LEVELS)))]+def permuted_questions(levels: Sequence[str] = LEVELS,+ instructions: str = INSTRUCTIONS) -> List[Tuple[Tuple[int, ...], Dict[str, Any]]]:+ """Every order of `levels`, as (order, question); order[slot] indexes `levels`."""+ return [(order, {"type": "score", "instructions": instructions,+ "criteria": [levels[i] for i in order]})+ for order in itertools.permutations(range(len(levels)))]def mean(xs: Sequence[float]) -> float:@@ -119,9 +207,12 @@ def passes(value: float, threshold: float) -> bool:return value >= threshold-def check_score_slot0_identical(score_fn: ScoreFn, states: Sequence[str] = STATES) -> Dict[str, Any]:- configs = [(text, k) for text in IDENTICAL_TEXTS for k in IDENTICAL_KS]- questions = [identical_question(text, k) for text, k in configs]+def check_score_slot0_identical(score_fn: ScoreFn, states: Optional[Sequence[str]] = None,+ lang: str = "en") -> Dict[str, Any]:+ spec = LANGUAGES[lang]+ states = spec["states"] if states is None else states+ configs = [(text, k) for text in spec["identical_texts"] for k in IDENTICAL_KS]+ questions = [identical_question(text, k, spec["instructions"]) for text, k in configs]per_state: List[float] = []per_config: Dict[str, List[float]] = {"%s/K=%d" % c: [] for c in configs}per_slot: Dict[str, List[float]] = {"%s/K=%d" % (t, k): [0.0] * k for t, k in configs}@@ -148,11 +239,15 @@ def check_score_slot0_identical(score_fn: ScoreFn, states: Sequence[str] = STATE}-def check_score_first_slot_permuted(score_fn: ScoreFn, states: Sequence[str] = STATES) -> Dict[str, Any]:- design = permuted_questions()+def check_score_first_slot_permuted(score_fn: ScoreFn, states: Optional[Sequence[str]] = None,+ lang: str = "en") -> Dict[str, Any]:+ spec = LANGUAGES[lang]+ states = spec["states"] if states is None else states+ levels = spec["levels"]+ design = permuted_questions(levels, spec["instructions"])questions = [q for _order, q in design]- by_slot = [0] * len(LEVELS)- by_label = {label: 0 for label in LEVELS}+ by_slot = [0] * len(levels)+ by_label = {label: 0 for label in levels}per_state: List[float] = []order_invariant = 0for state in states:@@ -160,8 +255,8 @@ def check_score_first_slot_permuted(score_fn: ScoreFn, states: Sequence[str] = Sfor (order, _q), z in zip(design, score_fn(state, questions)):slot = max(range(len(z)), key=lambda i: float(z[i]))by_slot[slot] += 1- by_label[LEVELS[order[slot]]] += 1- picked.append((slot, LEVELS[order[slot]]))+ by_label[levels[order[slot]]] += 1+ picked.append((slot, levels[order[slot]]))per_state.append(sum(1 for slot, _ in picked if slot == 0) / len(design))order_invariant += len({label for _, label in picked}) == 1metric = mean(per_state)@@ -186,12 +281,39 @@ CHECKS: Dict[str, Callable[..., Dict[str, Any]]] = {def run_checks(score_fn: ScoreFn, names: Optional[Sequence[str]] = None,- states: Sequence[str] = STATES) -> Dict[str, Any]:+ states: Optional[Sequence[str]] = None, lang: str = "en") -> Dict[str, Any]:names = list(names or CHECKS)- results = {name: CHECKS[name](score_fn, states) for name in names}+ results = {name: CHECKS[name](score_fn, states, lang) for name in names}return {"checks": results, "passed": all(r["passed"] for r in results.values())}+def parse_langs(values: Optional[Sequence[str]]) -> List[str]:+ """`--lang ja --lang ko` or `--lang ja,ko`, in order, without repeats. None means English only.++ Raises ValueError naming any code that has no fixed states here.+ """+ if not values:+ return ["en"]+ langs = list(dict.fromkeys(x.strip().lower() for v in values for x in v.split(",") if x.strip()))+ unknown = [x for x in langs if x not in LANGUAGES]+ if unknown or not langs:+ raise ValueError("unknown language %s; fixed states exist for: %s"+ % (", ".join(unknown) or "(none given)", ", ".join(LANGUAGES)))+ return langs+++def routes(states: Sequence[str]) -> Dict[str, int]:+ """Which checkpoint `Router` would pick for each state, counted. Routing only, no weights."""+ from laya.router import Router++ router = Router()+ counts: Dict[str, int] = {}+ for state in states:+ model = router.route(state)["model"]+ counts[model] = counts.get(model, 0) + 1+ return counts++def exit_code(report: Dict[str, Any], parity: float) -> int:if not parity <= PARITY_TOL:return 2@@ -205,15 +327,18 @@ def agent_score_fn(agent) -> ScoreFn:return score-def parity(agent, states: Sequence[str] = STATES) -> float:+def parity(agent, states: Optional[Sequence[str]] = None, lang: str = "en") -> float:"""Max |p| difference between this harness and Agent.system_one.system_one rounds probabilities to 4 decimals, so agreement shows up as <= 5e-5."""from laya.common import QTYPES- questions = {"identical": identical_question(IDENTICAL_TEXTS[0], 3),- "levels": {"type": "score", "instructions": INSTRUCTIONS, "criteria": list(LEVELS)}}+ spec = LANGUAGES[lang]+ states = spec["states"] if states is None else states+ questions = {"identical": identical_question(spec["identical_texts"][0], 3, spec["instructions"]),+ "levels": {"type": "score", "instructions": spec["instructions"],+ "criteria": list(spec["levels"])}}score = agent_score_fn(agent)worst = 0.0for state in states:@@ -238,6 +363,9 @@ def main(argv: Optional[Sequence[str]] = None) -> int:parser.add_argument("--checks", default=",".join(CHECKS),help="comma-separated subset of: %s" % ", ".join(CHECKS))parser.add_argument("--out", default=None, help="write the JSON report here")+ parser.add_argument("--lang", action="append", default=None,+ help="fixed-state language: %s; repeat or comma-separate (--lang ja,ko). "+ "Without it the run is English and the report is unchanged" % ", ".join(LANGUAGES))args = parser.parse_args(argv)names = [x.strip() for x in args.checks.split(",") if x.strip()]@@ -245,36 +373,66 @@ def main(argv: Optional[Sequence[str]] = None) -> int:if unknown:print("unknown checks: %s" % ", ".join(unknown), file=sys.stderr)return 2+ try:+ langs = parse_langs(args.lang)+ except ValueError as exc:+ print(str(exc), file=sys.stderr)+ return 2import layastarted = time.time()agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)agent.model.eval()- worst = parity(agent)- report = run_checks(agent_score_fn(agent), names)- payload = {- "config": {- "model": args.model,- "subfolder": args.subfolder,- "device": str(agent.device),- "states": len(STATES),- "instructions": INSTRUCTIONS,- "identical_texts": list(IDENTICAL_TEXTS),- "identical_ks": list(IDENTICAL_KS),- "levels": list(LEVELS),- "laya_version": getattr(laya, "__version__", "unknown"),- },- "parity_max_abs_diff": worst,- **report,- "seconds": round(time.time() - started, 1),- }+ if args.lang is None:+ worst = parity(agent)+ report = run_checks(agent_score_fn(agent), names)+ payload = {+ "config": {+ "model": args.model,+ "subfolder": args.subfolder,+ "device": str(agent.device),+ "states": len(STATES),+ "instructions": INSTRUCTIONS,+ "identical_texts": list(IDENTICAL_TEXTS),+ "identical_ks": list(IDENTICAL_KS),+ "levels": list(LEVELS),+ "laya_version": getattr(laya, "__version__", "unknown"),+ },+ "parity_max_abs_diff": worst,+ **report,+ "seconds": round(time.time() - started, 1),+ }+ print_report(report, worst)+ else:+ by_lang: Dict[str, Any] = {}+ for lang in langs:+ spec = LANGUAGES[lang]+ routed = routes(spec["states"])+ lang_worst = parity(agent, lang=lang)+ lang_report = run_checks(agent_score_fn(agent), names, lang=lang)+ by_lang[lang] = {+ "config": {"states": len(spec["states"]), "instructions": spec["instructions"],+ "identical_texts": list(spec["identical_texts"]),+ "identical_ks": list(IDENTICAL_KS), "levels": list(spec["levels"]),+ "router_picks": routed},+ "parity_max_abs_diff": lang_worst,+ **lang_report,+ }+ print("== %s: Router picks %s" % (lang, ", ".join(+ "%s for %d/%d states" % (m, n, len(spec["states"])) for m, n in sorted(routed.items()))))+ print_report(lang_report, lang_worst)+ worst = max(r["parity_max_abs_diff"] for r in by_lang.values())+ report = {"passed": all(r["passed"] for r in by_lang.values())}+ payload = {+ "config": {"model": args.model, "subfolder": args.subfolder, "device": str(agent.device),+ "langs": langs, "laya_version": getattr(laya, "__version__", "unknown")},+ "languages": by_lang,+ "parity_max_abs_diff": worst,+ **report,+ "seconds": round(time.time() - started, 1),+ }- for name, r in report["checks"].items():- print(" %-26s %8.4f >= %5.2f %s (leave-one-out %.4f .. %.4f)"- % (name, r["metric"], r["threshold"], "PASS" if r["passed"] else "FAIL",- r["leave_one_out"][0], r["leave_one_out"][1]))- print(" parity vs Agent.system_one: max |dp| %.2e (tolerance %.0e)" % (worst, PARITY_TOL))code = exit_code(report, worst)print(" verdict: %s" % {0: "PASS", 1: "FAIL", 2: "HARNESS MISMATCH"}[code])@@ -285,5 +443,13 @@ def main(argv: Optional[Sequence[str]] = None) -> int:return code+def print_report(report: Dict[str, Any], worst: float) -> None:+ for name, r in report["checks"].items():+ print(" %-26s %8.4f >= %5.2f %s (leave-one-out %.4f .. %.4f)"+ % (name, r["metric"], r["threshold"], "PASS" if r["passed"] else "FAIL",+ r["leave_one_out"][0], r["leave_one_out"][1]))+ print(" parity vs Agent.system_one: max |dp| %.2e (tolerance %.0e)" % (worst, PARITY_TOL))++if __name__ == "__main__": # pragma: no coversys.exit(main())diff --git a/research/eval/test_presentation_checks.py b/research/eval/test_presentation_checks.pyindex d376da55..de5dc9db 100644--- a/research/eval/test_presentation_checks.py+++ b/research/eval/test_presentation_checks.py@@ -13,9 +13,9 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspafrom laya.common import render_options # noqa: E402from research.eval.presentation_checks import ( # noqa: E402- CHECKS, FIRST_SLOT_MIN, IDENTICAL_KS, IDENTICAL_TEXTS, INSTRUCTIONS, LEVELS, PARITY_TOL,+ CHECKS, FIRST_SLOT_MIN, IDENTICAL_KS, IDENTICAL_TEXTS, INSTRUCTIONS, LANGUAGES, LEVELS, PARITY_TOL,SLOT0_MIN, STATES, check_score_first_slot_permuted, check_score_slot0_identical, exit_code,- identical_question, leave_one_out, passes, permuted_questions, run_checks,+ identical_question, leave_one_out, main, parse_langs, passes, permuted_questions, routes, run_checks,)PASS, FAIL = [], []@@ -162,6 +162,83 @@ check("exit/parity outranks the verdict", exit_code({"passed": True}, 2e-3), 2)check("exit/nan parity is a mismatch", exit_code({"passed": True}, float("nan")), 2)+# ------------------------------------------------------- fixed states in other languages+check("lang/registered", list(LANGUAGES), ["en", "ja", "ko", "hi", "tr"])+check_true("lang/en is the English constants",+ LANGUAGES["en"]["states"] is STATES and LANGUAGES["en"]["levels"] is LEVELS+ and LANGUAGES["en"]["instructions"] == INSTRUCTIONS+ and LANGUAGES["en"]["identical_texts"] is IDENTICAL_TEXTS)+for lang, spec in LANGUAGES.items():+ if lang == "en":+ continue+ check("lang/%s ten states" % lang, len(spec["states"]), 10)+ check("lang/%s no duplicates" % lang, len(set(spec["states"])), 10)+ check_true("lang/%s no empty state" % lang, all(s.strip() for s in spec["states"]))+ check_true("lang/%s not the English states" % lang, not set(spec["states"]) & set(STATES))+ check_true("lang/%s written in the language, not ASCII" % lang,+ not any(s.isascii() for s in spec["states"]))+ check("lang/%s three distinct levels" % lang, len(set(spec["levels"])), 3)+ check("lang/%s two distinct identical texts" % lang, len(set(spec["identical_texts"])), 2)+ check_true("lang/%s instructions" % lang, bool(spec["instructions"].strip()))+ for text in spec["identical_texts"]:+ q = identical_question(text, 3, spec["instructions"])+ rendered = render_options({"t": q["type"], "ins": q["instructions"], "crit": q["criteria"]})+ check("lang/%s identical %s renders level-indexed" % (lang, text),+ rendered, ["level %d: %s" % (i, text) for i in range(3)])+ design = permuted_questions(spec["levels"], spec["instructions"])+ check_true("lang/%s every level in every slot twice" % lang,+ all(sum(1 for order, _ in design if order[slot] == li) == 2+ for slot in range(3) for li in range(3)))+ check_true("lang/%s questions carry the language's instructions" % lang,+ all(q["instructions"] == spec["instructions"] for _, q in design))++ lang_calls = []++ def lang_counting(state, questions, _calls=lang_calls, _states=spec["states"]):+ assert state in _states+ _calls.append(len(questions))+ return FLAT(state, questions)++ run_checks(lang_counting, lang=lang)+ check("lang/%s one forward per state per check, on its own states" % lang, len(lang_calls), 20)+ by_label = scripted([0.0] * 5, dict(zip(spec["levels"], (0.0, 1.0, 2.0))))+ r = check_score_first_slot_permuted(by_label, lang=lang)+ check("lang/%s order-invariant model scores exactly 1/3" % lang, r["metric"], round(1 / 3, 4))+ check("lang/%s picks are counted by its own levels" % lang,+ r["picks_by_label"], {spec["levels"][0]: 0, spec["levels"][1]: 0, spec["levels"][2]: 60})+ r = check_score_slot0_identical(NO_SLOT0, lang=lang)+ check("lang/%s suppressed slot 0 gives the English value" % lang,+ r["metric"], round((-5 / 3 - 1.875 - 2.0) / 3, 4))++# Routing only, no weights: the non-English states go to the checkpoint #131 is about.+for lang, spec in LANGUAGES.items():+ check("route/%s" % lang, routes(spec["states"]),+ {"english": 10} if lang == "en" else {"multilingual": 10})+++# --------------------------------------------------- the default run is unchanged+for fn in (FLAT, NO_SLOT0, EARLY, BY_LABEL):+ check_true("default/run_checks equals lang=en and states=STATES",+ run_checks(fn) == run_checks(fn, lang="en") == run_checks(fn, states=STATES))+check("default/slot0 by hand still", check_score_slot0_identical(NO_SLOT0)["metric"],+ round((-5 / 3 - 1.875 - 2.0) / 3, 4))+++# ------------------------------------------------------------------- --lang parsing+check("langs/none means English", parse_langs(None), ["en"])+check("langs/single", parse_langs(["ja"]), ["ja"])+check("langs/comma-separated", parse_langs(["ja,ko"]), ["ja", "ko"])+check("langs/repeated flag, order kept, no repeats", parse_langs(["tr", "ja", "tr,ko"]), ["tr", "ja", "ko"])+check("langs/case and spaces", parse_langs([" JA , Hi "]), ["ja", "hi"])+for bad in (["xx"], ["ja,xx"], [","]):+ try:+ parse_langs(bad)+ FAIL.append("langs/%r should raise" % bad)+ except ValueError as exc:+ check_true("langs/%r raises naming the known codes" % bad, "ja, ko, hi, tr" in str(exc), str(exc))+check("langs/unknown code exits 2 before loading a checkpoint", main(["--lang", "xx"]), 2)++print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))for f in FAIL:print(" FAIL " + f)diff --git a/research/results/presentation_checks_langs.json b/research/results/presentation_checks_langs.jsonnew file mode 100644index 00000000..554564a3--- /dev/null+++ b/research/results/presentation_checks_langs.json@@ -0,0 +1,539 @@+{+ "about": "research/eval/presentation_checks.py --lang ja,ko,hi,tr on laya-multilingual (#602); same gates as the English run",+ "revision": "convaiinnovations/laya@55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 (subfolder multilingual)",+ "environment": {+ "device": "cpu",+ "dtype": "float32",+ "torch": "2.14.0+cpu",+ "transformers": "5.17.0",+ "python": "3.11.9",+ "laya": "0.3.21"+ },+ "multilingual": {+ "config": {+ "model": "convaiinnovations/laya",+ "subfolder": "multilingual",+ "device": "cpu",+ "langs": [+ "ja",+ "ko",+ "hi",+ "tr"+ ],+ "laya_version": "0.3.21"+ },+ "languages": {+ "ja": {+ "config": {+ "states": 10,+ "instructions": "この依頼の緊急度は?",+ "identical_texts": [+ "中程度",+ "依頼"+ ],+ "identical_ks": [+ 3,+ 4,+ 5+ ],+ "levels": [+ "急がない",+ "早めに",+ "業務が止まっている"+ ],+ "router_picks": {+ "multilingual": 10+ }+ },+ "parity_max_abs_diff": 4.6754520251379716e-05,+ "checks": {+ "score_slot0_identical": {+ "metric": -0.0183,+ "threshold": -0.2,+ "passed": true,+ "leave_one_out": [+ -0.1204,+ 0.0313+ ],+ "per_state": [+ 0.1001,+ 0.2712,+ -0.0314,+ 0.9,+ -0.2066,+ -0.3125,+ -0.1266,+ -0.0819,+ -0.2302,+ -0.4654+ ],+ "per_config": {+ "中程度/K=3": -0.4143,+ "中程度/K=4": -0.2208,+ "中程度/K=5": 0.0876,+ "依頼/K=3": 0.0129,+ "依頼/K=4": 0.1993,+ "依頼/K=5": 0.2252+ },+ "per_slot_centred": {+ "中程度/K=3": [+ -0.4143,+ 0.7693,+ -0.3551+ ],+ "中程度/K=4": [+ -0.2208,+ 0.9156,+ -0.0359,+ -0.6589+ ],+ "中程度/K=5": [+ 0.0876,+ 1.1026,+ -0.1029,+ -0.3562,+ -0.7311+ ],+ "依頼/K=3": [+ 0.0129,+ 0.0993,+ -0.1122+ ],+ "依頼/K=4": [+ 0.1993,+ 0.0885,+ -0.1,+ -0.1878+ ],+ "依頼/K=5": [+ 0.2252,+ 0.0781,+ -0.0912,+ -0.132,+ -0.0802+ ]+ }+ },+ "score_first_slot_permuted": {+ "metric": 0.0,+ "threshold": 0.15,+ "passed": false,+ "leave_one_out": [+ 0.0,+ 0.0+ ],+ "decisions": 60,+ "per_state": [+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0+ ],+ "argmax_by_slot": [+ 0,+ 29,+ 31+ ],+ "picks_by_label": {+ "急がない": 12,+ "早めに": 18,+ "業務が止まっている": 30+ },+ "order_invariant_states": 0+ }+ },+ "passed": false+ },+ "ko": {+ "config": {+ "states": 10,+ "instructions": "이 요청은 얼마나 긴급한가요?",+ "identical_texts": [+ "보통",+ "요청"+ ],+ "identical_ks": [+ 3,+ 4,+ 5+ ],+ "levels": [+ "급하지 않음",+ "빨리",+ "업무가 멈춤"+ ],+ "router_picks": {+ "multilingual": 10+ }+ },+ "parity_max_abs_diff": 4.968593182863579e-05,+ "checks": {+ "score_slot0_identical": {+ "metric": -0.5695,+ "threshold": -0.2,+ "passed": false,+ "leave_one_out": [+ -0.6217,+ -0.5057+ ],+ "per_state": [+ -0.522,+ -0.7999,+ -0.5239,+ -0.1021,+ -0.3256,+ -0.6993,+ -1.1435,+ -0.0994,+ -0.5991,+ -0.8804+ ],+ "per_config": {+ "보통/K=3": -0.9965,+ "보통/K=4": -0.753,+ "보통/K=5": -0.5084,+ "요청/K=3": -0.6171,+ "요청/K=4": -0.3946,+ "요청/K=5": -0.1475+ },+ "per_slot_centred": {+ "보통/K=3": [+ -0.9965,+ 0.7452,+ 0.2513+ ],+ "보통/K=4": [+ -0.753,+ 0.7988,+ 0.317,+ -0.3628+ ],+ "보통/K=5": [+ -0.5084,+ 0.8851,+ 0.2085,+ -0.0849,+ -0.5004+ ],+ "요청/K=3": [+ -0.6171,+ 0.7792,+ -0.1621+ ],+ "요청/K=4": [+ -0.3946,+ 0.7719,+ 0.2128,+ -0.5901+ ],+ "요청/K=5": [+ -0.1475,+ 0.8516,+ 0.2394,+ -0.3379,+ -0.6057+ ]+ }+ },+ "score_first_slot_permuted": {+ "metric": 0.05,+ "threshold": 0.15,+ "passed": false,+ "leave_one_out": [+ 0.0185,+ 0.0556+ ],+ "decisions": 60,+ "per_state": [+ 0.3333,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.1667,+ 0.0+ ],+ "argmax_by_slot": [+ 3,+ 34,+ 23+ ],+ "picks_by_label": {+ "급하지 않음": 30,+ "빨리": 12,+ "업무가 멈춤": 18+ },+ "order_invariant_states": 0+ }+ },+ "passed": false+ },+ "hi": {+ "config": {+ "states": 10,+ "instructions": "यह अनुरोध कितना ज़रूरी है?",+ "identical_texts": [+ "मध्यम",+ "एक अनुरोध"+ ],+ "identical_ks": [+ 3,+ 4,+ 5+ ],+ "levels": [+ "ज़रूरी नहीं",+ "जल्द",+ "काम रुका हुआ है"+ ],+ "router_picks": {+ "multilingual": 10+ }+ },+ "parity_max_abs_diff": 4.9100966272169755e-05,+ "checks": {+ "score_slot0_identical": {+ "metric": -0.2252,+ "threshold": -0.2,+ "passed": false,+ "leave_one_out": [+ -0.2998,+ -0.1824+ ],+ "per_state": [+ -0.3324,+ 0.0546,+ -0.3361,+ 0.4463,+ -0.3899,+ -0.6105,+ -0.1479,+ -0.4678,+ -0.4899,+ 0.0215+ ],+ "per_config": {+ "मध्यम/K=3": -0.6239,+ "मध्यम/K=4": -0.2763,+ "मध्यम/K=5": -0.0001,+ "एक अनुरोध/K=3": -0.3694,+ "एक अनुरोध/K=4": -0.092,+ "एक अनुरोध/K=5": 0.0104+ },+ "per_slot_centred": {+ "मध्यम/K=3": [+ -0.6239,+ 0.7648,+ -0.1409+ ],+ "मध्यम/K=4": [+ -0.2763,+ 0.8558,+ -0.1443,+ -0.4351+ ],+ "मध्यम/K=5": [+ -0.0001,+ 0.8603,+ -0.0434,+ -0.3431,+ -0.4737+ ],+ "एक अनुरोध/K=3": [+ -0.3694,+ 0.6559,+ -0.2865+ ],+ "एक अनुरोध/K=4": [+ -0.092,+ 0.6153,+ -0.248,+ -0.2753+ ],+ "एक अनुरोध/K=5": [+ 0.0104,+ 0.7579,+ -0.1775,+ -0.3007,+ -0.2901+ ]+ }+ },+ "score_first_slot_permuted": {+ "metric": 0.0667,+ "threshold": 0.15,+ "passed": false,+ "leave_one_out": [+ 0.037,+ 0.0741+ ],+ "decisions": 60,+ "per_state": [+ 0.0,+ 0.0,+ 0.1667,+ 0.0,+ 0.0,+ 0.3333,+ 0.0,+ 0.1667,+ 0.0,+ 0.0+ ],+ "argmax_by_slot": [+ 4,+ 28,+ 28+ ],+ "picks_by_label": {+ "ज़रूरी नहीं": 16,+ "जल्द": 10,+ "काम रुका हुआ है": 34+ },+ "order_invariant_states": 1+ }+ },+ "passed": false+ },+ "tr": {+ "config": {+ "states": 10,+ "instructions": "Bu talep ne kadar acil?",+ "identical_texts": [+ "orta",+ "bir talep"+ ],+ "identical_ks": [+ 3,+ 4,+ 5+ ],+ "levels": [+ "Acil değil",+ "Yakında",+ "İş durdu"+ ],+ "router_picks": {+ "multilingual": 10+ }+ },+ "parity_max_abs_diff": 4.95214576753944e-05,+ "checks": {+ "score_slot0_identical": {+ "metric": 0.1386,+ "threshold": -0.2,+ "passed": true,+ "leave_one_out": [+ 0.0684,+ 0.1739+ ],+ "per_state": [+ 0.3692,+ 0.0143,+ -0.048,+ 0.3167,+ -0.1295,+ -0.1789,+ 0.0882,+ -0.1014,+ 0.7705,+ 0.2849+ ],+ "per_config": {+ "orta/K=3": -0.0454,+ "orta/K=4": 0.1406,+ "orta/K=5": 0.422,+ "bir talep/K=3": -0.1856,+ "bir talep/K=4": 0.142,+ "bir talep/K=5": 0.358+ },+ "per_slot_centred": {+ "orta/K=3": [+ -0.0454,+ 0.5798,+ -0.5344+ ],+ "orta/K=4": [+ 0.1406,+ 0.8411,+ -0.1579,+ -0.8238+ ],+ "orta/K=5": [+ 0.422,+ 0.8838,+ -0.0243,+ -0.59,+ -0.6915+ ],+ "bir talep/K=3": [+ -0.1856,+ 0.4061,+ -0.2205+ ],+ "bir talep/K=4": [+ 0.142,+ 0.6643,+ -0.2369,+ -0.5694+ ],+ "bir talep/K=5": [+ 0.358,+ 0.7853,+ -0.1461,+ -0.4808,+ -0.5164+ ]+ }+ },+ "score_first_slot_permuted": {+ "metric": 0.0333,+ "threshold": 0.15,+ "passed": false,+ "leave_one_out": [+ 0.0185,+ 0.037+ ],+ "decisions": 60,+ "per_state": [+ 0.1667,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.0,+ 0.1667,+ 0.0+ ],+ "argmax_by_slot": [+ 2,+ 28,+ 30+ ],+ "picks_by_label": {+ "Acil değil": 15,+ "Yakında": 12,+ "İş durdu": 33+ },+ "order_invariant_states": 0+ }+ },+ "passed": false+ }+ },+ "parity_max_abs_diff": 4.968593182863579e-05,+ "passed": false,+ "seconds": 26.1+ }+}