Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(research): non-English fixed states for presentation_checks (#602, part b)

hiroki-abe-5817ee586ecd02
diff --git a/research/eval/README.md b/research/eval/README.md
index 3ba5f01a..f73f2c06 100644
--- a/research/eval/README.md
+++ b/research/eval/README.md
@@ -285,20 +285,56 @@ conditions were fixed before the 10-state run:
The tightest margin is the english first-slot rate, at 0.054 against the 0.05 rule.
An order-invariant checkpoint sits at exactly 0.333 on that check.
+### Other languages
+
+`--lang` runs both checks on fixed states in Japanese, Korean, Hindi or Turkish (#602).
+Each set translates the ten English states one for one, with the level texts in the
+same language. Every state routes to `multilingual` under `Router`: Japanese, Korean
+and Hindi by script, Turkish by its non-English letters. The run prints the checkpoint
+`Router` picks for each language. Korean, Hindi and Turkish are among the languages
+whose per-language MASSIVE gains the multilingual model card lists, and they cover
+three routing paths (Hangul, Devanagari, Latin with diacritics). Japanese is where
+#131 was found. The gates are the English ones. Without `--lang` the run and its report are
+unchanged.
+
+```bash
+python research/eval/presentation_checks.py --model convaiinnovations/laya \
+ --subfolder multilingual --lang ja,ko,hi,tr --out langs.json
+```
+
+CPU, fp32, `convaiinnovations/laya@55cf4c4` (subfolder `multilingual`), laya 0.3.21.
+Full output: `research/results/presentation_checks_langs.json`. The same run without
+`--lang` reproduces the English row above exactly.
+
+| language | `score_slot0_identical` (leave-one-out) | `score_first_slot_permuted` (leave-one-out) | parity max \|Δp\| |
+|---|---|---|---|
+| `ja` | −0.018 (−0.120 .. +0.031) PASS | **0.000** (0.000 .. 0.000) FAIL | 4.68e-5 |
+| `ko` | **−0.570** (−0.622 .. −0.506) FAIL | **0.050** (0.019 .. 0.056) FAIL | 4.97e-5 |
+| `hi` | **−0.225** (−0.300 .. −0.182) FAIL | **0.067** (0.037 .. 0.074) FAIL | 4.91e-5 |
+| `tr` | +0.139 (+0.068 .. +0.174) PASS | **0.033** (0.019 .. 0.037) FAIL | 4.95e-5 |
+
+The first-slot check fails in all four languages, as it does in English. The
+identical-option control passes in Japanese and Turkish, so on those states it would
+not catch the prior on its own. That is the case for running both checks in every
+language.
+
### Tests
`research/eval/test_presentation_checks.py` runs offline, with scripted logits in
place of a checkpoint:
```bash
-python research/eval/test_presentation_checks.py # 69 passed, 0 failed
+python research/eval/test_presentation_checks.py # 154 passed, 0 failed
```
It pins the fixed inputs and both gates. It checks that the identical-option
questions render as `level i: <same text>`, and that every level sits in every slot
exactly twice. It also checks the metric arithmetic by hand, the leave-one-out
bounds, the one-sided gates, and the exit codes. A scripted slot-0 hole fails both
-checks, and an order-invariant model scores exactly 1/3.
+checks, and an order-invariant model scores exactly 1/3. For each language it
+pins ten distinct states, three levels in every slot twice, and routing to
+`multilingual`. It also checks the `--lang` parsing. The default run gives the same
+report as `lang="en"`.
### Limits
@@ -312,8 +348,10 @@ checks, and an order-invariant model scores exactly 1/3.
(multilingual: −0.75 / −0.52 / −0.25 and −0.58 / −0.47 / −0.37.)
* Passing is not accuracy. A checkpoint can clear both gates and still rank urgency
badly; this checks one known failure, not `score` quality.
-* English only, `score` only, 10 states. The states are short support messages, so a
- checkpoint's behaviour on long inputs or other languages is not covered here.
+* `score` only, 10 states per language. The states are short support messages, so a
+ checkpoint's behaviour on long inputs is not covered here. The Korean, Hindi and
+ Turkish states and levels were written by a non-native speaker; corrections from
+ native speakers are welcome.
* Thresholds were set on CPU fp32. On CUDA, `Agent` runs the forward pass under
reduced-precision autocast and `score_cases` does not. The parity check reports that
difference instead of hiding it.
diff --git a/research/eval/presentation_checks.py b/research/eval/presentation_checks.py
index 5afabca4..d74e3762 100644
--- a/research/eval/presentation_checks.py
+++ b/research/eval/presentation_checks.py
@@ -29,11 +29,21 @@ Both metrics read raw marker logits (before temperature) through
uses. ``parity`` checks that path against ``Agent.system_one`` before anything is
reported.
+Languages
+---------
+``--lang`` runs the same two checks on fixed states in Japanese, Korean, Hindi or
+Turkish (#602): translations of the English states, with the level texts in the same
+language. Every one of them routes to ``multilingual`` under ``Router``, so they test
+the checkpoint #131 is about on the scripts it serves. The gates are the English ones.
+Without ``--lang`` the run and its report are exactly what they were.
+
Usage
-----
python research/eval/presentation_checks.py --model convaiinnovations/laya
python research/eval/presentation_checks.py --model convaiinnovations/laya \\
--subfolder multilingual --out multilingual.json
+ python research/eval/presentation_checks.py --model convaiinnovations/laya \\
+ --subfolder multilingual --lang ja,ko,hi,tr --out multilingual-langs.json
Exit status: 0 every check passed, 1 a check failed, 2 the harness could not be
trusted (parity with ``Agent.system_one`` above ``PARITY_TOL``).
@@ -84,6 +94,83 @@ IDENTICAL_TEXTS = ("moderate", "a request")
IDENTICAL_KS = (3, 4, 5)
LEVELS = ("Not urgent", "Soon", "Work is blocked")
+# Fixed states in other languages, for the multilingual checkpoint (#602). Each tuple follows
+# STATES item for item, so the urgency spread is the same in every language; every state routes
+# to `multilingual` under `Router` (tested offline). The level texts mirror LEVELS. They were
+# written for this check by a non-native speaker, except Japanese; corrections are welcome.
+LANGUAGES: Dict[str, Dict[str, Any]] = {
+ "en": {"states": STATES, "instructions": INSTRUCTIONS, "identical_texts": IDENTICAL_TEXTS,
+ "levels": LEVELS},
+ "ja": {
+ "states": (
+ "お手すきの際に、アカウントの請求先住所をハーバー通り12番地に変更していただけますでしょうか。急ぎではございません。",
+ "3月分の請求書に、同じシートの料金が二重に計上されております。今週中にご確認いただけますでしょうか。",
+ "今朝からCSVの書き出しが失敗しており、復旧するまで経理部門が月次の締めを行えない状況です。",
+ "昨日のオンボーディングのお打ち合わせ、ありがとうございました。こちらから特にお願いすることはございません。",
+ "来四半期にライセンスを10件追加することを検討しております。ご都合のよいときにお見積もりをお送りいただけますか。",
+ "弊社の全ユーザーがログインできない状態です。誰もダッシュボードにアクセスできず、お客様をお待たせしております。",
+ "モバイルアプリのレポートで、ときどきタイムゾーンが誤って表示されます。不便ではありますが、回避はできております。",
+ "APIキーが誤って無効化され、現在すべての本番リクエストが401エラーを返しております。",
+ "明朝に給与計算の実行を控えておりますが、連携機能が依然としてスキーマエラーで従業員ファイルを受け付けません。",
+ "いずれ管理画面にダークモードを追加していただくことは可能でしょうか。社内でも何名か希望しております。",
+ ),
+ "instructions": "この依頼の緊急度は?",
+ "identical_texts": ("中程度", "依頼"),
+ "levels": ("急がない", "早めに", "業務が止まっている"),
+ },
+ "ko": {
+ "states": (
+ "시간 되실 때 저희 계정의 청구 주소를 하버가 12번지로 변경해 주실 수 있을까요? 전혀 급하지 않습니다.",
+ "3월 청구서에 같은 좌석 요금이 두 번 청구되어 있습니다. 이번 주 안에 확인해 주실 수 있을까요?",
+ "오늘 아침부터 CSV 내보내기가 계속 실패하고 있어서, 해결될 때까지 재무팀이 월말 마감을 할 수 없습니다.",
+ "어제 온보딩 통화 감사했습니다. 저희 쪽에서 따로 필요한 것은 없습니다.",
+ "다음 분기에 라이선스를 10개 더 추가하려고 합니다. 편하실 때 가격 정보를 보내 주실 수 있나요?",
+ "회사 전체 사용자가 로그인할 수 없습니다. 아무도 대시보드에 접속하지 못하고 고객들이 기다리고 있습니다.",
+ "모바일 앱 보고서에서 가끔 시간대가 잘못 표시됩니다. 불편하지만 우회할 수는 있습니다.",
+ "API 키가 실수로 취소되어 지금 모든 운영 요청이 401 오류를 반환하고 있습니다.",
+ "내일 아침에 급여 처리가 실행되는데, 연동 기능이 아직도 스키마 오류로 직원 파일을 거부하고 있습니다.",
+ "나중에 관리 화면에 다크 모드를 추가해 주실 수 있을까요? 저희 중 몇 명이 원하고 있습니다.",
+ ),
+ "instructions": "이 요청은 얼마나 긴급한가요?",
+ "identical_texts": ("보통", "요청"),
+ "levels": ("급하지 않음", "빨리", "업무가 멈춤"),
+ },
+ "hi": {
+ "states": (
+ "जब भी आपके पास समय हो, क्या आप हमारे खाते का बिलिंग पता 12 हार्बर स्ट्रीट कर सकते हैं? कोई जल्दी नहीं है।",
+ "मार्च के इनवॉइस में एक ही सीट के लिए दो बार शुल्क लगा है। क्या आप इस हफ़्ते इसे देख सकते हैं?",
+ "आज सुबह से CSV एक्सपोर्ट विफल हो रहा है और जब तक यह ठीक नहीं होता, हमारी वित्त टीम महीने का हिसाब बंद नहीं कर सकती।",
+ "कल की ऑनबोर्डिंग कॉल के लिए धन्यवाद। आपकी ओर से कुछ भी करने की ज़रूरत नहीं है।",
+ "हम अगली तिमाही में दस और लाइसेंस जोड़ने के बारे में सोच रहे हैं। क्या कोई सुविधानुसार कीमतें भेज सकता है?",
+ "हमारी कंपनी के सभी उपयोगकर्ताओं के लिए लॉगिन बंद है। कोई भी डैशबोर्ड तक नहीं पहुँच पा रहा और हमारे ग्राहक इंतज़ार कर रहे हैं।",
+ "मोबाइल ऐप की रिपोर्ट में कभी-कभी गलत टाइम ज़ोन दिखता है। यह परेशान करता है, पर हम इससे काम चला सकते हैं।",
+ "हमारी API कुंजी गलती से रद्द हो गई है और अभी हर प्रोडक्शन अनुरोध 401 त्रुटि दे रहा है।",
+ "कल सुबह पेरोल चलना है और इंटीग्रेशन अब भी स्कीमा त्रुटि के कारण हमारी कर्मचारी फ़ाइल को अस्वीकार कर रहा है।",
+ "क्या आप कभी एडमिन स्क्रीन में डार्क मोड जोड़ सकते हैं? हम में से कई लोग इसे चाहते हैं।",
+ ),
+ "instructions": "यह अनुरोध कितना ज़रूरी है?",
+ "identical_texts": ("मध्यम", "एक अनुरोध"),
+ "levels": ("ज़रूरी नहीं", "जल्द", "काम रुका हुआ है"),
+ },
+ "tr": {
+ "states": (
+ "Müsait olduğunuzda hesabımızdaki fatura adresini Liman Sokak No. 12 olarak güncelleyebilir misiniz? Hiç acelesi yok.",
+ "Mart faturamızda aynı kullanıcı lisansı için iki kez ücret görünüyor. Bu hafta içinde bakabilir misiniz?",
+ "CSV dışa aktarımı bu sabahtan beri başarısız oluyor ve düzelene kadar finans ekibimiz ay sonu kapanışını yapamıyor.",
+ "Dünkü tanıtım görüşmesi için teşekkür etmek istedim. Sizden bir şey gerekmiyor.",
+ "Gelecek çeyrekte on lisans daha eklemeyi düşünüyoruz. Uygun olduğunuzda fiyat bilgisi gönderebilir misiniz?",
+ "Şirketimizdeki tüm kullanıcılar için giriş çalışmıyor. Kimse panele ulaşamıyor ve müşterilerimiz bekliyor.",
+ "Mobil uygulama raporlarda bazen yanlış saat dilimi gösteriyor. Can sıkıcı ama idare edebiliyoruz.",
+ "API anahtarımız yanlışlıkla iptal edildi ve şu anda tüm üretim istekleri 401 hatası döndürüyor.",
+ "Bordro yarın sabah çalışacak ve entegrasyon çalışan dosyamızı hâlâ şema hatasıyla reddediyor.",
+ "Bir ara yönetim ekranlarına karanlık mod ekleyebilir misiniz? Birkaçımız bunu istiyor.",
+ ),
+ "instructions": "Bu talep ne kadar acil?",
+ "identical_texts": ("orta", "bir talep"),
+ "levels": ("Acil değil", "Yakında", "İş durdu"),
+ },
+}
+
# Gates. Set from the shipped checkpoints' leave-one-out ranges (README.md, "Thresholds").
SLOT0_MIN = -0.20
FIRST_SLOT_MIN = 0.15
@@ -92,15 +179,16 @@ PARITY_TOL = 1e-3
ScoreFn = Callable[[str, Sequence[Dict[str, Any]]], List[Any]]
-def identical_question(text: str, k: int) -> Dict[str, Any]:
- return {"type": "score", "instructions": INSTRUCTIONS, "criteria": [text] * k}
+def identical_question(text: str, k: int, instructions: str = INSTRUCTIONS) -> Dict[str, Any]:
+ return {"type": "score", "instructions": instructions, "criteria": [text] * k}
-def permuted_questions() -> List[Tuple[Tuple[int, ...], Dict[str, Any]]]:
- """Every order of LEVELS, as (order, question); order[slot] indexes LEVELS."""
- return [(order, {"type": "score", "instructions": INSTRUCTIONS,
- "criteria": [LEVELS[i] for i in order]})
- for order in itertools.permutations(range(len(LEVELS)))]
+def permuted_questions(levels: Sequence[str] = LEVELS,
+ instructions: str = INSTRUCTIONS) -> List[Tuple[Tuple[int, ...], Dict[str, Any]]]:
+ """Every order of `levels`, as (order, question); order[slot] indexes `levels`."""
+ return [(order, {"type": "score", "instructions": instructions,
+ "criteria": [levels[i] for i in order]})
+ for order in itertools.permutations(range(len(levels)))]
def mean(xs: Sequence[float]) -> float:
@@ -119,9 +207,12 @@ def passes(value: float, threshold: float) -> bool:
return value >= threshold
-def check_score_slot0_identical(score_fn: ScoreFn, states: Sequence[str] = STATES) -> Dict[str, Any]:
- configs = [(text, k) for text in IDENTICAL_TEXTS for k in IDENTICAL_KS]
- questions = [identical_question(text, k) for text, k in configs]
+def check_score_slot0_identical(score_fn: ScoreFn, states: Optional[Sequence[str]] = None,
+ lang: str = "en") -> Dict[str, Any]:
+ spec = LANGUAGES[lang]
+ states = spec["states"] if states is None else states
+ configs = [(text, k) for text in spec["identical_texts"] for k in IDENTICAL_KS]
+ questions = [identical_question(text, k, spec["instructions"]) for text, k in configs]
per_state: List[float] = []
per_config: Dict[str, List[float]] = {"%s/K=%d" % c: [] for c in configs}
per_slot: Dict[str, List[float]] = {"%s/K=%d" % (t, k): [0.0] * k for t, k in configs}
@@ -148,11 +239,15 @@ def check_score_slot0_identical(score_fn: ScoreFn, states: Sequence[str] = STATE
}
-def check_score_first_slot_permuted(score_fn: ScoreFn, states: Sequence[str] = STATES) -> Dict[str, Any]:
- design = permuted_questions()
+def check_score_first_slot_permuted(score_fn: ScoreFn, states: Optional[Sequence[str]] = None,
+ lang: str = "en") -> Dict[str, Any]:
+ spec = LANGUAGES[lang]
+ states = spec["states"] if states is None else states
+ levels = spec["levels"]
+ design = permuted_questions(levels, spec["instructions"])
questions = [q for _order, q in design]
- by_slot = [0] * len(LEVELS)
- by_label = {label: 0 for label in LEVELS}
+ by_slot = [0] * len(levels)
+ by_label = {label: 0 for label in levels}
per_state: List[float] = []
order_invariant = 0
for state in states:
@@ -160,8 +255,8 @@ def check_score_first_slot_permuted(score_fn: ScoreFn, states: Sequence[str] = S
for (order, _q), z in zip(design, score_fn(state, questions)):
slot = max(range(len(z)), key=lambda i: float(z[i]))
by_slot[slot] += 1
- by_label[LEVELS[order[slot]]] += 1
- picked.append((slot, LEVELS[order[slot]]))
+ by_label[levels[order[slot]]] += 1
+ picked.append((slot, levels[order[slot]]))
per_state.append(sum(1 for slot, _ in picked if slot == 0) / len(design))
order_invariant += len({label for _, label in picked}) == 1
metric = mean(per_state)
@@ -186,12 +281,39 @@ CHECKS: Dict[str, Callable[..., Dict[str, Any]]] = {
def run_checks(score_fn: ScoreFn, names: Optional[Sequence[str]] = None,
- states: Sequence[str] = STATES) -> Dict[str, Any]:
+ states: Optional[Sequence[str]] = None, lang: str = "en") -> Dict[str, Any]:
names = list(names or CHECKS)
- results = {name: CHECKS[name](score_fn, states) for name in names}
+ results = {name: CHECKS[name](score_fn, states, lang) for name in names}
return {"checks": results, "passed": all(r["passed"] for r in results.values())}
+def parse_langs(values: Optional[Sequence[str]]) -> List[str]:
+ """`--lang ja --lang ko` or `--lang ja,ko`, in order, without repeats. None means English only.
+
+ Raises ValueError naming any code that has no fixed states here.
+ """
+ if not values:
+ return ["en"]
+ langs = list(dict.fromkeys(x.strip().lower() for v in values for x in v.split(",") if x.strip()))
+ unknown = [x for x in langs if x not in LANGUAGES]
+ if unknown or not langs:
+ raise ValueError("unknown language %s; fixed states exist for: %s"
+ % (", ".join(unknown) or "(none given)", ", ".join(LANGUAGES)))
+ return langs
+
+
+def routes(states: Sequence[str]) -> Dict[str, int]:
+ """Which checkpoint `Router` would pick for each state, counted. Routing only, no weights."""
+ from laya.router import Router
+
+ router = Router()
+ counts: Dict[str, int] = {}
+ for state in states:
+ model = router.route(state)["model"]
+ counts[model] = counts.get(model, 0) + 1
+ return counts
+
+
def exit_code(report: Dict[str, Any], parity: float) -> int:
if not parity <= PARITY_TOL:
return 2
@@ -205,15 +327,18 @@ def agent_score_fn(agent) -> ScoreFn:
return score
-def parity(agent, states: Sequence[str] = STATES) -> float:
+def parity(agent, states: Optional[Sequence[str]] = None, lang: str = "en") -> float:
"""Max |p| difference between this harness and Agent.system_one.
system_one rounds probabilities to 4 decimals, so agreement shows up as <= 5e-5.
"""
from laya.common import QTYPES
- questions = {"identical": identical_question(IDENTICAL_TEXTS[0], 3),
- "levels": {"type": "score", "instructions": INSTRUCTIONS, "criteria": list(LEVELS)}}
+ spec = LANGUAGES[lang]
+ states = spec["states"] if states is None else states
+ questions = {"identical": identical_question(spec["identical_texts"][0], 3, spec["instructions"]),
+ "levels": {"type": "score", "instructions": spec["instructions"],
+ "criteria": list(spec["levels"])}}
score = agent_score_fn(agent)
worst = 0.0
for state in states:
@@ -238,6 +363,9 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
parser.add_argument("--checks", default=",".join(CHECKS),
help="comma-separated subset of: %s" % ", ".join(CHECKS))
parser.add_argument("--out", default=None, help="write the JSON report here")
+ parser.add_argument("--lang", action="append", default=None,
+ help="fixed-state language: %s; repeat or comma-separate (--lang ja,ko). "
+ "Without it the run is English and the report is unchanged" % ", ".join(LANGUAGES))
args = parser.parse_args(argv)
names = [x.strip() for x in args.checks.split(",") if x.strip()]
@@ -245,36 +373,66 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
if unknown:
print("unknown checks: %s" % ", ".join(unknown), file=sys.stderr)
return 2
+ try:
+ langs = parse_langs(args.lang)
+ except ValueError as exc:
+ print(str(exc), file=sys.stderr)
+ return 2
import laya
started = time.time()
agent = laya.load(args.model, device=args.device, subfolder=args.subfolder)
agent.model.eval()
- worst = parity(agent)
- report = run_checks(agent_score_fn(agent), names)
- payload = {
- "config": {
- "model": args.model,
- "subfolder": args.subfolder,
- "device": str(agent.device),
- "states": len(STATES),
- "instructions": INSTRUCTIONS,
- "identical_texts": list(IDENTICAL_TEXTS),
- "identical_ks": list(IDENTICAL_KS),
- "levels": list(LEVELS),
- "laya_version": getattr(laya, "__version__", "unknown"),
- },
- "parity_max_abs_diff": worst,
- **report,
- "seconds": round(time.time() - started, 1),
- }
+ if args.lang is None:
+ worst = parity(agent)
+ report = run_checks(agent_score_fn(agent), names)
+ payload = {
+ "config": {
+ "model": args.model,
+ "subfolder": args.subfolder,
+ "device": str(agent.device),
+ "states": len(STATES),
+ "instructions": INSTRUCTIONS,
+ "identical_texts": list(IDENTICAL_TEXTS),
+ "identical_ks": list(IDENTICAL_KS),
+ "levels": list(LEVELS),
+ "laya_version": getattr(laya, "__version__", "unknown"),
+ },
+ "parity_max_abs_diff": worst,
+ **report,
+ "seconds": round(time.time() - started, 1),
+ }
+ print_report(report, worst)
+ else:
+ by_lang: Dict[str, Any] = {}
+ for lang in langs:
+ spec = LANGUAGES[lang]
+ routed = routes(spec["states"])
+ lang_worst = parity(agent, lang=lang)
+ lang_report = run_checks(agent_score_fn(agent), names, lang=lang)
+ by_lang[lang] = {
+ "config": {"states": len(spec["states"]), "instructions": spec["instructions"],
+ "identical_texts": list(spec["identical_texts"]),
+ "identical_ks": list(IDENTICAL_KS), "levels": list(spec["levels"]),
+ "router_picks": routed},
+ "parity_max_abs_diff": lang_worst,
+ **lang_report,
+ }
+ print("== %s: Router picks %s" % (lang, ", ".join(
+ "%s for %d/%d states" % (m, n, len(spec["states"])) for m, n in sorted(routed.items()))))
+ print_report(lang_report, lang_worst)
+ worst = max(r["parity_max_abs_diff"] for r in by_lang.values())
+ report = {"passed": all(r["passed"] for r in by_lang.values())}
+ payload = {
+ "config": {"model": args.model, "subfolder": args.subfolder, "device": str(agent.device),
+ "langs": langs, "laya_version": getattr(laya, "__version__", "unknown")},
+ "languages": by_lang,
+ "parity_max_abs_diff": worst,
+ **report,
+ "seconds": round(time.time() - started, 1),
+ }
- for name, r in report["checks"].items():
- print(" %-26s %8.4f >= %5.2f %s (leave-one-out %.4f .. %.4f)"
- % (name, r["metric"], r["threshold"], "PASS" if r["passed"] else "FAIL",
- r["leave_one_out"][0], r["leave_one_out"][1]))
- print(" parity vs Agent.system_one: max |dp| %.2e (tolerance %.0e)" % (worst, PARITY_TOL))
code = exit_code(report, worst)
print(" verdict: %s" % {0: "PASS", 1: "FAIL", 2: "HARNESS MISMATCH"}[code])
@@ -285,5 +443,13 @@ def main(argv: Optional[Sequence[str]] = None) -> int:
return code
+def print_report(report: Dict[str, Any], worst: float) -> None:
+ for name, r in report["checks"].items():
+ print(" %-26s %8.4f >= %5.2f %s (leave-one-out %.4f .. %.4f)"
+ % (name, r["metric"], r["threshold"], "PASS" if r["passed"] else "FAIL",
+ r["leave_one_out"][0], r["leave_one_out"][1]))
+ print(" parity vs Agent.system_one: max |dp| %.2e (tolerance %.0e)" % (worst, PARITY_TOL))
+
+
if __name__ == "__main__": # pragma: no cover
sys.exit(main())
diff --git a/research/eval/test_presentation_checks.py b/research/eval/test_presentation_checks.py
index d376da55..de5dc9db 100644
--- a/research/eval/test_presentation_checks.py
+++ b/research/eval/test_presentation_checks.py
@@ -13,9 +13,9 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspa
from laya.common import render_options # noqa: E402
from research.eval.presentation_checks import ( # noqa: E402
- CHECKS, FIRST_SLOT_MIN, IDENTICAL_KS, IDENTICAL_TEXTS, INSTRUCTIONS, LEVELS, PARITY_TOL,
+ CHECKS, FIRST_SLOT_MIN, IDENTICAL_KS, IDENTICAL_TEXTS, INSTRUCTIONS, LANGUAGES, LEVELS, PARITY_TOL,
SLOT0_MIN, STATES, check_score_first_slot_permuted, check_score_slot0_identical, exit_code,
- identical_question, leave_one_out, passes, permuted_questions, run_checks,
+ identical_question, leave_one_out, main, parse_langs, passes, permuted_questions, routes, run_checks,
)
PASS, FAIL = [], []
@@ -162,6 +162,83 @@ check("exit/parity outranks the verdict", exit_code({"passed": True}, 2e-3), 2)
check("exit/nan parity is a mismatch", exit_code({"passed": True}, float("nan")), 2)
+# ------------------------------------------------------- fixed states in other languages
+check("lang/registered", list(LANGUAGES), ["en", "ja", "ko", "hi", "tr"])
+check_true("lang/en is the English constants",
+ LANGUAGES["en"]["states"] is STATES and LANGUAGES["en"]["levels"] is LEVELS
+ and LANGUAGES["en"]["instructions"] == INSTRUCTIONS
+ and LANGUAGES["en"]["identical_texts"] is IDENTICAL_TEXTS)
+for lang, spec in LANGUAGES.items():
+ if lang == "en":
+ continue
+ check("lang/%s ten states" % lang, len(spec["states"]), 10)
+ check("lang/%s no duplicates" % lang, len(set(spec["states"])), 10)
+ check_true("lang/%s no empty state" % lang, all(s.strip() for s in spec["states"]))
+ check_true("lang/%s not the English states" % lang, not set(spec["states"]) & set(STATES))
+ check_true("lang/%s written in the language, not ASCII" % lang,
+ not any(s.isascii() for s in spec["states"]))
+ check("lang/%s three distinct levels" % lang, len(set(spec["levels"])), 3)
+ check("lang/%s two distinct identical texts" % lang, len(set(spec["identical_texts"])), 2)
+ check_true("lang/%s instructions" % lang, bool(spec["instructions"].strip()))
+ for text in spec["identical_texts"]:
+ q = identical_question(text, 3, spec["instructions"])
+ rendered = render_options({"t": q["type"], "ins": q["instructions"], "crit": q["criteria"]})
+ check("lang/%s identical %s renders level-indexed" % (lang, text),
+ rendered, ["level %d: %s" % (i, text) for i in range(3)])
+ design = permuted_questions(spec["levels"], spec["instructions"])
+ check_true("lang/%s every level in every slot twice" % lang,
+ all(sum(1 for order, _ in design if order[slot] == li) == 2
+ for slot in range(3) for li in range(3)))
+ check_true("lang/%s questions carry the language's instructions" % lang,
+ all(q["instructions"] == spec["instructions"] for _, q in design))
+
+ lang_calls = []
+
+ def lang_counting(state, questions, _calls=lang_calls, _states=spec["states"]):
+ assert state in _states
+ _calls.append(len(questions))
+ return FLAT(state, questions)
+
+ run_checks(lang_counting, lang=lang)
+ check("lang/%s one forward per state per check, on its own states" % lang, len(lang_calls), 20)
+ by_label = scripted([0.0] * 5, dict(zip(spec["levels"], (0.0, 1.0, 2.0))))
+ r = check_score_first_slot_permuted(by_label, lang=lang)
+ check("lang/%s order-invariant model scores exactly 1/3" % lang, r["metric"], round(1 / 3, 4))
+ check("lang/%s picks are counted by its own levels" % lang,
+ r["picks_by_label"], {spec["levels"][0]: 0, spec["levels"][1]: 0, spec["levels"][2]: 60})
+ r = check_score_slot0_identical(NO_SLOT0, lang=lang)
+ check("lang/%s suppressed slot 0 gives the English value" % lang,
+ r["metric"], round((-5 / 3 - 1.875 - 2.0) / 3, 4))
+
+# Routing only, no weights: the non-English states go to the checkpoint #131 is about.
+for lang, spec in LANGUAGES.items():
+ check("route/%s" % lang, routes(spec["states"]),
+ {"english": 10} if lang == "en" else {"multilingual": 10})
+
+
+# --------------------------------------------------- the default run is unchanged
+for fn in (FLAT, NO_SLOT0, EARLY, BY_LABEL):
+ check_true("default/run_checks equals lang=en and states=STATES",
+ run_checks(fn) == run_checks(fn, lang="en") == run_checks(fn, states=STATES))
+check("default/slot0 by hand still", check_score_slot0_identical(NO_SLOT0)["metric"],
+ round((-5 / 3 - 1.875 - 2.0) / 3, 4))
+
+
+# ------------------------------------------------------------------- --lang parsing
+check("langs/none means English", parse_langs(None), ["en"])
+check("langs/single", parse_langs(["ja"]), ["ja"])
+check("langs/comma-separated", parse_langs(["ja,ko"]), ["ja", "ko"])
+check("langs/repeated flag, order kept, no repeats", parse_langs(["tr", "ja", "tr,ko"]), ["tr", "ja", "ko"])
+check("langs/case and spaces", parse_langs([" JA , Hi "]), ["ja", "hi"])
+for bad in (["xx"], ["ja,xx"], [","]):
+ try:
+ parse_langs(bad)
+ FAIL.append("langs/%r should raise" % bad)
+ except ValueError as exc:
+ check_true("langs/%r raises naming the known codes" % bad, "ja, ko, hi, tr" in str(exc), str(exc))
+check("langs/unknown code exits 2 before loading a checkpoint", main(["--lang", "xx"]), 2)
+
+
print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL " + f)
diff --git a/research/results/presentation_checks_langs.json b/research/results/presentation_checks_langs.json
new file mode 100644
index 00000000..554564a3
--- /dev/null
+++ b/research/results/presentation_checks_langs.json
@@ -0,0 +1,539 @@
+{
+ "about": "research/eval/presentation_checks.py --lang ja,ko,hi,tr on laya-multilingual (#602); same gates as the English run",
+ "revision": "convaiinnovations/laya@55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 (subfolder multilingual)",
+ "environment": {
+ "device": "cpu",
+ "dtype": "float32",
+ "torch": "2.14.0+cpu",
+ "transformers": "5.17.0",
+ "python": "3.11.9",
+ "laya": "0.3.21"
+ },
+ "multilingual": {
+ "config": {
+ "model": "convaiinnovations/laya",
+ "subfolder": "multilingual",
+ "device": "cpu",
+ "langs": [
+ "ja",
+ "ko",
+ "hi",
+ "tr"
+ ],
+ "laya_version": "0.3.21"
+ },
+ "languages": {
+ "ja": {
+ "config": {
+ "states": 10,
+ "instructions": "この依頼の緊急度は?",
+ "identical_texts": [
+ "中程度",
+ "依頼"
+ ],
+ "identical_ks": [
+ 3,
+ 4,
+ 5
+ ],
+ "levels": [
+ "急がない",
+ "早めに",
+ "業務が止まっている"
+ ],
+ "router_picks": {
+ "multilingual": 10
+ }
+ },
+ "parity_max_abs_diff": 4.6754520251379716e-05,
+ "checks": {
+ "score_slot0_identical": {
+ "metric": -0.0183,
+ "threshold": -0.2,
+ "passed": true,
+ "leave_one_out": [
+ -0.1204,
+ 0.0313
+ ],
+ "per_state": [
+ 0.1001,
+ 0.2712,
+ -0.0314,
+ 0.9,
+ -0.2066,
+ -0.3125,
+ -0.1266,
+ -0.0819,
+ -0.2302,
+ -0.4654
+ ],
+ "per_config": {
+ "中程度/K=3": -0.4143,
+ "中程度/K=4": -0.2208,
+ "中程度/K=5": 0.0876,
+ "依頼/K=3": 0.0129,
+ "依頼/K=4": 0.1993,
+ "依頼/K=5": 0.2252
+ },
+ "per_slot_centred": {
+ "中程度/K=3": [
+ -0.4143,
+ 0.7693,
+ -0.3551
+ ],
+ "中程度/K=4": [
+ -0.2208,
+ 0.9156,
+ -0.0359,
+ -0.6589
+ ],
+ "中程度/K=5": [
+ 0.0876,
+ 1.1026,
+ -0.1029,
+ -0.3562,
+ -0.7311
+ ],
+ "依頼/K=3": [
+ 0.0129,
+ 0.0993,
+ -0.1122
+ ],
+ "依頼/K=4": [
+ 0.1993,
+ 0.0885,
+ -0.1,
+ -0.1878
+ ],
+ "依頼/K=5": [
+ 0.2252,
+ 0.0781,
+ -0.0912,
+ -0.132,
+ -0.0802
+ ]
+ }
+ },
+ "score_first_slot_permuted": {
+ "metric": 0.0,
+ "threshold": 0.15,
+ "passed": false,
+ "leave_one_out": [
+ 0.0,
+ 0.0
+ ],
+ "decisions": 60,
+ "per_state": [
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0
+ ],
+ "argmax_by_slot": [
+ 0,
+ 29,
+ 31
+ ],
+ "picks_by_label": {
+ "急がない": 12,
+ "早めに": 18,
+ "業務が止まっている": 30
+ },
+ "order_invariant_states": 0
+ }
+ },
+ "passed": false
+ },
+ "ko": {
+ "config": {
+ "states": 10,
+ "instructions": "이 요청은 얼마나 긴급한가요?",
+ "identical_texts": [
+ "보통",
+ "요청"
+ ],
+ "identical_ks": [
+ 3,
+ 4,
+ 5
+ ],
+ "levels": [
+ "급하지 않음",
+ "빨리",
+ "업무가 멈춤"
+ ],
+ "router_picks": {
+ "multilingual": 10
+ }
+ },
+ "parity_max_abs_diff": 4.968593182863579e-05,
+ "checks": {
+ "score_slot0_identical": {
+ "metric": -0.5695,
+ "threshold": -0.2,
+ "passed": false,
+ "leave_one_out": [
+ -0.6217,
+ -0.5057
+ ],
+ "per_state": [
+ -0.522,
+ -0.7999,
+ -0.5239,
+ -0.1021,
+ -0.3256,
+ -0.6993,
+ -1.1435,
+ -0.0994,
+ -0.5991,
+ -0.8804
+ ],
+ "per_config": {
+ "보통/K=3": -0.9965,
+ "보통/K=4": -0.753,
+ "보통/K=5": -0.5084,
+ "요청/K=3": -0.6171,
+ "요청/K=4": -0.3946,
+ "요청/K=5": -0.1475
+ },
+ "per_slot_centred": {
+ "보통/K=3": [
+ -0.9965,
+ 0.7452,
+ 0.2513
+ ],
+ "보통/K=4": [
+ -0.753,
+ 0.7988,
+ 0.317,
+ -0.3628
+ ],
+ "보통/K=5": [
+ -0.5084,
+ 0.8851,
+ 0.2085,
+ -0.0849,
+ -0.5004
+ ],
+ "요청/K=3": [
+ -0.6171,
+ 0.7792,
+ -0.1621
+ ],
+ "요청/K=4": [
+ -0.3946,
+ 0.7719,
+ 0.2128,
+ -0.5901
+ ],
+ "요청/K=5": [
+ -0.1475,
+ 0.8516,
+ 0.2394,
+ -0.3379,
+ -0.6057
+ ]
+ }
+ },
+ "score_first_slot_permuted": {
+ "metric": 0.05,
+ "threshold": 0.15,
+ "passed": false,
+ "leave_one_out": [
+ 0.0185,
+ 0.0556
+ ],
+ "decisions": 60,
+ "per_state": [
+ 0.3333,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.1667,
+ 0.0
+ ],
+ "argmax_by_slot": [
+ 3,
+ 34,
+ 23
+ ],
+ "picks_by_label": {
+ "급하지 않음": 30,
+ "빨리": 12,
+ "업무가 멈춤": 18
+ },
+ "order_invariant_states": 0
+ }
+ },
+ "passed": false
+ },
+ "hi": {
+ "config": {
+ "states": 10,
+ "instructions": "यह अनुरोध कितना ज़रूरी है?",
+ "identical_texts": [
+ "मध्यम",
+ "एक अनुरोध"
+ ],
+ "identical_ks": [
+ 3,
+ 4,
+ 5
+ ],
+ "levels": [
+ "ज़रूरी नहीं",
+ "जल्द",
+ "काम रुका हुआ है"
+ ],
+ "router_picks": {
+ "multilingual": 10
+ }
+ },
+ "parity_max_abs_diff": 4.9100966272169755e-05,
+ "checks": {
+ "score_slot0_identical": {
+ "metric": -0.2252,
+ "threshold": -0.2,
+ "passed": false,
+ "leave_one_out": [
+ -0.2998,
+ -0.1824
+ ],
+ "per_state": [
+ -0.3324,
+ 0.0546,
+ -0.3361,
+ 0.4463,
+ -0.3899,
+ -0.6105,
+ -0.1479,
+ -0.4678,
+ -0.4899,
+ 0.0215
+ ],
+ "per_config": {
+ "मध्यम/K=3": -0.6239,
+ "मध्यम/K=4": -0.2763,
+ "मध्यम/K=5": -0.0001,
+ "एक अनुरोध/K=3": -0.3694,
+ "एक अनुरोध/K=4": -0.092,
+ "एक अनुरोध/K=5": 0.0104
+ },
+ "per_slot_centred": {
+ "मध्यम/K=3": [
+ -0.6239,
+ 0.7648,
+ -0.1409
+ ],
+ "मध्यम/K=4": [
+ -0.2763,
+ 0.8558,
+ -0.1443,
+ -0.4351
+ ],
+ "मध्यम/K=5": [
+ -0.0001,
+ 0.8603,
+ -0.0434,
+ -0.3431,
+ -0.4737
+ ],
+ "एक अनुरोध/K=3": [
+ -0.3694,
+ 0.6559,
+ -0.2865
+ ],
+ "एक अनुरोध/K=4": [
+ -0.092,
+ 0.6153,
+ -0.248,
+ -0.2753
+ ],
+ "एक अनुरोध/K=5": [
+ 0.0104,
+ 0.7579,
+ -0.1775,
+ -0.3007,
+ -0.2901
+ ]
+ }
+ },
+ "score_first_slot_permuted": {
+ "metric": 0.0667,
+ "threshold": 0.15,
+ "passed": false,
+ "leave_one_out": [
+ 0.037,
+ 0.0741
+ ],
+ "decisions": 60,
+ "per_state": [
+ 0.0,
+ 0.0,
+ 0.1667,
+ 0.0,
+ 0.0,
+ 0.3333,
+ 0.0,
+ 0.1667,
+ 0.0,
+ 0.0
+ ],
+ "argmax_by_slot": [
+ 4,
+ 28,
+ 28
+ ],
+ "picks_by_label": {
+ "ज़रूरी नहीं": 16,
+ "जल्द": 10,
+ "काम रुका हुआ है": 34
+ },
+ "order_invariant_states": 1
+ }
+ },
+ "passed": false
+ },
+ "tr": {
+ "config": {
+ "states": 10,
+ "instructions": "Bu talep ne kadar acil?",
+ "identical_texts": [
+ "orta",
+ "bir talep"
+ ],
+ "identical_ks": [
+ 3,
+ 4,
+ 5
+ ],
+ "levels": [
+ "Acil değil",
+ "Yakında",
+ "İş durdu"
+ ],
+ "router_picks": {
+ "multilingual": 10
+ }
+ },
+ "parity_max_abs_diff": 4.95214576753944e-05,
+ "checks": {
+ "score_slot0_identical": {
+ "metric": 0.1386,
+ "threshold": -0.2,
+ "passed": true,
+ "leave_one_out": [
+ 0.0684,
+ 0.1739
+ ],
+ "per_state": [
+ 0.3692,
+ 0.0143,
+ -0.048,
+ 0.3167,
+ -0.1295,
+ -0.1789,
+ 0.0882,
+ -0.1014,
+ 0.7705,
+ 0.2849
+ ],
+ "per_config": {
+ "orta/K=3": -0.0454,
+ "orta/K=4": 0.1406,
+ "orta/K=5": 0.422,
+ "bir talep/K=3": -0.1856,
+ "bir talep/K=4": 0.142,
+ "bir talep/K=5": 0.358
+ },
+ "per_slot_centred": {
+ "orta/K=3": [
+ -0.0454,
+ 0.5798,
+ -0.5344
+ ],
+ "orta/K=4": [
+ 0.1406,
+ 0.8411,
+ -0.1579,
+ -0.8238
+ ],
+ "orta/K=5": [
+ 0.422,
+ 0.8838,
+ -0.0243,
+ -0.59,
+ -0.6915
+ ],
+ "bir talep/K=3": [
+ -0.1856,
+ 0.4061,
+ -0.2205
+ ],
+ "bir talep/K=4": [
+ 0.142,
+ 0.6643,
+ -0.2369,
+ -0.5694
+ ],
+ "bir talep/K=5": [
+ 0.358,
+ 0.7853,
+ -0.1461,
+ -0.4808,
+ -0.5164
+ ]
+ }
+ },
+ "score_first_slot_permuted": {
+ "metric": 0.0333,
+ "threshold": 0.15,
+ "passed": false,
+ "leave_one_out": [
+ 0.0185,
+ 0.037
+ ],
+ "decisions": 60,
+ "per_state": [
+ 0.1667,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.0,
+ 0.1667,
+ 0.0
+ ],
+ "argmax_by_slot": [
+ 2,
+ 28,
+ 30
+ ],
+ "picks_by_label": {
+ "Acil değil": 15,
+ "Yakında": 12,
+ "İş durdu": 33
+ },
+ "order_invariant_states": 0
+ }
+ },
+ "passed": false
+ }
+ },
+ "parity_max_abs_diff": 4.968593182863579e-05,
+ "passed": false,
+ "seconds": 26.1
+ }
+}