Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(bench): freeze the Chinese short-command comparison from #218
GaotianJin48a148eadc07
diff --git a/research/README.md b/research/README.mdindex 15783662..861c3f3b 100644--- a/research/README.md+++ b/research/README.md@@ -6,6 +6,7 @@ the numbers quoted in the main README — nothing here is imported by the `laya`## Community diagnostics- [Chinese workplace decisions (Feishu-style)](benchmarks/feishu_zh/README.md) — 64 synthetic scenarios, paired recorded Laya/Jev responses, English/Chinese cards, and a model-free audit. [中文入口](benchmarks/feishu_zh/README.zh-CN.md). Start with `python research/benchmarks/feishu_zh/audit.py`; no downloads or API keys required. This is a contributed historical snapshot, separate from the upstream sweeps below.+- [Chinese short-command routing](benchmarks/zh_short_commands/README.md) — 18 frozen Chinese voice commands, a seven-rung ablation of the documented prompt guidance on both the six-way `choice` path and the four-question `noul` path, and every per-case decision archived. [中文入口](benchmarks/zh_short_commands/README.zh-CN.md). Start with `python research/benchmarks/zh_short_commands/audit.py`; the audit needs no downloads and the archive records which weights produced the numbers.## Scriptsdiff --git a/research/benchmarks/zh_short_commands/README.md b/research/benchmarks/zh_short_commands/README.mdnew file mode 100644index 00000000..2eae1be0--- /dev/null+++ b/research/benchmarks/zh_short_commands/README.md@@ -0,0 +1,151 @@+# Chinese short-command routing++[简体中文](README.zh-CN.md) · [Sibling benchmark: Chinese workplace decisions](../feishu_zh/README.md) · [Issue #218](https://github.com/NandhaKishorM/laya/issues/218)++18 frozen Chinese voice commands for a cleaning robot, six labels, and a seven-rung ablation over the repository's own prompt guidance. One checkpoint, one run, every decision archived — the point is to turn [#218](https://github.com/NandhaKishorM/laya/issues/218)'s one-off report ("adding criteria, a scenario and a structured JSON state made Chinese decisions *worse*") into an artifact anyone can re-derive, and to say which primitive it is actually true of.++**These are 18 hand-written fixtures with a label policy fixed before any model was run. They are not a held-out test set, not an independently annotated corpus, and not an official evaluation.** The numbers below are one checkpoint's decisions on those fixtures.++## Start here: verify offline++From the repository root, using Python 3.10+:++```bash+python research/benchmarks/zh_short_commands/audit.py+python -m unittest discover -s research/benchmarks/zh_short_commands/tests -v+```++No model download, no network. The audit needs nothing but the standard library, and CI runs it in a job where nothing is installed. The test suite drives `run.py --stub`, so it needs numpy — which every environment that can run Laya already has.++The audit re-checks the hashes of the frozen cases and prompts, rebuilds every request (state, instructions, options, gold) instead of trusting the record, and recomputes accuracy, macro-F1, ECE and confidence spread from the per-case records alone. The tests corrupt a copy of the archive in 18 different ways and require the audit to refuse each one; those exist to show the audit can fail.++## The ladder++`choice` cannot exist without criteria — the criteria keys *are* its options — so task A starts one rung higher than task B.++**Task A, `choice`: one six-way question per command.**++| Rung | Correct | Accuracy | macro-F1 | ECE | Mean conf | SD conf |+|---|---:|---:|---:|---:|---:|---:|+| `choice_criteria` (six options only) | 13/18 | 0.7222 | 0.7149 | 0.2659 | 0.8008 | 0.1728 |+| `choice_scenario` (+ scenario sentence) | 14/18 | 0.7778 | 0.7942 | 0.1827 | 0.8257 | 0.1725 |+| `choice_json_state` (+ structured state) | 12/18 | 0.6667 | 0.6817 | 0.2484 | 0.7961 | 0.1238 |++The most frequent label covers 5/18 cases, so the majority-class baseline is 0.2778.++**Task B, `noul`: four independent yes/no questions per command, 72 decisions.**++| Rung | Correct | Accuracy | Mean conf | SD conf |+|---|---:|---:|---:|---:|+| `noul_plain` (no criteria, one question) | 48/72 | 0.6667 | 0.8128 | 0.1786 |+| `noul_criteria` | 33/72 | 0.4583 | 0.8920 | 0.0944 |+| `noul_scenario` | 34/72 | 0.4722 | 0.8944 | 0.0807 |+| `noul_json_state` | 30/72 | 0.4167 | 0.9437 | 0.0418 |++Answering "true" to all 72 questions scores 30/72 = 0.4167, because only 30 of the 72 gold answers are true. That is exactly what `noul_json_state` scored.++## What the ladder shows++The documented guidance does not make the `noul` path read Chinese differently; it makes the four questions stop discriminating. Count how many of the 18 commands each dimension answered "true" for:++| Rung | `wants_faster` (4 truly true) | `wants_slower` (5) | `wants_stop` (4) | `is_command` (17) |+|---|---|---|---|---|+| `noul_plain` | 9 — acc 0.611 | 14 — 0.389 | 5 — 0.944 | 12 — 0.722 |+| `noul_criteria` | 17 — 0.278 | 17 — 0.333 | **18** — 0.222 | 17 — **1.000** |+| `noul_scenario` | 17 — 0.278 | 17 — 0.333 | 17 — 0.278 | 17 — **1.000** |+| `noul_json_state` | **18** — 0.222 | **18** — 0.278 | **18** — 0.222 | **18** — 0.944 |++Once criteria are present, `wants_stop` answers "true" for all 18 commands — including the 14 that are not stop commands, with p(true) ≥ 0.794 in every case. At the top rung every dimension answers "true" for every case. `is_command` reaches 1.000 under criteria only because "true" is the common answer there (17/18), and it drops back to the always-true baseline (0.944) once the JSON state is added — at which point it also says "yes, a movement command" about 今天天气不错 ("the weather is nice today") with p(true) = 0.938, where `noul_plain` gave 0.005.++So the accuracy loss is a property of the answer distribution, not of the guidance teaching the model anything about Chinese. The confidence spread says the same thing from the other side: the top rung's 72 confidences sit in 0.750–0.995 while `noul_plain` spans 0.504–1.000.++**Task A does not show this.** No choice rung collapses: 0.7222 → 0.7778 → 0.6667, with the scenario helping by one case and the JSON state costing two. A `choice` question is not asked to return "yes" or "no" — the criteria keys are its options, so the answer cannot degenerate into a constant. That is the difference [#218](https://github.com/NandhaKishorM/laya/issues/218) could not see: the guidance it tested is harmful on the `noul` path and not on the `choice` path, and the two must not be averaged into one "Chinese accuracy".++## What the ladder does not change++Six of the failures are identical across all three choice rungs, so they are properties of the checkpoint on these inputs, not of the prompt format:++| # | Command | Gold | All three rungs | Confidence |+|---|---|---|---|---|+| 001 | 快一点 | `faster` | `slower` | 0.652 / 0.690 / 0.703 |+| 003 | 太慢了 | `faster` | `slower` | 0.990 / 0.993 / 0.898 |+| 006 | 太快了 | `slower` | `faster` | 0.795 / 0.901 / 0.767 |+| 016 | 往右边靠一下 | `right` | `left` | 0.395 / 0.399 / 0.556 |+| 010 | 别动了 | `stop` | `none` (2 of 3 rungs) | 0.607 / – / 0.796 |+| 017 | 别太快了 | `slower` | `faster` (JSON state only) | – / – / 0.556 |++Reading down the speed family: 太慢了 means "too slow" (go faster) and 太快了 means "too fast" (slow down) — the model picks the direction the adjective points to, not the correction the phrase asks for, and does it with 0.90–0.99 confidence. Negation is handled: 别太快了 and 别动了 are both read correctly in at least one rung, and 别太快了 is even corrected by the scenario sentence. So the pattern is specific to the 太 ("excessively") construction inverting the action, not to Chinese generally.++Direction routing is not uniformly broken either: 左转, 往左一点 and 向右转 are all correct in all three rungs, and the single failure (往右边靠一下 → `left`) is the least confident decision in the archive. Chit-chat is rejected: 今天天气不错 → `none` in all three rungs. Each of these rows is **one fixture** — treat them as cheap-to-falsify hypotheses about where to put the next 100 cases, not as established morphology claims.++**The two shapes disagree about what a command is at all.** For 6 of the 18 commands, `choice_criteria` and `noul_plain`'s `is_command` contradict each other:++| # | Command | `choice` says | `is_command` p(true) |+|---|---|---|---|+| 001 | 快一点 | `slower` | 0.254 |+| 005 | 慢一点 | `slower` | 0.496 |+| 010 | 别动了 | `none` | 0.681 |+| 013 | 左转 | `left` | 0.115 |+| 014 | 往左一点 | `left` | 0.226 |+| 017 | 别太快了 | `slower` | 0.462 |++A product has to pick one workflow and calibrate it, not consume both. This is the same conclusion the [sibling Feishu benchmark](../feishu_zh/README.md) reaches for its `choice`/`four_noul` pair.++## Provenance++Recorded by `run.py` in the archive's `config.checkpoint` block; the audit refuses an archive that does not pin the weights.++- Checkpoint: `multilingual/` subfolder of `convaiinnovations/laya`, read from a local directory (`E:/home/laya-models`). The bundled copy and the standalone `convaiinnovations/laya-multilingual` repo currently report the same LFS digest for the weights, so this is the published multilingual checkpoint.+- `model.safetensors`: 643835514 bytes, `sha256 9d628fd971b700382ac6f65920a86f149777b2e748e0c955fb3b19695aa8f204`.+- Run: 2026-09-24T07:56:04Z, CPU, float32, `laya 0.3.20`, `torch 2.14.0+cpu`, `transformers 5.17.0`, Python 3.12.+- Scoring: the checkpoint's own per-bucket temperatures, clamped exactly as `Agent` applies them (`unclamped: false` in the archive), the same code path as [`research/eval/laya_eval.py`](../../eval/laya_eval.py). No thresholds were fitted, and nothing here is fine-tuned.+- The whole seven-configuration sweep takes about 20 seconds on CPU.++`research/eval/README.md` records a different value for the same file — `sha256 b99c8bea…`. That string is the git blob id (`SHA-1`) of the *LFS pointer* stored in the Hub repository, not the SHA-256 of the weights: hashing the pointer text `version https://git-lfs.github.com/spec/v1\noid sha256:9d628fd9…\nsize 643835514\n` as a git blob reproduces `b99c8bea239c53f6f6bce734557dc6c403fa6b3e` exactly. The same file's `rl_agent_config.json` entry (`sha256 00e35f88…`) is likewise the blob id of the JSON itself. Anyone who checks these with `sha256sum` will see a mismatch and conclude the checkpoint changed when it did not. Flagged here rather than silently fixed, since it is a separate change to a file this contribution does not own.++## Run it on your machine++Download only the multilingual runtime files if you do not already have them (about 614 MiB):++```python+from huggingface_hub import snapshot_download+root = snapshot_download(+ "convaiinnovations/laya",+ allow_patterns=["multilingual/*.json", "multilingual/model.safetensors",+ "multilingual/encoder/*", "multilingual/tokenizer/*"],+)+print(root + "/multilingual")+```++Set `CHECKPOINT` to that directory, check the file, then run the sweep. `--stub` scores with a fixed pseudo-random vector instead of a checkpoint and exercises the whole pipeline offline; `--configs` runs a subset:++```bash+python research/benchmarks/zh_short_commands/run.py --checkpoint "$CHECKPOINT" \+ --device cpu --out /tmp/zh-short-commands+python research/benchmarks/zh_short_commands/audit.py --run-dir /tmp/zh-short-commands+```++The runner hashes the cases, `prompts.py` and the checkpoint's `model.safetensors` into the report, so an archive from another machine can be compared with the committed one field by field. Use a new output directory for every run. `--unclamped` reproduces the raw pre-clamp temperatures the older committed sweeps used; the archived numbers above use the clamped ones.++## Files++| Path | Purpose |+|---|---|+| `data/cases.jsonl` | The 18 frozen commands: id, text, family, gold label, notes |+| `data/manifest.json` | Label/family counts, the ladder, the policy, and the hashes of both frozen inputs |+| `prompts.py` | The seven rungs: instructions, six criteria, four `noul` dimensions and their criteria |+| `run.py` | The runner: loads a checkpoint, scores the ladder, writes `report.json` |+| `audit.py`, `tests/` | Offline re-derivation, 18 archive-corruption cases, and the metric pins against `laya_eval` |+| `results/v1/laya-multilingual/report.json` | The archived run: 7 report blocks and all 342 per-case records |++The three parts of the archive are the same three parts `research/eval/laya_eval.py` emits — `config`, `report`, `cases` — plus a `summary`. Nothing in `report` needs the model to be re-derived.++## Limitations and attribution++- 18 cases, hand-written by the contributor, no independent annotation and no inter-annotator agreement. Families are unbalanced (9 speed, 4 stop, 4 direction, 1 chit-chat) and so are the labels (1 to 5 per label), which is why the `noul` base rate matters more than the rung ordering.+- One checkpoint, one language, one device, greedy decoding at fixed temperatures. Nothing here characterizes the English or typed-decisions checkpoints.+- The comparison reproduces the *shape* of [#218](https://github.com/NandhaKishorM/laya/issues/218), not its prompts: that report's exact text was never published, so its ~50% figure is not directly comparable with the 0.4167–0.6667 above.+- These fixtures are a regression diagnostic. Do not tune on them and then report them as held-out.+- Chinese post-training remains an open research question; 中文 short-command routing is not solved by this file, it is measured by it.++Contributed by GaotianJin, with AI assistance for the harness, the audit and the tests. The case text is the contributor's own. No user data, recordings, credentials or model weights are included; the checkpoint stays under its own license. This directory is MIT-licensed like the rest of the repository.diff --git a/research/benchmarks/zh_short_commands/README.zh-CN.md b/research/benchmarks/zh_short_commands/README.zh-CN.mdnew file mode 100644index 00000000..2637ee68--- /dev/null+++ b/research/benchmarks/zh_short_commands/README.zh-CN.md@@ -0,0 +1,151 @@+# 中文短指令路由++[English](README.md) · [同类基准:中文职场决策](../feishu_zh/README.zh-CN.md) · [Issue #218](https://github.com/NandhaKishorM/laya/issues/218)++18 条冻结的中文清洁机器人语音指令、6 个标签,以及针对仓库自身提示词建议的 7 档消融阶梯。一个 checkpoint、一次运行、逐条决策全部归档——目的是把 [#218](https://github.com/NandhaKishorM/laya/issues/218) 的一次性报告("加上 criteria、场景描述和结构化 JSON state 之后,中文决策反而更糟")变成任何人都能复算的工件,并且说清它究竟对哪个原语成立。++**这 18 条是手写夹具,标签策略在任何模型运行之前就已固定。它不是留出测试集,不是独立标注语料,也不是官方评测。** 下面的数字只是一个 checkpoint 在这些夹具上的决策。++## 先做这一步:离线自证++在仓库根目录下,Python 3.10+:++```bash+python research/benchmarks/zh_short_commands/audit.py+python -m unittest discover -s research/benchmarks/zh_short_commands/tests -v+```++不需要下载模型、不需要联网。审计只用标准库,CI 把它放在一个什么都没安装的 job 里运行。测试套件会驱动 `run.py --stub`,因此需要 numpy——任何能跑 Laya 的环境本来就有。++审计会重新校验冻结用例与提示词的哈希、重新构造每一个请求(state、instructions、options、gold)而不是信任记录本身,并仅凭逐条记录复算 accuracy、macro-F1、ECE 和置信度离散度。测试会用 18 种方式篡改归档副本,要求审计全部拒绝;这些测试存在的意义就是证明这个审计真的会失败。++## 阶梯++`choice` 不可能没有 criteria——criteria 的键**就是**它的选项——所以任务 A 的起点比任务 B 高一档。++**任务 A,`choice`:每条指令一个问题,六选一。**++| 档位 | 正确 | accuracy | macro-F1 | ECE | 平均置信 | 置信标准差 |+|---|---:|---:|---:|---:|---:|---:|+| `choice_criteria`(只有六个选项) | 13/18 | 0.7222 | 0.7149 | 0.2659 | 0.8008 | 0.1728 |+| `choice_scenario`(+场景句) | 14/18 | 0.7778 | 0.7942 | 0.1827 | 0.8257 | 0.1725 |+| `choice_json_state`(+结构化 state) | 12/18 | 0.6667 | 0.6817 | 0.2484 | 0.7961 | 0.1238 |++最高频标签只覆盖 5/18,所以多数类基线是 0.2778。++**任务 B,`noul`:每条指令四个独立的 yes/no 问题,共 72 个决策。**++| 档位 | 正确 | accuracy | 平均置信 | 置信标准差 |+|---|---:|---:|---:|---:|+| `noul_plain`(无 criteria,单问题) | 48/72 | 0.6667 | 0.8128 | 0.1786 |+| `noul_criteria` | 33/72 | 0.4583 | 0.8920 | 0.0944 |+| `noul_scenario` | 34/72 | 0.4722 | 0.8944 | 0.0807 |+| `noul_json_state` | 30/72 | 0.4167 | 0.9437 | 0.0418 |++对 72 个问题全答 "true" 得 30/72 = 0.4167,因为 72 个 gold 答案里只有 30 个为真。`noul_json_state` 拿到的正好就是这个分数。++## 阶梯说明了什么++文档建议并没有让 `noul` 路径"更懂中文",而是让这四个问题不再有区分度。统计每个维度对 18 条指令答 "true" 的条数:++| 档位 | `wants_faster`(真值 4) | `wants_slower`(5) | `wants_stop`(4) | `is_command`(17) |+|---|---|---|---|---|+| `noul_plain` | 9 — acc 0.611 | 14 — 0.389 | 5 — 0.944 | 12 — 0.722 |+| `noul_criteria` | 17 — 0.278 | 17 — 0.333 | **18** — 0.222 | 17 — **1.000** |+| `noul_scenario` | 17 — 0.278 | 17 — 0.333 | 17 — 0.278 | 17 — **1.000** |+| `noul_json_state` | **18** — 0.222 | **18** — 0.278 | **18** — 0.222 | **18** — 0.944 |++一旦加上 criteria,`wants_stop` 对全部 18 条指令都答 "true"——包括那 14 条根本不是停止指令的,且每条的 p(true) ≥ 0.794。到最高档,四个维度对每一条指令都答 "true"。`is_command` 在 criteria 档拿到 1.000,只是因为那里 "true" 本来就是常见答案(17/18);再叠上 JSON state 后它退回全真基线(0.944)——此时它还会对 今天天气不错 说"是运动指令",p(true) = 0.938,而 `noul_plain` 给的是 0.005。++所以准确率的下降是答案分布的属性,不是建议真的教给了模型什么中文知识。置信度离散度从另一侧印证同一件事:最高档 72 个置信度全部落在 0.750–0.995,而 `noul_plain` 覆盖 0.504–1.000。++**任务 A 没有这个问题。** 三个 choice 档都没有塌缩:0.7222 → 0.7778 → 0.6667,场景句多对了 1 条,JSON state 少对了 2 条。`choice` 问题不需要回答 "是/否"——criteria 的键就是它的选项,答案无法退化成常数。这正是 [#218](https://github.com/NandhaKishorM/laya/issues/218) 看不到的差别:它测的那套建议对 `noul` 路径有害、对 `choice` 路径无害,两者不能平均成一个"中文准确率"。++## 阶梯改变不了的部分++有 6 个错误在三个 choice 档里完全一致,说明它们是 checkpoint 在这些输入上的属性,而不是提示词格式造成的:++| # | 指令 | Gold | 三档结果 | 置信度 |+|---|---|---|---|---|+| 001 | 快一点 | `faster` | `slower` | 0.652 / 0.690 / 0.703 |+| 003 | 太慢了 | `faster` | `slower` | 0.990 / 0.993 / 0.898 |+| 006 | 太快了 | `slower` | `faster` | 0.795 / 0.901 / 0.767 |+| 016 | 往右边靠一下 | `right` | `left` | 0.395 / 0.399 / 0.556 |+| 010 | 别动了 | `stop` | `none`(3 档中的 2 档) | 0.607 / – / 0.796 |+| 017 | 别太快了 | `slower` | `faster`(仅 JSON state 档) | – / – / 0.556 |++只看速度族:太慢了 意为"太慢"(要更快),太快了 意为"太快"(要减速)——模型选的是形容词所指的方向,而不是整句要求的纠正动作,而且带着 0.90–0.99 的置信度。否定是被处理的:别太快了 和 别动了 至少在一个档里被读对,别太快了 甚至被场景句纠正过来。所以这个模式是针对 太("过度")构式反转动作的,不是中文整体的失败。++方向路由也不是整体崩坏:左转、往左一点、向右转 在三个档里全对,唯一的失败(往右边靠一下 → `left`)是归档里置信度最低的一次决策。闲聊会被拒绝:今天天气不错 → `none`,三个档全对。上面每一行都**只是一个夹具**——它们是"下一批 100 条该往哪儿放"的廉价可证伪假设,不是已确立的构式结论。++**两个任务的形状本身就不一致。** 有 6 条指令,`choice_criteria` 与 `noul_plain` 的 `is_command` 互相矛盾:++| # | 指令 | `choice` 判定 | `is_command` p(true) |+|---|---|---|---|+| 001 | 快一点 | `slower` | 0.254 |+| 005 | 慢一点 | `slower` | 0.496 |+| 010 | 别动了 | `none` | 0.681 |+| 013 | 左转 | `left` | 0.115 |+| 014 | 往左一点 | `left` | 0.226 |+| 017 | 别太快了 | `slower` | 0.462 |++产品侧只能选一种工作流并针对它标定,不能同时消费两者。这与[同类飞书基准](../feishu_zh/README.zh-CN.md)对它的 `choice`/`four_noul` 组合得出的结论一致。++## 溯源++由 `run.py` 写入归档的 `config.checkpoint`;缺少权重指纹的归档会被审计直接拒绝。++- Checkpoint:`convaiinnovations/laya` 的 `multilingual/` 子目录,从本地目录读取(`E:/home/laya-models`)。仓库内置副本与独立仓库 `convaiinnovations/laya-multilingual` 目前对权重报告同一个 LFS 摘要,因此这就是已发布的多语言 checkpoint。+- `model.safetensors`:643835514 字节,`sha256 9d628fd971b700382ac6f65920a86f149777b2e748e0c955fb3b19695aa8f204`。+- 本次运行:2026-09-24T07:56:04Z,CPU,float32,`laya 0.3.20`,`torch 2.14.0+cpu`,`transformers 5.17.0`,Python 3.12。+- 打分方式:使用 checkpoint 自带的分桶温度,并按 `Agent` 的方式做同样的钳制(归档里 `unclamped: false`),与 [`research/eval/laya_eval.py`](../../eval/laya_eval.py) 同一条代码路径。没有拟合任何阈值,也没有做微调。+- 七个配置的完整扫描在 CPU 上约 20 秒。++`research/eval/README.md` 对同一个文件记录的是另一个值——`sha256 b99c8bea…`。那个字符串是 Hub 仓库里**LFS 指针**的 git blob id(SHA-1),不是权重的 SHA-256:把指针文本 `version https://git-lfs.github.com/spec/v1\noid sha256:9d628fd9…\nsize 643835514\n` 按 git blob 规则哈希,得到的正是 `b99c8bea239c53f6f6bce734557dc6c403fa6b3e`。同一段里 `rl_agent_config.json` 的 `sha256 00e35f88…` 同样是该 JSON 自身的 blob id。任何人用 `sha256sum` 去核对都会看到不一致,从而误判"checkpoint 变了"。这里只做标记,不静默修改,因为那属于本贡献所不拥有的另一个文件的改动。++## 在自己的机器上运行++如果本地还没有,只下载多语言运行时文件(约 614 MiB):++```python+from huggingface_hub import snapshot_download+root = snapshot_download(+ "convaiinnovations/laya",+ allow_patterns=["multilingual/*.json", "multilingual/model.safetensors",+ "multilingual/encoder/*", "multilingual/tokenizer/*"],+)+print(root + "/multilingual")+```++把 `CHECKPOINT` 设成该目录,核对文件,然后跑扫描。`--stub` 用固定伪随机向量代替 checkpoint,可离线跑通整条流水线;`--configs` 只跑子集:++```bash+python research/benchmarks/zh_short_commands/run.py --checkpoint "$CHECKPOINT" \+ --device cpu --out /tmp/zh-short-commands+python research/benchmarks/zh_short_commands/audit.py --run-dir /tmp/zh-short-commands+```++runner 会把用例、`prompts.py` 和 checkpoint 的 `model.safetensors` 哈希一并写入报告,所以来自另一台机器的归档可以与已提交的归档逐字段比对。每次运行请使用新的输出目录。`--unclamped` 复现旧提交扫描使用的原始未钳制温度;上面的归档数字用的是钳制后的值。++## 文件++| 路径 | 用途 |+|---|---|+| `data/cases.jsonl` | 18 条冻结指令:id、文本、family、gold 标签、备注 |+| `data/manifest.json` | 标签/family 计数、阶梯、判定策略,以及两个冻结输入的哈希 |+| `prompts.py` | 七个档位:instructions、六个 criteria、四个 `noul` 维度及其 criteria |+| `run.py` | runner:加载 checkpoint、跑阶梯、写 `report.json` |+| `audit.py`、`tests/` | 离线复算、18 种归档篡改用例,以及与 `laya_eval` 的指标对照 |+| `results/v1/laya-multilingual/report.json` | 归档运行:7 个 report 块和全部 342 条逐例记录 |++归档的三个部分与 `research/eval/laya_eval.py` 输出的三个部分相同——`config`、`report`、`cases`——外加一个 `summary`。`report` 里的每个数字都不需要模型即可复算。++## 局限与署名++- 18 条用例,由贡献者手写,没有独立标注,也没有标注者一致性。family 不平衡(速度 9、停止 4、方向 4、闲聊 1),标签也不平衡(每个 1–5 条),这正是 `noul` 的基率比档位排序更重要的原因。+- 单一 checkpoint、单一语言、单一设备,固定温度下的贪心解码。这里没有任何结论适用于英文或 typed-decisions checkpoint。+- 本对比复现的是 [#218](https://github.com/NandhaKishorM/laya/issues/218) 的**形态**,不是它的提示词:那份报告的原文从未公开,因此它的约 50% 与上面的 0.4167–0.6667 不能直接比较。+- 这些夹具是回归诊断。不要在上面调参,再把结果当作留出集报告。+- 中文后训练仍是开放的研究问题;这份文件没有解决中文短指令路由,它只是把它测量出来。++由 GaotianJin 贡献,harness、审计与测试借助 AI 协助完成。用例文本为贡献者本人所写。不包含任何用户数据、录音、凭据或模型权重;checkpoint 仍遵循其自身许可。本目录与仓库其余部分一样采用 MIT 许可。diff --git a/research/benchmarks/zh_short_commands/__pycache__/audit.cpython-312.pyc b/research/benchmarks/zh_short_commands/__pycache__/audit.cpython-312.pycnew file mode 100644index 00000000..aac6f66eBinary files /dev/null and b/research/benchmarks/zh_short_commands/__pycache__/audit.cpython-312.pyc differdiff --git a/research/benchmarks/zh_short_commands/__pycache__/prompts.cpython-312.pyc b/research/benchmarks/zh_short_commands/__pycache__/prompts.cpython-312.pycnew file mode 100644index 00000000..200c1b98Binary files /dev/null and b/research/benchmarks/zh_short_commands/__pycache__/prompts.cpython-312.pyc differdiff --git a/research/benchmarks/zh_short_commands/__pycache__/run.cpython-312.pyc b/research/benchmarks/zh_short_commands/__pycache__/run.cpython-312.pycnew file mode 100644index 00000000..1a79ba42Binary files /dev/null and b/research/benchmarks/zh_short_commands/__pycache__/run.cpython-312.pyc differdiff --git a/research/benchmarks/zh_short_commands/audit.py b/research/benchmarks/zh_short_commands/audit.pynew file mode 100644index 00000000..0a2ac8ad--- /dev/null+++ b/research/benchmarks/zh_short_commands/audit.py@@ -0,0 +1,297 @@+"""Re-derive every published number from an archive, offline.++ python research/benchmarks/zh_short_commands/audit.py+ python research/benchmarks/zh_short_commands/audit.py --run-dir DIR++No model, no network and no third-party package: the frozen cases, the frozen prompts+and the per-case records are enough to recompute each ``report`` block. What is checked:++* the three hashes (cases, prompts, and the ones the archive recorded) still agree, so+ a record cannot be swapped for one scored under a different prompt;+* every record names a real case, and its text/state/instructions/gold are re-built+ from ``prompts.py`` rather than trusted;+* every config is present with the right multiplicity and no duplicates;+* probabilities are finite and in [0, 1], and the prediction follows from them;+* ``report`` is recomputed from ``cases`` alone and must match what was archived.++The metrics below are re-implemented without numpy on purpose. ``tests/test_audit.py``+pins them against ``research/eval/laya_eval.py``, so this file cannot drift from the+harness that produced the numbers.+"""+import argparse+import hashlib+import json+import math+import os+import sys++HERE = os.path.dirname(os.path.abspath(__file__))+ROOT = os.path.dirname(os.path.dirname(os.path.dirname(HERE)))+if ROOT not in sys.path:+ sys.path.insert(0, ROOT)+if HERE not in sys.path:+ sys.path.insert(0, HERE)++import prompts # noqa: E402++from research.eval.laya_eval import ECE_BINS # noqa: E402++DEFAULT_RUN = os.path.join(HERE, "results", "v1", "laya-multilingual")+PLACES = 4 # the archive's own rounding for accuracy / macro_f1 / ece+++def sha256_file(path):+ with open(path, "rb") as f:+ return hashlib.sha256(f.read()).hexdigest()+++def load_json(path):+ with open(path, encoding="utf-8") as f:+ return json.load(f)+++def load_frozen():+ """The two frozen inputs plus the manifest that pins them."""+ cases = []+ with open(os.path.join(HERE, "data", "cases.jsonl"), encoding="utf-8") as f:+ for line in f:+ if line.strip():+ cases.append(json.loads(line))+ manifest = load_json(os.path.join(HERE, "data", "manifest.json"))+ if len({c["id"] for c in cases}) != len(cases):+ raise ValueError("duplicate case id in data/cases.jsonl")+ if any(c["gold"] not in prompts.LABELS for c in cases):+ raise ValueError("a gold label is outside LABELS")+ for key, path in (("cases_sha256", "data/cases.jsonl"), ("prompts_sha256", "prompts.py")):+ if sha256_file(os.path.join(HERE, path)) != manifest[key]:+ raise ValueError(+ "%s no longer matches the manifest: %s changed after the archive was "+ "produced, so the archived numbers cannot be re-derived from it" % (key, path))+ if manifest["cases"] != len(cases):+ raise ValueError("manifest count %r != %d cases" % (manifest["cases"], len(cases)))+ return cases, manifest+++# ------------------------------------------------------------------ metrics (no numpy)+def macro_f1(gold, pred):+ """Unweighted mean F1 over the classes gold and pred use, as laya_eval computes it."""+ scores = []+ for cls in sorted(set(gold) | set(pred)):+ tp = sum(1 for g, p in zip(gold, pred) if p == cls and g == cls)+ fp = sum(1 for g, p in zip(gold, pred) if p == cls and g != cls)+ fn = sum(1 for g, p in zip(gold, pred) if p != cls and g == cls)+ scores.append(2 * tp / max(1, 2 * tp + fp + fn))+ return sum(scores) / len(scores) if scores else float("nan")+++def ece(confidence, correct, bins=ECE_BINS):+ """Equal-width bins; the first is closed at the bottom, matching laya_eval."""+ total = 0.0+ for i in range(bins):+ lo, hi = i / bins, (i + 1) / bins+ sel = [j for j, c in enumerate(confidence)+ if (c >= lo if i == 0 else c > lo) and c <= hi]+ if sel:+ total += (len(sel) / len(confidence)) * abs(+ sum(confidence[j] for j in sel) / len(sel)+ - sum(correct[j] for j in sel) / len(sel))+ return total+++def mean(values):+ return sum(values) / len(values)+++def std(values):+ mu = mean(values)+ return (sum((v - mu) ** 2 for v in values) / len(values)) ** 0.5+++def close(got, want, places):+ """Equal once both are rounded to the archive's precision."""+ return abs(got - want) <= 0.5 * 10 ** -places+++def check(name, got, want, places=PLACES):+ if not close(got, want, places):+ raise ValueError("%s: re-derived %.8f, archived %s" % (name, got, want))+++# ---------------------------------------------------------------------------- records+def audit_records(doc, cases, manifest):+ """Every record must describe a real request, and no request may be missing."""+ by_id = {c["id"]: c for c in cases}+ configs = doc["config"]["configs"]+ if configs != list(manifest["ladder"]["choice"]) + list(manifest["ladder"]["noul"]):+ raise ValueError("the archive scored a different ladder than the manifest freezes")+ expected = {}+ for config in configs:+ if config in prompts.CHOICE_CONFIGS:+ expected[config] = {c["id"]: {None} for c in cases}+ else:+ expected[config] = {c["id"]: set(prompts.NOUL_DIMENSIONS) for c in cases}+ seen = {}+ temperatures = {}+ for record in doc["cases"]:+ config = record["config"]+ if config not in expected:+ raise ValueError("record for a config the archive did not score: %r" % config)+ case_id, dim = record["case_id"], record.get("dimension")+ if case_id not in by_id:+ raise ValueError("record for an unknown case: %r" % case_id)+ if dim in seen.setdefault(config, {}).get(case_id, set()):+ raise ValueError("duplicate record: %s / %s / %s" % (config, case_id, dim))+ seen.setdefault(config, {}).setdefault(case_id, set()).add(dim)+ case = by_id[case_id]+ where = "%s / %s / %s" % (config, case_id, dim)+ if record["text"] != case["text"] or record["gold_label"] != case["gold"]:+ raise ValueError("%s: the frozen text or gold label was altered" % where)+ if record["state"] != prompts.state_for(config, case["text"]):+ raise ValueError("%s: state is not what this rung sends" % where)+ if record["family"] != case["family"]:+ raise ValueError("%s: family was altered" % where)+ temperatures.setdefault(config, set()).add(record["temperature"])+ if dim is None:+ expected_instructions = prompts.choice_question(config)["intent"]["instructions"]+ if record["instructions"] != expected_instructions:+ raise ValueError("%s: instructions are not what this rung sends" % where)+ if record["options"] != list(prompts.CHOICE_CRITERIA):+ raise ValueError("%s: options are not the label policy" % where)+ for key in ("probability", "p_gold", "confidence"):+ if not 0.0 <= record[key] <= 1.0 or not math.isfinite(record[key]):+ raise ValueError("%s: %s is not a probability" % (where, key))+ if record["pred_label"] != prompts.LABELS[record["pred_index"]]:+ raise ValueError("%s: pred_label/pred_index disagree" % where)+ if record["correct"] != int(record["pred_index"] == record["gold_index"]):+ raise ValueError("%s: correct contradicts the prediction" % where)+ if record["probability"] != record["confidence"]:+ raise ValueError("%s: the recorded probability is not the argmax one" % where)+ if record["pred_label"] != case["gold"] and record["correct"]:+ raise ValueError("%s: scored correct but the label differs" % where)+ else:+ if dim not in prompts.NOUL_DIMENSIONS:+ raise ValueError("%s: not a frozen dimension" % where)+ expected_instructions = prompts.noul_questions(config)[dim]["instructions"]+ if record["instructions"] != expected_instructions:+ raise ValueError("%s: instructions are not what this rung sends" % where)+ if record["options"] != ["false", "true"]:+ raise ValueError("%s: noul options must be false/true" % where)+ if not 0.0 <= record["p_true"] <= 1.0 or not math.isfinite(record["p_true"]):+ raise ValueError("%s: p_true is not a probability" % where)+ if record["pred_bool"] != (record["p_true"] >= 0.5):+ raise ValueError("%s: pred_bool does not follow from p_true" % where)+ if record["gold_bool"] != bool(prompts.NOUL_GOLD[dim](case["gold"])):+ raise ValueError("%s: gold_bool does not follow from the gold label" % where)+ if record["correct"] != int(record["pred_bool"] == record["gold_bool"]):+ raise ValueError("%s: correct contradicts the prediction" % where)+ if record["confidence"] != round(max(record["p_true"], 1.0 - record["p_true"]), 6):+ raise ValueError("%s: confidence is not the two-way max" % where)+ missing = [(config, case_id, dim)+ for config, per_case in expected.items()+ for case_id, dims in per_case.items()+ for dim in dims+ if dim not in seen.get(config, {}).get(case_id, set())]+ if missing:+ raise ValueError("incomplete archive: %d missing records, first %r" % (len(missing), missing[0]))+ for config, values in temperatures.items():+ if len(values) != 1 or doc["report"][config]["temperature"] not in values:+ raise ValueError("%s: the report's temperature is not the one the records used" % config)+ return configs+++def audit_report(doc):+ """Recompute every report block from the per-case records."""+ for config, report in doc["report"].items():+ records = [r for r in doc["cases"] if r["config"] == config]+ confidences = [r["confidence"] for r in records]+ corrects = [r["correct"] for r in records]+ check("%s/accuracy" % config, mean(corrects), report["accuracy"])+ check("%s/mean_confidence" % config, mean(confidences), report["mean_confidence"], 6)+ if config in prompts.NOUL_CONFIGS:+ check("%s/n"% config, len(records), report["n_cases"] * report["n_dimensions"])+ check("%s/n_cases" % config, len(records) / len(prompts.NOUL_DIMENSIONS), report["n_cases"])+ check("%s/n_dimensions" % config, len(prompts.NOUL_DIMENSIONS), report["n_dimensions"])+ check("%s/std_confidence" % config, std(confidences), report["std_confidence"], 6)+ for dim, value in report["accuracy_by_dimension"].items():+ subset = [r["correct"] for r in records if r["dimension"] == dim]+ if not subset:+ raise ValueError("%s: %s has no records" % (config, dim))+ check("%s/%s" % (config, dim), mean(subset), value, 6)+ if sorted(report["accuracy_by_dimension"]) != sorted(prompts.NOUL_DIMENSIONS):+ raise ValueError("%s: report dims are not the frozen ones" % config)+ else:+ check("%s/n" % config, len(records), report["n"])+ golds = [r["gold_index"] for r in records]+ preds = [r["pred_index"] for r in records]+ check("%s/macro_f1" % config, macro_f1(golds, preds), report["macro_f1"])+ check("%s/ece" % config, ece(confidences, corrects), report["ece"])+ order = sorted(range(len(records)), key=lambda i: (-confidences[i], i))+ top_half = [corrects[i] for i in order[:max(1, len(records) // 2)]]+ check("%s/acc_at_50_coverage" % config, mean(top_half),+ report["acc_at_50_coverage"])+ if set(report) - {"n", "n_cases", "n_dimensions", "accuracy", "macro_f1", "ece",+ "mean_confidence", "std_confidence", "acc_at_50_coverage",+ "temperature", "accuracy_by_dimension"}:+ raise ValueError("%s: the report carries an unexplained field" % config)+++def audit_provenance(doc):+ """Say out loud what was scored. A real run must pin a weight, not just a repo id."""+ checkpoint = doc["config"]["checkpoint"]+ for key in ("checkpoint", "subfolder", "device"):+ if not checkpoint.get(key):+ raise ValueError("provenance is missing %r" % key)+ if checkpoint.get("stub"):+ return "stub run: no checkpoint was loaded"+ for key in ("weight_sha256", "weight_bytes"):+ if not checkpoint.get(key):+ raise ValueError(+ "no %s: the archive does not say which weights produced these numbers, "+ "and the reported hash is the only way to tell two checkpoints apart" % key)+ return "%s/%s %s bytes sha256 %s" % (+ checkpoint["checkpoint"], checkpoint["subfolder"], checkpoint["weight_bytes"],+ checkpoint["weight_sha256"][:16] + "...")+++def audit_run(folder=DEFAULT_RUN):+ cases, manifest = load_frozen()+ path = os.path.join(folder, "report.json")+ if not os.path.exists(path):+ raise ValueError("no report.json under %s" % folder)+ doc = load_json(path)+ if sorted(doc) != ["cases", "config", "report", "summary"]:+ raise ValueError("unexpected document parts: %r" % sorted(doc))+ for key in ("cases_sha256", "prompts_sha256"):+ if doc["config"][key] != manifest[key]:+ raise ValueError("the archive was scored under a different frozen protocol")+ if not isinstance(doc["config"]["unclamped"], bool):+ raise ValueError("`unclamped` must be recorded as a boolean")+ configs = audit_records(doc, cases, manifest)+ audit_report(doc)+ for config in configs:+ if doc["summary"][config]["accuracy"] != doc["report"][config]["accuracy"]:+ raise ValueError("summary/%s disagrees with its report block" % config)+ return {"folder": os.path.basename(os.path.normpath(folder)),+ "cases": len(cases), "configs": configs, "records": len(doc["cases"]),+ "provenance": audit_provenance(doc)}+++def main(argv=None):+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])+ parser.add_argument("--run-dir", default=DEFAULT_RUN,+ help="directory holding report.json (default: the committed archive)")+ args = parser.parse_args(argv)+ try:+ result = audit_run(args.run_dir)+ except ValueError as exc:+ print("FAIL %s" % exc)+ return 1+ print("ok %s: %d cases, %d configs, %d records" % (+ result["folder"], result["cases"], len(result["configs"]), result["records"]))+ print("ok re-derived from cases alone: accuracy, macro_f1, ece, mean/std confidence")+ print("ok provenance: %s" % result["provenance"])+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/research/benchmarks/zh_short_commands/data/cases.jsonl b/research/benchmarks/zh_short_commands/data/cases.jsonlnew file mode 100644index 00000000..caeb0cfc--- /dev/null+++ b/research/benchmarks/zh_short_commands/data/cases.jsonl@@ -0,0 +1,18 @@+{"id": "zh-cmd-001", "text": "快一点", "family": "speed", "gold": "faster", "notes": ""}+{"id": "zh-cmd-002", "text": "加速", "family": "speed", "gold": "faster", "notes": ""}+{"id": "zh-cmd-003", "text": "太慢了", "family": "speed", "gold": "faster", "notes": ""}+{"id": "zh-cmd-004", "text": "再快些", "family": "speed", "gold": "faster", "notes": ""}+{"id": "zh-cmd-005", "text": "慢一点", "family": "speed", "gold": "slower", "notes": ""}+{"id": "zh-cmd-006", "text": "太快了", "family": "speed", "gold": "slower", "notes": "adversarial: contains 太 (degree marker) but means slow down"}+{"id": "zh-cmd-007", "text": "减速", "family": "speed", "gold": "slower", "notes": ""}+{"id": "zh-cmd-008", "text": "慢下来", "family": "speed", "gold": "slower", "notes": ""}+{"id": "zh-cmd-009", "text": "停下", "family": "stop", "gold": "stop", "notes": ""}+{"id": "zh-cmd-010", "text": "别动了", "family": "stop", "gold": "stop", "notes": "negation form"}+{"id": "zh-cmd-011", "text": "急停", "family": "stop", "gold": "stop", "notes": ""}+{"id": "zh-cmd-012", "text": "停下来别动", "family": "stop", "gold": "stop", "notes": "composed of two stop phrases"}+{"id": "zh-cmd-013", "text": "左转", "family": "direction", "gold": "left", "notes": ""}+{"id": "zh-cmd-014", "text": "往左一点", "family": "direction", "gold": "left", "notes": "with degree modifier 一点"}+{"id": "zh-cmd-015", "text": "向右转", "family": "direction", "gold": "right", "notes": ""}+{"id": "zh-cmd-016", "text": "往右边靠一下", "family": "direction", "gold": "right", "notes": ""}+{"id": "zh-cmd-017", "text": "别太快了", "family": "speed", "gold": "slower", "notes": "adversarial: negation 别 + degree 太"}+{"id": "zh-cmd-018", "text": "今天天气不错", "family": "other", "gold": "none", "notes": "chit-chat, not a command"}diff --git a/research/benchmarks/zh_short_commands/data/manifest.json b/research/benchmarks/zh_short_commands/data/manifest.jsonnew file mode 100644index 00000000..6e1ce74d--- /dev/null+++ b/research/benchmarks/zh_short_commands/data/manifest.json@@ -0,0 +1,30 @@+{+ "version": "1.0",+ "benchmark": "zh_short_commands",+ "provenance": "Hand-written Chinese commands with a label policy fixed before any model was run; no private recordings, no user data. The 18 cases are fixtures for a regression diagnostic, not a held-out test set.",+ "cases": 18,+ "labels": {+ "faster": 4,+ "slower": 5,+ "stop": 4,+ "left": 2,+ "right": 2,+ "none": 1+ },+ "families": {+ "speed": 9,+ "stop": 4,+ "direction": 4,+ "other": 1+ },+ "ladder": {+ "choice": ["choice_criteria", "choice_scenario", "choice_json_state"],+ "noul": ["noul_plain", "noul_criteria", "noul_scenario", "noul_json_state"]+ },+ "cases_sha256": "2d789879fe11328fb8b8099aaf35ddf7b2e301b393f9525701368339da28181e",+ "prompts_sha256": "d26c8357a302f887258382c5a40d1c1320adab8b3e86c364ef150a913a06686d",+ "policy": "标签就是机器人该执行的运动意图。快一点/加速/太慢了/再快些=faster;慢一点/太快了/减速/慢下来/别太快了=slower;停下/别动了/急停/停下来别动=stop;左转/往左一点=left;向右转/往右边靠一下=right;今天天气不错=none。程度副词与否定词按整句意图判断,不按字面:太快了 是 slower 而不是 faster。",+ "unclamped": false,+ "scoring": "choice 取六路的 temperature-scaled softmax 的 argmax;noul 每个维度独立取 p(true) >= 0.5。温度来自 checkpoint 自己的分桶值,与 research/eval/laya_eval.py 相同,未做任何阈值拟合。",+ "cases_without_a_dimension": "四个 noul 维度覆盖速度与停止,不覆盖方向:left/right 只应让 is_command 为真。方向标签正是 task A(六路 choice)存在的原因,两个任务不能合并成一个分数。"+}diff --git a/research/benchmarks/zh_short_commands/prompts.py b/research/benchmarks/zh_short_commands/prompts.pynew file mode 100644index 00000000..4c62cb74--- /dev/null+++ b/research/benchmarks/zh_short_commands/prompts.py@@ -0,0 +1,128 @@+"""Frozen prompts and label policy for the Chinese short-command benchmark.++Nothing in this file may change once a results archive is produced: run.py records+this file's hash and audit.py re-checks it, so a published number can always be+re-derived from the archive alone.++Motivation: #218 reported ~50% accuracy for `laya-multilingual` on 20 short Chinese+voice commands, and that following the documented guidance (adding criteria, a+scenario description, a structured JSON state) made discrimination *worse* — outputs+flattened into the 0.6–1.0 range. That report is not reproducible from the repository+(the prompt text was never published, and no per-case record exists), so this+benchmark freezes the inputs and records every decision. It reproduces the *shape* of+that comparison, not its exact prompts.++The ablation is a ladder over both task shapes, so the guidance claim is tested where+it was made (a single `noul` dimension) and where routing actually happens (six-way+`choice`):++ task B (noul) noul_plain -> noul_criteria -> noul_scenario -> noul_json_state+ task A (choice) choice_criteria -> choice_scenario -> choice_json_state++A `choice` question cannot exist without criteria — the criteria keys *are* its+options — so task A starts one rung higher.+"""++LABELS = ("faster", "slower", "stop", "left", "right", "none")++# --------------------------------------------------------------- choice (task A)+CHOICE_INSTRUCTIONS = "这条指令要求机器人做什么?"++CHOICE_CRITERIA = {+ "faster": "要求加快速度",+ "slower": "要求放慢速度",+ "stop": "要求停止运动",+ "left": "要求向左转向",+ "right": "要求向右转向",+ "none": "不是对机器人的运动指令",+}++SCENARIO_PREFIX = "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。"++# ---------------------------------------------------------------- noul (task B)+# Four independent yes/no dimensions. `noul_plain` omits criteria on purpose, so the+# checkpoint reads its default `false:` / `true:` slots — the "plain string, no+# criteria, single question" shape #218 measured. The known label sensitivity of+# `noul` (README, "Honest limits") applies to that rung and is why task A exists.+NOUL_DIMENSIONS = ("wants_faster", "wants_slower", "wants_stop", "is_command")++NOUL_INSTRUCTIONS = {+ "wants_faster": "这条指令是否要求加快速度?",+ "wants_slower": "这条指令是否要求放慢速度?",+ "wants_stop": "这条指令是否要求完全停止?",+ "is_command": "这条指令是否是对机器人的运动指令?",+}++# The criteria a `noul` carries once the ladder adds them. These two keys ARE the+# option text the model reads (docs, "Decision Primitives"), so they spell out the+# meaning rather than leaning on the default `false:` / `true:` labels.+NOUL_CRITERIA = {+ "wants_faster": {"true": "要求加快速度", "false": "不要求加快速度"},+ "wants_slower": {"true": "要求放慢速度", "false": "不要求放慢速度"},+ "wants_stop": {"true": "要求完全停止", "false": "不要求完全停止"},+ "is_command": {"true": "是对机器人的运动指令", "false": "不是对机器人的运动指令"},+}++# The gold boolean each dimension should return, derived from the case's gold label.+NOUL_GOLD = {+ "wants_faster": lambda gold: gold == "faster",+ "wants_slower": lambda gold: gold == "slower",+ "wants_stop": lambda gold: gold == "stop",+ "is_command": lambda gold: gold != "none",+}++# ------------------------------------------------------------------- ladder axes+CHOICE_CONFIGS = ("choice_criteria", "choice_scenario", "choice_json_state")+NOUL_CONFIGS = ("noul_plain", "noul_criteria", "noul_scenario", "noul_json_state")+CONFIGS = CHOICE_CONFIGS + NOUL_CONFIGS++_SCENARIO_CONFIGS = ("choice_scenario", "noul_scenario")+_JSON_STATE_CONFIGS = ("choice_json_state", "noul_json_state")+_NO_CRITERIA_CONFIGS = ("noul_plain",)+++def _instructions(base: str, config: str) -> str:+ if config not in CONFIGS:+ raise ValueError("unknown config: %r" % config)+ return (SCENARIO_PREFIX + base) if config in _SCENARIO_CONFIGS else base+++def choice_question(config: str) -> dict:+ """The single six-option choice question a config asks."""+ if config not in CHOICE_CONFIGS:+ raise ValueError("not a choice config: %r" % config)+ return {"intent": {"type": "choice",+ "instructions": _instructions(CHOICE_INSTRUCTIONS, config),+ "criteria": dict(CHOICE_CRITERIA)}}+++def noul_questions(config: str) -> dict:+ """The four dimension questions, with criteria wherever the ladder has them."""+ if config not in NOUL_CONFIGS:+ raise ValueError("not a noul config: %r" % config)+ questions = {}+ for dim in NOUL_DIMENSIONS:+ qdef = {"type": "noul", "instructions": _instructions(NOUL_INSTRUCTIONS[dim], config)}+ if config not in _NO_CRITERIA_CONFIGS:+ qdef["criteria"] = dict(NOUL_CRITERIA[dim])+ questions[dim] = qdef+ return questions+++def questions_for(config: str) -> dict:+ return choice_question(config) if config in CHOICE_CONFIGS else noul_questions(config)+++def state_for(config: str, text: str):+ """The state the config feeds the model; the json rungs wrap it in a dict."""+ if config not in CONFIGS:+ raise ValueError("unknown config: %r" % config)+ if config in _JSON_STATE_CONFIGS:+ return {"text": text, "source": "voice", "device": "vacuum"}+ return text+++def cases_for(config: str, texts):+ """`(state, questions)` pairs in the research/eval harness's own shape."""+ questions = questions_for(config)+ return [(state_for(config, text), questions) for text in texts]diff --git a/research/benchmarks/zh_short_commands/results/v1/laya-multilingual/report.json b/research/benchmarks/zh_short_commands/results/v1/laya-multilingual/report.jsonnew file mode 100644index 00000000..e17429c1--- /dev/null+++ b/research/benchmarks/zh_short_commands/results/v1/laya-multilingual/report.json@@ -0,0 +1,7620 @@+{+ "config": {+ "benchmark": "zh_short_commands",+ "cases_file": "research/benchmarks/zh_short_commands/data/cases.jsonl",+ "cases_sha256": "2d789879fe11328fb8b8099aaf35ddf7b2e301b393f9525701368339da28181e",+ "prompts_sha256": "d26c8357a302f887258382c5a40d1c1320adab8b3e86c364ef150a913a06686d",+ "configs": [+ "choice_criteria",+ "choice_scenario",+ "choice_json_state",+ "noul_plain",+ "noul_criteria",+ "noul_scenario",+ "noul_json_state"+ ],+ "unclamped": false,+ "timestamp": "2026-09-24T08:03:26Z",+ "checkpoint": {+ "checkpoint": "E:/home/laya-models",+ "subfolder": "multilingual",+ "device": "cpu",+ "stub": false,+ "torch": "2.14.0+cpu",+ "transformers": "5.17.0",+ "laya": "0.3.20",+ "weight_sha256": "9d628fd971b700382ac6f65920a86f149777b2e748e0c955fb3b19695aa8f204",+ "weight_bytes": 643835514+ }+ },+ "report": {+ "choice_criteria": {+ "n": 18,+ "accuracy": 0.7222,+ "macro_f1": 0.7149,+ "ece": 0.2659,+ "mean_confidence": 0.800806,+ "acc_at_50_coverage": 0.8889,+ "temperature": 1.0,+ "std_confidence": 0.172794+ },+ "choice_scenario": {+ "n": 18,+ "accuracy": 0.7778,+ "macro_f1": 0.7942,+ "ece": 0.1827,+ "mean_confidence": 0.82569,+ "acc_at_50_coverage": 0.8889,+ "temperature": 1.0,+ "std_confidence": 0.172482+ },+ "choice_json_state": {+ "n": 18,+ "accuracy": 0.6667,+ "macro_f1": 0.6817,+ "ece": 0.2484,+ "mean_confidence": 0.796137,+ "acc_at_50_coverage": 0.8889,+ "temperature": 1.0,+ "std_confidence": 0.123797+ },+ "noul_plain": {+ "n_cases": 18,+ "n_dimensions": 4,+ "accuracy_by_dimension": {+ "wants_faster": 0.611111,+ "wants_slower": 0.388889,+ "wants_stop": 0.944444,+ "is_command": 0.722222+ },+ "accuracy": 0.666667,+ "mean_confidence": 0.812754,+ "std_confidence": 0.178601,+ "temperature": 1.0+ },+ "noul_criteria": {+ "n_cases": 18,+ "n_dimensions": 4,+ "accuracy_by_dimension": {+ "wants_faster": 0.277778,+ "wants_slower": 0.333333,+ "wants_stop": 0.222222,+ "is_command": 1.0+ },+ "accuracy": 0.458333,+ "mean_confidence": 0.892017,+ "std_confidence": 0.09442,+ "temperature": 1.0+ },+ "noul_scenario": {+ "n_cases": 18,+ "n_dimensions": 4,+ "accuracy_by_dimension": {+ "wants_faster": 0.277778,+ "wants_slower": 0.333333,+ "wants_stop": 0.277778,+ "is_command": 1.0+ },+ "accuracy": 0.472222,+ "mean_confidence": 0.894393,+ "std_confidence": 0.080708,+ "temperature": 1.0+ },+ "noul_json_state": {+ "n_cases": 18,+ "n_dimensions": 4,+ "accuracy_by_dimension": {+ "wants_faster": 0.222222,+ "wants_slower": 0.277778,+ "wants_stop": 0.222222,+ "is_command": 0.944444+ },+ "accuracy": 0.416667,+ "mean_confidence": 0.94372,+ "std_confidence": 0.041769,+ "temperature": 1.0+ }+ },+ "summary": {+ "choice_criteria": {+ "accuracy": 0.7222,+ "mean_confidence": 0.800806+ },+ "choice_scenario": {+ "accuracy": 0.7778,+ "mean_confidence": 0.82569+ },+ "choice_json_state": {+ "accuracy": 0.6667,+ "mean_confidence": 0.796137+ },+ "noul_plain": {+ "accuracy": 0.666667,+ "mean_confidence": 0.812754+ },+ "noul_criteria": {+ "accuracy": 0.458333,+ "mean_confidence": 0.892017+ },+ "noul_scenario": {+ "accuracy": 0.472222,+ "mean_confidence": 0.894393+ },+ "noul_json_state": {+ "accuracy": 0.416667,+ "mean_confidence": 0.94372+ }+ },+ "cases": [+ {+ "case_id": "zh-cmd-001",+ "config": "choice_criteria",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.651619,+ "p_gold": 0.181172,+ "confidence": 0.651619,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "choice_criteria",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.997332,+ "p_gold": 0.997332,+ "confidence": 0.997332,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "choice_criteria",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.990078,+ "p_gold": 0.001747,+ "confidence": 0.990078,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "choice_criteria",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.812575,+ "p_gold": 0.812575,+ "confidence": 0.812575,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "choice_criteria",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.950299,+ "p_gold": 0.950299,+ "confidence": 0.950299,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "choice_criteria",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.795233,+ "p_gold": 0.075296,+ "confidence": 0.795233,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "choice_criteria",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.900545,+ "p_gold": 0.900545,+ "confidence": 0.900545,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "choice_criteria",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.672757,+ "p_gold": 0.672757,+ "confidence": 0.672757,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "choice_criteria",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.993855,+ "p_gold": 0.993855,+ "confidence": 0.993855,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "choice_criteria",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 5,+ "pred_label": "none",+ "probability": 0.606616,+ "p_gold": 0.352283,+ "confidence": 0.606616,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "choice_criteria",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.99788,+ "p_gold": 0.99788,+ "confidence": 0.99788,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "choice_criteria",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.972951,+ "p_gold": 0.972951,+ "confidence": 0.972951,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "choice_criteria",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 3,+ "gold_label": "left",+ "pred_index": 3,+ "pred_label": "left",+ "probability": 0.803919,+ "p_gold": 0.803919,+ "confidence": 0.803919,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "choice_criteria",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 3,+ "gold_label": "left",+ "pred_index": 3,+ "pred_label": "left",+ "probability": 0.806514,+ "p_gold": 0.806514,+ "confidence": 0.806514,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "choice_criteria",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 4,+ "gold_label": "right",+ "pred_index": 4,+ "pred_label": "right",+ "probability": 0.749254,+ "p_gold": 0.749254,+ "confidence": 0.749254,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "choice_criteria",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 4,+ "gold_label": "right",+ "pred_index": 3,+ "pred_label": "left",+ "probability": 0.395082,+ "p_gold": 0.317776,+ "confidence": 0.395082,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "choice_criteria",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.516001,+ "p_gold": 0.516001,+ "confidence": 0.516001,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "choice_criteria",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 5,+ "gold_label": "none",+ "pred_index": 5,+ "pred_label": "none",+ "probability": 0.801997,+ "p_gold": 0.801997,+ "confidence": 0.801997,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "choice_scenario",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.689976,+ "p_gold": 0.18651,+ "confidence": 0.689976,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "choice_scenario",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.995723,+ "p_gold": 0.995723,+ "confidence": 0.995723,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "choice_scenario",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.992945,+ "p_gold": 0.001966,+ "confidence": 0.992945,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "choice_scenario",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.900936,+ "p_gold": 0.900936,+ "confidence": 0.900936,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "choice_scenario",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.943041,+ "p_gold": 0.943041,+ "confidence": 0.943041,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "choice_scenario",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.900526,+ "p_gold": 0.042301,+ "confidence": 0.900526,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "choice_scenario",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.911091,+ "p_gold": 0.911091,+ "confidence": 0.911091,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "choice_scenario",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.69002,+ "p_gold": 0.69002,+ "confidence": 0.69002,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "choice_scenario",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.99106,+ "p_gold": 0.99106,+ "confidence": 0.99106,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "choice_scenario",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.567869,+ "p_gold": 0.567869,+ "confidence": 0.567869,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "choice_scenario",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.996281,+ "p_gold": 0.996281,+ "confidence": 0.996281,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "choice_scenario",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.960333,+ "p_gold": 0.960333,+ "confidence": 0.960333,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "choice_scenario",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 3,+ "gold_label": "left",+ "pred_index": 3,+ "pred_label": "left",+ "probability": 0.823617,+ "p_gold": 0.823617,+ "confidence": 0.823617,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "choice_scenario",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 3,+ "gold_label": "left",+ "pred_index": 3,+ "pred_label": "left",+ "probability": 0.793242,+ "p_gold": 0.793242,+ "confidence": 0.793242,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "choice_scenario",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 4,+ "gold_label": "right",+ "pred_index": 4,+ "pred_label": "right",+ "probability": 0.722302,+ "p_gold": 0.722302,+ "confidence": 0.722302,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "choice_scenario",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 4,+ "gold_label": "right",+ "pred_index": 3,+ "pred_label": "left",+ "probability": 0.398636,+ "p_gold": 0.36203,+ "confidence": 0.398636,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "choice_scenario",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.602565,+ "p_gold": 0.602565,+ "confidence": 0.602565,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "choice_scenario",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 5,+ "gold_label": "none",+ "pred_index": 5,+ "pred_label": "none",+ "probability": 0.98225,+ "p_gold": 0.98225,+ "confidence": 0.98225,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "choice_json_state",+ "text": "快一点",+ "state": {+ "text": "快一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.702641,+ "p_gold": 0.255393,+ "confidence": 0.702641,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "choice_json_state",+ "text": "加速",+ "state": {+ "text": "加速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.767622,+ "p_gold": 0.767622,+ "confidence": 0.767622,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "choice_json_state",+ "text": "太慢了",+ "state": {+ "text": "太慢了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.898452,+ "p_gold": 0.023512,+ "confidence": 0.898452,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "choice_json_state",+ "text": "再快些",+ "state": {+ "text": "再快些",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 0,+ "gold_label": "faster",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.819346,+ "p_gold": 0.819346,+ "confidence": 0.819346,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "choice_json_state",+ "text": "慢一点",+ "state": {+ "text": "慢一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.936087,+ "p_gold": 0.936087,+ "confidence": 0.936087,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "choice_json_state",+ "text": "太快了",+ "state": {+ "text": "太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.766948,+ "p_gold": 0.132466,+ "confidence": 0.766948,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "choice_json_state",+ "text": "减速",+ "state": {+ "text": "减速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.640793,+ "p_gold": 0.640793,+ "confidence": 0.640793,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "choice_json_state",+ "text": "慢下来",+ "state": {+ "text": "慢下来",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 1,+ "pred_label": "slower",+ "probability": 0.775337,+ "p_gold": 0.775337,+ "confidence": 0.775337,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "choice_json_state",+ "text": "停下",+ "state": {+ "text": "停下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.942234,+ "p_gold": 0.942234,+ "confidence": 0.942234,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "choice_json_state",+ "text": "别动了",+ "state": {+ "text": "别动了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 5,+ "pred_label": "none",+ "probability": 0.796476,+ "p_gold": 0.105588,+ "confidence": 0.796476,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "choice_json_state",+ "text": "急停",+ "state": {+ "text": "急停",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.961997,+ "p_gold": 0.961997,+ "confidence": 0.961997,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "choice_json_state",+ "text": "停下来别动",+ "state": {+ "text": "停下来别动",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 2,+ "gold_label": "stop",+ "pred_index": 2,+ "pred_label": "stop",+ "probability": 0.970309,+ "p_gold": 0.970309,+ "confidence": 0.970309,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "choice_json_state",+ "text": "左转",+ "state": {+ "text": "左转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 3,+ "gold_label": "left",+ "pred_index": 3,+ "pred_label": "left",+ "probability": 0.818902,+ "p_gold": 0.818902,+ "confidence": 0.818902,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "choice_json_state",+ "text": "往左一点",+ "state": {+ "text": "往左一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 3,+ "gold_label": "left",+ "pred_index": 3,+ "pred_label": "left",+ "probability": 0.871877,+ "p_gold": 0.871877,+ "confidence": 0.871877,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "choice_json_state",+ "text": "向右转",+ "state": {+ "text": "向右转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 4,+ "gold_label": "right",+ "pred_index": 4,+ "pred_label": "right",+ "probability": 0.840254,+ "p_gold": 0.840254,+ "confidence": 0.840254,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "choice_json_state",+ "text": "往右边靠一下",+ "state": {+ "text": "往右边靠一下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 4,+ "gold_label": "right",+ "pred_index": 3,+ "pred_label": "left",+ "probability": 0.555916,+ "p_gold": 0.211429,+ "confidence": 0.555916,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "choice_json_state",+ "text": "别太快了",+ "state": {+ "text": "别太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 1,+ "gold_label": "slower",+ "pred_index": 0,+ "pred_label": "faster",+ "probability": 0.556027,+ "p_gold": 0.390303,+ "confidence": 0.556027,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "choice_json_state",+ "text": "今天天气不错",+ "state": {+ "text": "今天天气不错",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "other",+ "instructions": "这条指令要求机器人做什么?",+ "options": [+ "faster",+ "slower",+ "stop",+ "left",+ "right",+ "none"+ ],+ "gold_index": 5,+ "gold_label": "none",+ "pred_index": 5,+ "pred_label": "none",+ "probability": 0.709246,+ "p_gold": 0.709246,+ "confidence": 0.709246,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_plain",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.954476,+ "confidence": 0.954476,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_plain",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.599379,+ "confidence": 0.599379,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_plain",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": false,+ "p_true": 0.042572,+ "confidence": 0.957428,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_plain",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": false,+ "p_true": 0.253537,+ "confidence": 0.746463,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_plain",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.98982,+ "confidence": 0.98982,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_plain",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": false,+ "p_true": 0.483596,+ "confidence": 0.516404,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_plain",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": false,+ "p_true": 0.004238,+ "confidence": 0.995762,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_plain",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.80819,+ "confidence": 0.80819,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_plain",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": false,+ "p_true": 0.039422,+ "confidence": 0.960578,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_plain",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.951992,+ "confidence": 0.951992,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_plain",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": false,+ "p_true": 0.249589,+ "confidence": 0.750411,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_plain",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.814598,+ "confidence": 0.814598,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_plain",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.994174,+ "confidence": 0.994174,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_plain",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": false,+ "p_true": 0.472889,+ "confidence": 0.527111,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_plain",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": false,+ "p_true": 0.010075,+ "confidence": 0.989925,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_plain",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.713537,+ "confidence": 0.713537,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_plain",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.526816,+ "confidence": 0.526816,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_plain",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.991903,+ "confidence": 0.991903,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_plain",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": false,+ "p_true": 0.012858,+ "confidence": 0.987142,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_plain",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": false,+ "p_true": 0.496322,+ "confidence": 0.503678,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_plain",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.765595,+ "confidence": 0.765595,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_plain",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": false,+ "p_true": 0.014266,+ "confidence": 0.985734,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_plain",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": false,+ "p_true": 0.010283,+ "confidence": 0.989717,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_plain",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.676686,+ "confidence": 0.676686,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_plain",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": false,+ "p_true": 0.007463,+ "confidence": 0.992537,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_plain",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.970536,+ "confidence": 0.970536,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_plain",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": false,+ "p_true": 0.092154,+ "confidence": 0.907846,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_plain",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.636748,+ "confidence": 0.636748,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_plain",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.519541,+ "confidence": 0.519541,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_plain",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.988187,+ "confidence": 0.988187,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_plain",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": false,+ "p_true": 0.327777,+ "confidence": 0.672223,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_plain",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.620821,+ "confidence": 0.620821,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_plain",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": false,+ "p_true": 0.045298,+ "confidence": 0.954702,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_plain",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.959682,+ "confidence": 0.959682,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_plain",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.990001,+ "confidence": 0.990001,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_plain",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.56316,+ "confidence": 0.56316,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_plain",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": false,+ "p_true": 0.307061,+ "confidence": 0.692939,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_plain",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.933885,+ "confidence": 0.933885,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_plain",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.966055,+ "confidence": 0.966055,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_plain",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.680622,+ "confidence": 0.680622,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_plain",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": false,+ "p_true": 0.488016,+ "confidence": 0.511984,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_plain",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.524,+ "confidence": 0.524,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_plain",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.96952,+ "confidence": 0.96952,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_plain",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.595636,+ "confidence": 0.595636,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_plain",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": false,+ "p_true": 0.23929,+ "confidence": 0.76071,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_plain",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.977958,+ "confidence": 0.977958,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_plain",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.994788,+ "confidence": 0.994788,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_plain",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.703027,+ "confidence": 0.703027,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_plain",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": false,+ "p_true": 0.210506,+ "confidence": 0.789494,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_plain",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.644998,+ "confidence": 0.644998,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_plain",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": false,+ "p_true": 0.007574,+ "confidence": 0.992426,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_plain",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "left",+ "pred_bool": false,+ "p_true": 0.114698,+ "confidence": 0.885302,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_plain",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.991587,+ "confidence": 0.991587,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_plain",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.935833,+ "confidence": 0.935833,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_plain",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": false,+ "p_true": 0.000611,+ "confidence": 0.999389,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_plain",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "left",+ "pred_bool": false,+ "p_true": 0.225746,+ "confidence": 0.774254,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_plain",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.766413,+ "confidence": 0.766413,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_plain",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.704833,+ "confidence": 0.704833,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_plain",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": false,+ "p_true": 0.00692,+ "confidence": 0.99308,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_plain",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.565615,+ "confidence": 0.565615,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_plain",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": false,+ "p_true": 0.475074,+ "confidence": 0.524926,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_plain",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.544827,+ "confidence": 0.544827,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_plain",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": false,+ "p_true": 0.059072,+ "confidence": 0.940928,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_plain",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.640157,+ "confidence": 0.640157,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_plain",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.647588,+ "confidence": 0.647588,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_plain",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.995087,+ "confidence": 0.995087,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_plain",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.868402,+ "confidence": 0.868402,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_plain",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": false,+ "p_true": 0.461903,+ "confidence": 0.538097,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_plain",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.000733,+ "confidence": 0.999267,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_plain",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.003372,+ "confidence": 0.996628,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_plain",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 4.2e-05,+ "confidence": 0.999958,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_plain",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.005458,+ "confidence": 0.994542,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_criteria",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.805725,+ "confidence": 0.805725,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_criteria",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.956139,+ "confidence": 0.956139,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_criteria",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.960723,+ "confidence": 0.960723,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_criteria",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.89739,+ "confidence": 0.89739,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_criteria",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.947568,+ "confidence": 0.947568,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_criteria",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.921595,+ "confidence": 0.921595,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_criteria",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.926041,+ "confidence": 0.926041,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_criteria",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.897551,+ "confidence": 0.897551,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_criteria",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.55781,+ "confidence": 0.55781,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_criteria",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.883181,+ "confidence": 0.883181,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_criteria",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.964062,+ "confidence": 0.964062,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_criteria",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.92244,+ "confidence": 0.92244,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_criteria",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.861991,+ "confidence": 0.861991,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_criteria",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.908948,+ "confidence": 0.908948,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_criteria",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.980353,+ "confidence": 0.980353,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_criteria",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.949627,+ "confidence": 0.949627,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_criteria",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.900026,+ "confidence": 0.900026,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_criteria",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.972174,+ "confidence": 0.972174,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_criteria",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.954328,+ "confidence": 0.954328,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_criteria",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.919837,+ "confidence": 0.919837,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_criteria",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.666823,+ "confidence": 0.666823,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_criteria",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.834933,+ "confidence": 0.834933,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_criteria",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.963844,+ "confidence": 0.963844,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_criteria",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.916419,+ "confidence": 0.916419,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_criteria",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.936497,+ "confidence": 0.936497,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_criteria",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.917183,+ "confidence": 0.917183,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_criteria",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.97869,+ "confidence": 0.97869,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_criteria",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.943498,+ "confidence": 0.943498,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_criteria",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.857419,+ "confidence": 0.857419,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_criteria",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.949304,+ "confidence": 0.949304,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_criteria",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.962356,+ "confidence": 0.962356,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_criteria",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.939307,+ "confidence": 0.939307,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_criteria",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.842504,+ "confidence": 0.842504,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_criteria",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.91113,+ "confidence": 0.91113,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_criteria",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.992434,+ "confidence": 0.992434,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_criteria",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.971271,+ "confidence": 0.971271,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_criteria",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.776586,+ "confidence": 0.776586,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_criteria",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.781802,+ "confidence": 0.781802,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_criteria",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.961058,+ "confidence": 0.961058,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_criteria",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.954891,+ "confidence": 0.954891,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_criteria",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.902188,+ "confidence": 0.902188,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_criteria",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.960203,+ "confidence": 0.960203,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_criteria",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.996282,+ "confidence": 0.996282,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_criteria",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.961597,+ "confidence": 0.961597,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_criteria",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.923742,+ "confidence": 0.923742,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_criteria",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.971927,+ "confidence": 0.971927,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_criteria",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.99607,+ "confidence": 0.99607,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_criteria",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.98296,+ "confidence": 0.98296,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_criteria",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.592094,+ "confidence": 0.592094,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_criteria",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.743342,+ "confidence": 0.743342,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_criteria",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.861027,+ "confidence": 0.861027,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_criteria",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.891144,+ "confidence": 0.891144,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_criteria",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.710524,+ "confidence": 0.710524,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_criteria",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.917786,+ "confidence": 0.917786,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_criteria",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.900599,+ "confidence": 0.900599,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_criteria",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.875497,+ "confidence": 0.875497,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_criteria",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.727729,+ "confidence": 0.727729,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_criteria",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.965864,+ "confidence": 0.965864,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_criteria",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.948551,+ "confidence": 0.948551,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_criteria",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.934045,+ "confidence": 0.934045,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_criteria",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.74497,+ "confidence": 0.74497,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_criteria",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.830948,+ "confidence": 0.830948,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_criteria",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.893034,+ "confidence": 0.893034,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_criteria",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.91249,+ "confidence": 0.91249,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_criteria",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.826767,+ "confidence": 0.826767,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_criteria",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.974913,+ "confidence": 0.974913,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_criteria",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.967734,+ "confidence": 0.967734,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_criteria",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.919886,+ "confidence": 0.919886,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_criteria",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.117075,+ "confidence": 0.882925,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_criteria",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.085332,+ "confidence": 0.914668,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_criteria",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": true,+ "p_true": 0.79433,+ "confidence": 0.79433,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_criteria",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.346036,+ "confidence": 0.653964,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_scenario",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.873428,+ "confidence": 0.873428,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_scenario",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.848264,+ "confidence": 0.848264,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_scenario",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.951039,+ "confidence": 0.951039,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_scenario",+ "text": "快一点",+ "state": "快一点",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.814989,+ "confidence": 0.814989,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_scenario",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.965392,+ "confidence": 0.965392,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_scenario",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.929448,+ "confidence": 0.929448,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_scenario",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.921501,+ "confidence": 0.921501,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_scenario",+ "text": "加速",+ "state": "加速",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.873533,+ "confidence": 0.873533,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_scenario",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.811005,+ "confidence": 0.811005,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_scenario",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.883149,+ "confidence": 0.883149,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_scenario",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.96679,+ "confidence": 0.96679,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_scenario",+ "text": "太慢了",+ "state": "太慢了",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.891167,+ "confidence": 0.891167,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_scenario",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.940289,+ "confidence": 0.940289,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_scenario",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.880876,+ "confidence": 0.880876,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_scenario",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.979531,+ "confidence": 0.979531,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_scenario",+ "text": "再快些",+ "state": "再快些",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.913644,+ "confidence": 0.913644,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_scenario",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.948032,+ "confidence": 0.948032,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_scenario",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.8595,+ "confidence": 0.8595,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_scenario",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.963868,+ "confidence": 0.963868,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_scenario",+ "text": "慢一点",+ "state": "慢一点",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.887192,+ "confidence": 0.887192,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_scenario",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.856617,+ "confidence": 0.856617,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_scenario",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.830039,+ "confidence": 0.830039,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_scenario",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.964707,+ "confidence": 0.964707,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_scenario",+ "text": "太快了",+ "state": "太快了",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.886223,+ "confidence": 0.886223,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_scenario",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.963813,+ "confidence": 0.963813,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_scenario",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.918895,+ "confidence": 0.918895,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_scenario",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.973649,+ "confidence": 0.973649,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_scenario",+ "text": "减速",+ "state": "减速",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.909603,+ "confidence": 0.909603,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_scenario",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.957227,+ "confidence": 0.957227,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_scenario",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.884462,+ "confidence": 0.884462,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_scenario",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.964297,+ "confidence": 0.964297,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_scenario",+ "text": "慢下来",+ "state": "慢下来",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.88703,+ "confidence": 0.88703,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_scenario",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.911988,+ "confidence": 0.911988,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_scenario",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.78218,+ "confidence": 0.78218,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_scenario",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.993699,+ "confidence": 0.993699,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_scenario",+ "text": "停下",+ "state": "停下",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.908587,+ "confidence": 0.908587,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_scenario",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.827109,+ "confidence": 0.827109,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_scenario",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.835131,+ "confidence": 0.835131,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_scenario",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.973559,+ "confidence": 0.973559,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_scenario",+ "text": "别动了",+ "state": "别动了",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.934002,+ "confidence": 0.934002,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_scenario",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.940393,+ "confidence": 0.940393,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_scenario",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.911913,+ "confidence": 0.911913,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_scenario",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.986466,+ "confidence": 0.986466,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_scenario",+ "text": "急停",+ "state": "急停",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.934599,+ "confidence": 0.934599,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_scenario",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.910335,+ "confidence": 0.910335,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_scenario",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.867386,+ "confidence": 0.867386,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_scenario",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.990542,+ "confidence": 0.990542,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_scenario",+ "text": "停下来别动",+ "state": "停下来别动",+ "family": "stop",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.962144,+ "confidence": 0.962144,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_scenario",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.723377,+ "confidence": 0.723377,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_scenario",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.725028,+ "confidence": 0.725028,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_scenario",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.956851,+ "confidence": 0.956851,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_scenario",+ "text": "左转",+ "state": "左转",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.875196,+ "confidence": 0.875196,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_scenario",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.847608,+ "confidence": 0.847608,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_scenario",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.843815,+ "confidence": 0.843815,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_scenario",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.967972,+ "confidence": 0.967972,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_scenario",+ "text": "往左一点",+ "state": "往左一点",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.861928,+ "confidence": 0.861928,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_scenario",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.757204,+ "confidence": 0.757204,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_scenario",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.755771,+ "confidence": 0.755771,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_scenario",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.96655,+ "confidence": 0.96655,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_scenario",+ "text": "向右转",+ "state": "向右转",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.920208,+ "confidence": 0.920208,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_scenario",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.789489,+ "confidence": 0.789489,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_scenario",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.783108,+ "confidence": 0.783108,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_scenario",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.965946,+ "confidence": 0.965946,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_scenario",+ "text": "往右边靠一下",+ "state": "往右边靠一下",+ "family": "direction",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.855464,+ "confidence": 0.855464,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_scenario",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.9697,+ "confidence": 0.9697,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_scenario",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.960589,+ "confidence": 0.960589,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_scenario",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.972534,+ "confidence": 0.972534,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_scenario",+ "text": "别太快了",+ "state": "别太快了",+ "family": "speed",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.843961,+ "confidence": 0.843961,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_scenario",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "wants_faster",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.021209,+ "confidence": 0.978791,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_scenario",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "wants_slower",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.031402,+ "confidence": 0.968598,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_scenario",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "wants_stop",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.189823,+ "confidence": 0.810177,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_scenario",+ "text": "今天天气不错",+ "state": "今天天气不错",+ "family": "other",+ "dimension": "is_command",+ "instructions": "你是一台清洁机器人的控制模块,用户通过语音发出简短指令。这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": false,+ "p_true": 0.472774,+ "confidence": 0.527226,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_json_state",+ "text": "快一点",+ "state": {+ "text": "快一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.92522,+ "confidence": 0.92522,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_json_state",+ "text": "快一点",+ "state": {+ "text": "快一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.972345,+ "confidence": 0.972345,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_json_state",+ "text": "快一点",+ "state": {+ "text": "快一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.88568,+ "confidence": 0.88568,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-001",+ "config": "noul_json_state",+ "text": "快一点",+ "state": {+ "text": "快一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.930986,+ "confidence": 0.930986,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_json_state",+ "text": "加速",+ "state": {+ "text": "加速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.988962,+ "confidence": 0.988962,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_json_state",+ "text": "加速",+ "state": {+ "text": "加速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.961185,+ "confidence": 0.961185,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_json_state",+ "text": "加速",+ "state": {+ "text": "加速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.961192,+ "confidence": 0.961192,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-002",+ "config": "noul_json_state",+ "text": "加速",+ "state": {+ "text": "加速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.981096,+ "confidence": 0.981096,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_json_state",+ "text": "太慢了",+ "state": {+ "text": "太慢了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.876383,+ "confidence": 0.876383,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_json_state",+ "text": "太慢了",+ "state": {+ "text": "太慢了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.97284,+ "confidence": 0.97284,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_json_state",+ "text": "太慢了",+ "state": {+ "text": "太慢了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.934859,+ "confidence": 0.934859,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-003",+ "config": "noul_json_state",+ "text": "太慢了",+ "state": {+ "text": "太慢了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.947968,+ "confidence": 0.947968,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_json_state",+ "text": "再快些",+ "state": {+ "text": "再快些",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.953122,+ "confidence": 0.953122,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_json_state",+ "text": "再快些",+ "state": {+ "text": "再快些",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.972354,+ "confidence": 0.972354,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_json_state",+ "text": "再快些",+ "state": {+ "text": "再快些",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.897289,+ "confidence": 0.897289,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-004",+ "config": "noul_json_state",+ "text": "再快些",+ "state": {+ "text": "再快些",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "faster",+ "pred_bool": true,+ "p_true": 0.93779,+ "confidence": 0.93779,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_json_state",+ "text": "慢一点",+ "state": {+ "text": "慢一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.920528,+ "confidence": 0.920528,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_json_state",+ "text": "慢一点",+ "state": {+ "text": "慢一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.9789,+ "confidence": 0.9789,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_json_state",+ "text": "慢一点",+ "state": {+ "text": "慢一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.896125,+ "confidence": 0.896125,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-005",+ "config": "noul_json_state",+ "text": "慢一点",+ "state": {+ "text": "慢一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.952429,+ "confidence": 0.952429,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_json_state",+ "text": "太快了",+ "state": {+ "text": "太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.891174,+ "confidence": 0.891174,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_json_state",+ "text": "太快了",+ "state": {+ "text": "太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.954349,+ "confidence": 0.954349,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_json_state",+ "text": "太快了",+ "state": {+ "text": "太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.892616,+ "confidence": 0.892616,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-006",+ "config": "noul_json_state",+ "text": "太快了",+ "state": {+ "text": "太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.954126,+ "confidence": 0.954126,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_json_state",+ "text": "减速",+ "state": {+ "text": "减速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.915108,+ "confidence": 0.915108,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_json_state",+ "text": "减速",+ "state": {+ "text": "减速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.967634,+ "confidence": 0.967634,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_json_state",+ "text": "减速",+ "state": {+ "text": "减速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.975214,+ "confidence": 0.975214,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-007",+ "config": "noul_json_state",+ "text": "减速",+ "state": {+ "text": "减速",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.975306,+ "confidence": 0.975306,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_json_state",+ "text": "慢下来",+ "state": {+ "text": "慢下来",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.93728,+ "confidence": 0.93728,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_json_state",+ "text": "慢下来",+ "state": {+ "text": "慢下来",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.983074,+ "confidence": 0.983074,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_json_state",+ "text": "慢下来",+ "state": {+ "text": "慢下来",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.920282,+ "confidence": 0.920282,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-008",+ "config": "noul_json_state",+ "text": "慢下来",+ "state": {+ "text": "慢下来",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.961527,+ "confidence": 0.961527,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_json_state",+ "text": "停下",+ "state": {+ "text": "停下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.935443,+ "confidence": 0.935443,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_json_state",+ "text": "停下",+ "state": {+ "text": "停下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.980399,+ "confidence": 0.980399,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_json_state",+ "text": "停下",+ "state": {+ "text": "停下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.994605,+ "confidence": 0.994605,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-009",+ "config": "noul_json_state",+ "text": "停下",+ "state": {+ "text": "停下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.980708,+ "confidence": 0.980708,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_json_state",+ "text": "别动了",+ "state": {+ "text": "别动了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.806499,+ "confidence": 0.806499,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_json_state",+ "text": "别动了",+ "state": {+ "text": "别动了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.914314,+ "confidence": 0.914314,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_json_state",+ "text": "别动了",+ "state": {+ "text": "别动了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.912872,+ "confidence": 0.912872,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-010",+ "config": "noul_json_state",+ "text": "别动了",+ "state": {+ "text": "别动了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.956662,+ "confidence": 0.956662,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_json_state",+ "text": "急停",+ "state": {+ "text": "急停",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.95566,+ "confidence": 0.95566,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_json_state",+ "text": "急停",+ "state": {+ "text": "急停",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.977647,+ "confidence": 0.977647,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_json_state",+ "text": "急停",+ "state": {+ "text": "急停",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.994262,+ "confidence": 0.994262,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-011",+ "config": "noul_json_state",+ "text": "急停",+ "state": {+ "text": "急停",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.977482,+ "confidence": 0.977482,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_json_state",+ "text": "停下来别动",+ "state": {+ "text": "停下来别动",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.953101,+ "confidence": 0.953101,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_json_state",+ "text": "停下来别动",+ "state": {+ "text": "停下来别动",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.985391,+ "confidence": 0.985391,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_json_state",+ "text": "停下来别动",+ "state": {+ "text": "停下来别动",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.993237,+ "confidence": 0.993237,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-012",+ "config": "noul_json_state",+ "text": "停下来别动",+ "state": {+ "text": "停下来别动",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "stop",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "stop",+ "pred_bool": true,+ "p_true": 0.9812,+ "confidence": 0.9812,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_json_state",+ "text": "左转",+ "state": {+ "text": "左转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.933631,+ "confidence": 0.933631,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_json_state",+ "text": "左转",+ "state": {+ "text": "左转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.9547,+ "confidence": 0.9547,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_json_state",+ "text": "左转",+ "state": {+ "text": "左转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.967582,+ "confidence": 0.967582,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-013",+ "config": "noul_json_state",+ "text": "左转",+ "state": {+ "text": "左转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.962896,+ "confidence": 0.962896,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_json_state",+ "text": "往左一点",+ "state": {+ "text": "往左一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.885004,+ "confidence": 0.885004,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_json_state",+ "text": "往左一点",+ "state": {+ "text": "往左一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.926241,+ "confidence": 0.926241,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_json_state",+ "text": "往左一点",+ "state": {+ "text": "往左一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.939696,+ "confidence": 0.939696,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-014",+ "config": "noul_json_state",+ "text": "往左一点",+ "state": {+ "text": "往左一点",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "left",+ "pred_bool": true,+ "p_true": 0.962878,+ "confidence": 0.962878,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_json_state",+ "text": "向右转",+ "state": {+ "text": "向右转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.925998,+ "confidence": 0.925998,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_json_state",+ "text": "向右转",+ "state": {+ "text": "向右转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.950849,+ "confidence": 0.950849,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_json_state",+ "text": "向右转",+ "state": {+ "text": "向右转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.962676,+ "confidence": 0.962676,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-015",+ "config": "noul_json_state",+ "text": "向右转",+ "state": {+ "text": "向右转",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.97112,+ "confidence": 0.97112,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_json_state",+ "text": "往右边靠一下",+ "state": {+ "text": "往右边靠一下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.899087,+ "confidence": 0.899087,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_json_state",+ "text": "往右边靠一下",+ "state": {+ "text": "往右边靠一下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.95283,+ "confidence": 0.95283,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_json_state",+ "text": "往右边靠一下",+ "state": {+ "text": "往右边靠一下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.962352,+ "confidence": 0.962352,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-016",+ "config": "noul_json_state",+ "text": "往右边靠一下",+ "state": {+ "text": "往右边靠一下",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "direction",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "right",+ "pred_bool": true,+ "p_true": 0.970221,+ "confidence": 0.970221,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_json_state",+ "text": "别太快了",+ "state": {+ "text": "别太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.910267,+ "confidence": 0.910267,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_json_state",+ "text": "别太快了",+ "state": {+ "text": "别太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.986135,+ "confidence": 0.986135,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_json_state",+ "text": "别太快了",+ "state": {+ "text": "别太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.970576,+ "confidence": 0.970576,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-017",+ "config": "noul_json_state",+ "text": "别太快了",+ "state": {+ "text": "别太快了",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "speed",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": true,+ "gold_label": "slower",+ "pred_bool": true,+ "p_true": 0.979213,+ "confidence": 0.979213,+ "correct": 1,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_json_state",+ "text": "今天天气不错",+ "state": {+ "text": "今天天气不错",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "other",+ "dimension": "wants_faster",+ "instructions": "这条指令是否要求加快速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": true,+ "p_true": 0.887423,+ "confidence": 0.887423,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_json_state",+ "text": "今天天气不错",+ "state": {+ "text": "今天天气不错",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "other",+ "dimension": "wants_slower",+ "instructions": "这条指令是否要求放慢速度?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": true,+ "p_true": 0.924268,+ "confidence": 0.924268,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_json_state",+ "text": "今天天气不错",+ "state": {+ "text": "今天天气不错",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "other",+ "dimension": "wants_stop",+ "instructions": "这条指令是否要求完全停止?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": true,+ "p_true": 0.750037,+ "confidence": 0.750037,+ "correct": 0,+ "temperature": 1.0+ },+ {+ "case_id": "zh-cmd-018",+ "config": "noul_json_state",+ "text": "今天天气不错",+ "state": {+ "text": "今天天气不错",+ "source": "voice",+ "device": "vacuum"+ },+ "family": "other",+ "dimension": "is_command",+ "instructions": "这条指令是否是对机器人的运动指令?",+ "options": [+ "false",+ "true"+ ],+ "gold_bool": false,+ "gold_label": "none",+ "pred_bool": true,+ "p_true": 0.937737,+ "confidence": 0.937737,+ "correct": 0,+ "temperature": 1.0+ }+ ]+}diff --git a/research/benchmarks/zh_short_commands/run.py b/research/benchmarks/zh_short_commands/run.pynew file mode 100644index 00000000..cf176152--- /dev/null+++ b/research/benchmarks/zh_short_commands/run.py@@ -0,0 +1,285 @@+"""Run the Chinese short-command benchmark against one Laya checkpoint.++ PYTHONPATH=. python research/benchmarks/zh_short_commands/run.py \+ --checkpoint convaiinnovations/laya --subfolder multilingual \+ --out research/benchmarks/zh_short_commands/results/v1/laya-multilingual++Writes ``<out>/report.json`` with the same four parts as ``research/eval/laya_eval.py``:+``config``, ``report``, ``summary`` and ``cases``. Every number in ``report`` can be+re-derived from ``cases`` alone, with no model and no network.++``--stub`` scores with a fixed pseudo-random vector instead of loading a checkpoint,+which exercises the whole pipeline offline; ``tests/test_benchmark.py`` uses it.+"""+import argparse+import hashlib+import json+import os+import sys+from datetime import datetime, timezone++HERE = os.path.dirname(os.path.abspath(__file__))+ROOT = os.path.dirname(os.path.dirname(os.path.dirname(HERE)))+if ROOT not in sys.path:+ sys.path.insert(0, ROOT)++import prompts # noqa: E402++from research.eval.laya_eval import ( # noqa: E402+ ece, macro_f1, score_cases, softmax_t, summarise, temperature_for,+)++CASES_PATH = os.path.join(HERE, "data", "cases.jsonl")+++def sha256_file(path: str) -> str:+ with open(path, "rb") as f:+ return hashlib.sha256(f.read()).hexdigest()+++def load_cases(path: str = CASES_PATH):+ cases = []+ with open(path, encoding="utf-8") as f:+ for line in f:+ line = line.strip()+ if line:+ cases.append(json.loads(line))+ unknown = sorted({c["gold"] for c in cases} - set(prompts.LABELS))+ if unknown:+ raise ValueError("gold labels outside the policy: %r" % unknown)+ return cases+++def stub_logits(cases):+ """Deterministic pseudo-random marker logits, so the pipeline can run offline."""+ import numpy as np+ rng = np.random.RandomState(13)+ out = []+ for _state, questions in cases:+ for _qid, qdef in questions.items():+ k = len(qdef.get("criteria") or (0, 1))+ out.append(rng.randn(k))+ return out+++def choice_records(config, cases, logits, agent, unclamped):+ """One record per case, in the shape laya_eval.py emits."""+ import numpy as np+ from laya.common import QTYPES++ gold_index = [prompts.LABELS.index(c["gold"]) for c in cases]+ confidences, corrects, preds, records = [], [], [], []+ for i, z in enumerate(logits):+ k = len(z)+ temperature = 1.0 if agent is None else temperature_for(+ agent, QTYPES["choice"], k, unclamped)+ probs = softmax_t(z, temperature)+ pred = int(np.argmax(probs))+ correct = int(pred == gold_index[i])+ confidences.append(float(probs.max()))+ corrects.append(float(correct))+ preds.append(pred)+ records.append({+ "case_id": cases[i]["id"],+ "config": config,+ "text": cases[i]["text"],+ "state": prompts.state_for(config, cases[i]["text"]),+ "family": cases[i]["family"],+ "instructions": prompts.choice_question(config)["intent"]["instructions"],+ "options": list(prompts.CHOICE_CRITERIA),+ "gold_index": gold_index[i],+ "gold_label": cases[i]["gold"],+ "pred_index": pred,+ "pred_label": prompts.LABELS[pred],+ "probability": round(float(probs[pred]), 6),+ "p_gold": round(float(probs[gold_index[i]]), 6),+ "confidence": round(float(probs.max()), 6),+ "correct": correct,+ "temperature": round(float(temperature), 6),+ })+ report = summarise(confidences, corrects, gold_index, preds)+ report["temperature"] = records[0]["temperature"]+ report["mean_confidence"] = round(+ sum(confidences) / len(confidences), 6) if confidences else None+ # Same population std the noul block records, so both shapes read the same way.+ report["std_confidence"] = round(float(np.std(confidences)), 6) if confidences else None+ return {"report": report, "cases": records}+++def noul_records(config, cases, logits, agent, unclamped):+ """One record per (case, dimension); accuracy is per dimension, not top-1."""+ import numpy as np+ from laya.common import QTYPES++ dims = prompts.NOUL_DIMENSIONS+ questions = prompts.noul_questions(config)+ records, per_dim = [], {d: [] for d in dims}+ for i, case in enumerate(cases):+ for j, dim in enumerate(dims):+ z = logits[i * len(dims) + j]+ temperature = 1.0 if agent is None else temperature_for(+ agent, QTYPES["noul"], len(z), unclamped)+ probs = softmax_t(z, temperature)+ p_true = float(probs[1]) # noul scores [false, true] in that order+ # Store the rounded value and derive everything else from it, so the record+ # stays self-consistent: confidence must equal max(p_true, 1 - p_true) of+ # the number that was actually written down, not of the full-precision one.+ p_true = round(p_true, 6)+ gold_bool = bool(prompts.NOUL_GOLD[dim](case["gold"]))+ pred_bool = p_true >= 0.5+ correct = int(pred_bool == gold_bool)+ per_dim[dim].append(correct)+ records.append({+ "case_id": case["id"],+ "config": config,+ "text": case["text"],+ "state": prompts.state_for(config, case["text"]),+ "family": case["family"],+ "dimension": dim,+ "instructions": questions[dim]["instructions"],+ "options": ["false", "true"],+ "gold_bool": gold_bool,+ "gold_label": case["gold"],+ "pred_bool": pred_bool,+ "p_true": p_true,+ "confidence": round(max(p_true, 1.0 - p_true), 6),+ "correct": correct,+ "temperature": round(float(temperature), 6),+ })+ confidences = [r["confidence"] for r in records]+ report = {+ "n_cases": len(cases),+ "n_dimensions": len(dims),+ "accuracy_by_dimension": {+ d: round(sum(v) / len(v), 6) for d, v in per_dim.items() if v+ },+ "accuracy": round(sum(r["correct"] for r in records) / len(records), 6)+ if records else None,+ "mean_confidence": round(sum(confidences) / len(confidences), 6)+ if confidences else None,+ "std_confidence": round(float(np.std(confidences)), 6) if confidences else None,+ "temperature": records[0]["temperature"] if records else None,+ }+ return {"report": report, "cases": records}+++def checkpoint_provenance(args, agent):+ """Best-effort record of what was actually scored: repo, snapshot, weight hash."""+ info = {"checkpoint": args.checkpoint, "subfolder": args.subfolder,+ "device": "stub" if args.stub else str(getattr(agent, "device", None)),+ "stub": bool(args.stub)}+ if agent is None:+ return info+ try:+ import torch+ import transformers+ import laya+ info["torch"] = torch.__version__+ info["transformers"] = transformers.__version__+ info["laya"] = laya.__version__+ except Exception: # pragma: no cover - provenance only+ pass+ subfolder = (args.subfolder + "/") if args.subfolder else ""+ try:+ if os.path.isdir(args.checkpoint):+ weight = os.path.join(args.checkpoint, subfolder, "model.safetensors")+ if os.path.exists(weight):+ info["weight_sha256"] = sha256_file(weight)+ info["weight_bytes"] = os.path.getsize(weight)+ else:+ info["provenance_error"] = "no model.safetensors under %s" % args.checkpoint+ else:+ from huggingface_hub import snapshot_download+ snap = snapshot_download(args.checkpoint, allow_patterns=["*.safetensors"],+ local_files_only=True)+ info["snapshot"] = os.path.basename(snap)+ weight = os.path.join(snap, subfolder + "model.safetensors")+ if os.path.exists(weight):+ info["weight_sha256"] = sha256_file(weight)+ info["weight_bytes"] = os.path.getsize(weight)+ except Exception as exc: # pragma: no cover - provenance only+ info["provenance_error"] = "%s: %s" % (type(exc).__name__, exc)+ return info+++def main(argv=None) -> int:+ parser = argparse.ArgumentParser(prog="zh-short-commands")+ parser.add_argument("--checkpoint", default="convaiinnovations/laya",+ help="checkpoint repo id or local path")+ parser.add_argument("--subfolder", default="multilingual")+ parser.add_argument("--device", default=None, help="cpu, cuda, mps (default: auto)")+ parser.add_argument("--configs", default=",".join(prompts.CONFIGS),+ help="comma list of configs to run")+ parser.add_argument("--out", default=None, help="output directory for report.json")+ parser.add_argument("--unclamped", action="store_true",+ help="score with the checkpoint's raw bucket temperatures")+ parser.add_argument("--stub", action="store_true",+ help="skip the checkpoint and score with a fixed pseudo-random vector")+ args = parser.parse_args(argv)++ configs = [c.strip() for c in args.configs.split(",") if c.strip()]+ for config in configs:+ if config not in prompts.CONFIGS:+ parser.error("unknown config %r; known: %s" % (config, ", ".join(prompts.CONFIGS)))++ cases = load_cases()+ print("cases: %d configs: %s" % (len(cases), ", ".join(configs)))++ agent = None+ if not args.stub:+ import laya+ agent = laya.load(args.checkpoint, subfolder=args.subfolder, device=args.device)++ report, summary = {}, {}+ for config in configs:+ pairs = prompts.cases_for(config, [c["text"] for c in cases])+ logits = stub_logits(pairs) if args.stub else score_cases(agent, pairs)+ if config in prompts.NOUL_CONFIGS:+ out = noul_records(config, cases, logits, agent, args.unclamped)+ else:+ out = choice_records(config, cases, logits, agent, args.unclamped)+ report[config] = out["report"]+ summary[config] = {"accuracy": out["report"].get("accuracy"),+ "mean_confidence": out["report"].get("mean_confidence")}+ line = " %-18s acc=%s" % (config, out["report"].get("accuracy"))+ extras = []+ if out["report"].get("macro_f1") is not None:+ extras.append("macro_f1=%s" % out["report"]["macro_f1"])+ if out["report"].get("ece") is not None:+ extras.append("ece=%s" % out["report"]["ece"])+ if out["report"].get("std_confidence") is not None:+ extras.append("std_conf=%s" % out["report"]["std_confidence"])+ extras.append("mean_conf=%s" % out["report"].get("mean_confidence"))+ print("%s %s" % (line, " ".join(extras)))+ report[config]["cases"] = out["cases"]++ document = {+ "config": {+ "benchmark": "zh_short_commands",+ "cases_file": os.path.relpath(CASES_PATH, ROOT).replace(os.sep, "/"),+ "cases_sha256": sha256_file(CASES_PATH),+ "prompts_sha256": sha256_file(os.path.join(HERE, "prompts.py")),+ "configs": configs,+ "unclamped": bool(args.unclamped),+ "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),+ "checkpoint": checkpoint_provenance(args, agent),+ },+ "report": {c: {k: v for k, v in report[c].items() if k != "cases"} for c in configs},+ "summary": summary,+ "cases": [record for config in configs for record in report[config]["cases"]],+ }++ if args.out:+ os.makedirs(args.out, exist_ok=True)+ target = os.path.join(args.out, "report.json")+ # newline="\n" so the archive is the same bytes on Windows as on Linux; a report+ # written with CRLF makes every later run on another platform look like a rewrite.+ with open(target, "w", encoding="utf-8", newline="\n") as f:+ json.dump(document, f, ensure_ascii=False, indent=2)+ f.write("\n")+ print("wrote %s" % target)+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/research/benchmarks/zh_short_commands/tests/__pycache__/test_audit.cpython-312.pyc b/research/benchmarks/zh_short_commands/tests/__pycache__/test_audit.cpython-312.pycnew file mode 100644index 00000000..23875ffaBinary files /dev/null and b/research/benchmarks/zh_short_commands/tests/__pycache__/test_audit.cpython-312.pyc differdiff --git a/research/benchmarks/zh_short_commands/tests/__pycache__/test_benchmark.cpython-312.pyc b/research/benchmarks/zh_short_commands/tests/__pycache__/test_benchmark.cpython-312.pycnew file mode 100644index 00000000..feecbd90Binary files /dev/null and b/research/benchmarks/zh_short_commands/tests/__pycache__/test_benchmark.cpython-312.pyc differdiff --git a/research/benchmarks/zh_short_commands/tests/test_audit.py b/research/benchmarks/zh_short_commands/tests/test_audit.pynew file mode 100644index 00000000..7fbc45de--- /dev/null+++ b/research/benchmarks/zh_short_commands/tests/test_audit.py@@ -0,0 +1,329 @@+"""Offline tests for audit.py: the archive must stay re-derivable and tamper-evident.++Run from the repository root:++ python -m unittest discover -s research/benchmarks/zh_short_commands/tests -v++Half of these deliberately corrupt a copy of the committed archive and require the+audit to refuse it. An audit nobody has seen fail is not evidence of anything.+"""+import contextlib+import io+import json+import os+import sys+import tempfile+import unittest++HERE = os.path.dirname(os.path.abspath(__file__))+BENCH = os.path.dirname(HERE)+ROOT = os.path.dirname(os.path.dirname(os.path.dirname(BENCH)))+for path in (BENCH, ROOT):+ if path not in sys.path:+ sys.path.insert(0, path)++import audit # noqa: E402+import prompts # noqa: E402+import run as runner # noqa: E402++ARCHIVE = os.path.join(BENCH, "results", "v1", "laya-multilingual")+++class TestFrozenInputs(unittest.TestCase):+ def setUp(self):+ self.cases, self.manifest = audit.load_frozen()++ def test_manifest_hashes_pin_the_current_files(self):+ self.assertEqual(len(self.cases), 18)+ self.assertEqual(audit.sha256_file(os.path.join(BENCH, "data", "cases.jsonl")),+ self.manifest["cases_sha256"])+ self.assertEqual(audit.sha256_file(os.path.join(BENCH, "prompts.py")),+ self.manifest["prompts_sha256"])++ def test_manifest_counts_match_the_cases(self):+ labels = {}+ families = {}+ for case in self.cases:+ labels[case["gold"]] = labels.get(case["gold"], 0) + 1+ families[case["family"]] = families.get(case["family"], 0) + 1+ self.assertEqual(labels, self.manifest["labels"])+ self.assertEqual(families, self.manifest["families"])++ def test_manifest_freezes_the_ladder_this_file_builds(self):+ self.assertEqual(self.manifest["ladder"]["choice"], list(prompts.CHOICE_CONFIGS))+ self.assertEqual(self.manifest["ladder"]["noul"], list(prompts.NOUL_CONFIGS))+++class TestCommittedArchive(unittest.TestCase):+ def test_archive_audits_clean(self):+ result = audit.audit_run(ARCHIVE)+ self.assertEqual(result["cases"], 18)+ self.assertEqual(len(result["configs"]), 7)+ self.assertEqual(result["records"], 342)++ def test_archive_pins_the_weights(self):+ doc = audit.load_json(os.path.join(ARCHIVE, "report.json"))+ self.assertIn("sha256", audit.audit_run(ARCHIVE)["provenance"])+ self.assertEqual(doc["config"]["checkpoint"]["weight_bytes"], 643835514)+ self.assertFalse(doc["config"]["checkpoint"]["stub"])++ def test_stub_archive_has_no_weight_to_pin(self):+ with tempfile.TemporaryDirectory() as tmp:+ with contextlib.redirect_stdout(io.StringIO()):+ runner.main(["--stub", "--out", tmp])+ # The stub archive has to pass the whole audit, provenance included.+ self.assertIn("stub run", audit.audit_run(tmp)["provenance"])+++class TestTamperingIsRefused(unittest.TestCase):+ """Each case mutates one thing in a real archive; the audit must raise."""++ def setUp(self):+ self.tmp = tempfile.TemporaryDirectory()+ self.addCleanup(self.tmp.cleanup)+ self.path = os.path.join(self.tmp.name, "report.json")+ with open(os.path.join(ARCHIVE, "report.json"), encoding="utf-8") as f:+ self.doc = json.load(f)++ def rewrite(self, mutate):+ doc = json.loads(json.dumps(self.doc))+ mutate(doc)+ with open(self.path, "w", encoding="utf-8") as f:+ json.dump(doc, f, ensure_ascii=False)+ return self.tmp.name++ def assert_refused(self, mutate, needle):+ folder = self.rewrite(mutate)+ with self.assertRaises(ValueError) as caught:+ audit.audit_run(folder)+ self.assertIn(needle, str(caught.exception))++ def first(self, doc, config):+ return next(r for r in doc["cases"] if r["config"] == config)++ def test_flipped_correctness(self):+ def mutate(doc):+ record = self.first(doc, "choice_criteria")+ record["correct"] = 1 - record["correct"]+ self.assert_refused(mutate, "contradicts the prediction")++ def test_swapped_state(self):+ self.assert_refused(+ lambda doc: self.first(doc, "choice_json_state").update(state="停下"),+ "state is not what this rung sends")++ def test_swapped_instructions(self):+ def mutate(doc):+ self.first(doc, "noul_scenario")["instructions"] = "这条指令是否要求加快速度?"+ self.assert_refused(mutate, "instructions are not what this rung sends")++ def test_rewritten_gold_label(self):+ self.assert_refused(+ lambda doc: self.first(doc, "choice_criteria").update(gold_label="none"),+ "frozen text or gold label was altered")++ def test_dropped_record(self):+ def mutate(doc):+ doc["cases"] = [r for r in doc["cases"] if r["case_id"] != "zh-cmd-007"]+ self.assert_refused(mutate, "incomplete archive")++ def test_duplicated_record(self):+ def mutate(doc):+ doc["cases"].append(dict(self.first(doc, "noul_plain")))+ doc["cases"][-1]["case_id"] = "zh-cmd-001"+ self.assert_refused(mutate, "duplicate record")++ def test_probability_out_of_range(self):+ def mutate(doc):+ self.first(doc, "noul_criteria")["p_true"] = 1.5+ self.assert_refused(mutate, "p_true is not a probability")++ def test_prediction_that_does_not_follow_from_the_probability(self):+ def mutate(doc):+ record = self.first(doc, "noul_plain")+ record["pred_bool"] = not record["pred_bool"]+ self.assert_refused(mutate, "pred_bool does not follow")++ def test_noul_confidence_that_is_not_the_maximum(self):+ def mutate(doc):+ self.first(doc, "noul_json_state")["confidence"] = 0.5+ self.assert_refused(mutate, "confidence is not the two-way max")++ def test_protocol_from_a_different_prompt_revision(self):+ self.assert_refused(+ lambda doc: doc["config"].update(prompts_sha256="0" * 64),+ "different frozen protocol")++ def test_relabelled_accuracy(self):+ def mutate(doc):+ doc["report"]["choice_scenario"]["accuracy"] = 0.95+ self.assert_refused(mutate, "choice_scenario/accuracy")++ def test_relabelled_macro_f1(self):+ def mutate(doc):+ doc["report"]["choice_criteria"]["macro_f1"] = 0.5+ self.assert_refused(mutate, "choice_criteria/macro_f1")++ def test_relabelled_ece(self):+ def mutate(doc):+ doc["report"]["choice_json_state"]["ece"] = 0.01+ self.assert_refused(mutate, "choice_json_state/ece")++ def test_relabelled_dimension(self):+ def mutate(doc):+ doc["report"]["noul_scenario"]["accuracy_by_dimension"]["wants_stop"] = 0.9+ self.assert_refused(mutate, "noul_scenario/wants_stop")++ def test_summary_that_disagrees_with_its_report(self):+ def mutate(doc):+ doc["summary"]["noul_criteria"]["accuracy"] = 0.9+ self.assert_refused(mutate, "summary/noul_criteria disagrees")++ def test_unexplained_report_field(self):+ self.assert_refused(+ lambda doc: doc["report"]["choice_criteria"].update(score=1.0),+ "unexplained field")++ def test_a_ladder_other_than_the_frozen_one(self):+ def mutate(doc):+ doc["config"]["configs"] = ["choice_criteria"]+ self.assert_refused(mutate, "different ladder")++ def test_missing_weights_for_a_real_run(self):+ self.assert_refused(+ lambda doc: doc["config"]["checkpoint"].pop("weight_sha256"),+ "no weight_sha256")+++class TestMetricsMatchTheHarness(unittest.TestCase):+ """audit.py re-implements the metrics without numpy; pin them to laya_eval's."""++ def setUp(self):+ try:+ import numpy as np+ except ImportError: # pragma: no cover - numpy ships with laya+ self.skipTest("numpy is not installed")+ from research.eval import laya_eval+ self.np = np+ self.harness = laya_eval++ def sample(self, n, seed):+ rng = self.np.random.RandomState(seed)+ logits = rng.randn(n, 6) * 3.0+ probs = self.np.exp(logits - logits.max(axis=1, keepdims=True))+ probs /= probs.sum(axis=1, keepdims=True)+ gold = rng.randint(0, 6, n)+ pred = probs.argmax(axis=1)+ return [float(p) for p in probs.max(axis=1)], [int(p == g) for p, g in zip(pred, gold)], \+ [int(g) for g in gold], [int(p) for p in pred]++ def test_macro_f1_matches(self):+ for seed in range(5):+ confidences, corrects, gold, pred = self.sample(60, seed)+ self.assertAlmostEqual(audit.macro_f1(gold, pred),+ self.harness.macro_f1(gold, pred), places=12)++ def test_ece_matches(self):+ for seed in range(5):+ confidences, corrects, _, _ = self.sample(60, seed)+ self.assertAlmostEqual(audit.ece(confidences, corrects),+ self.harness.ece(confidences, corrects), places=12)++ def test_a_confidence_of_exactly_one_lands_in_the_last_bin(self):+ self.assertAlmostEqual(audit.ece([1.0, 1.0], [1, 1]), 0.0, places=12)+ self.assertAlmostEqual(audit.ece([1.0, 1.0], [0, 1]), 0.5, places=12)++ def test_a_confidence_of_exactly_zero_is_counted(self):+ self.assertAlmostEqual(audit.ece([0.0], [0]), 0.0, places=12)+ self.assertAlmostEqual(audit.ece([0.0], [1]), 1.0, places=12)++ def test_summarise_rounding_matches(self):+ confidences, corrects, gold, pred = self.sample(18, 7)+ block = self.harness.summarise(confidences, corrects, gold, pred)+ self.assertEqual(round(audit.mean(corrects), 4), block["accuracy"])+ self.assertEqual(round(audit.macro_f1(gold, pred), 4), block["macro_f1"])+ self.assertEqual(round(audit.ece(confidences, corrects), 4), block["ece"])+ self.assertEqual(round(audit.mean(confidences), 6),+ round(float(self.np.mean(confidences)), 6))++ def test_std_matches_numpy_population_std(self):+ for seed in range(5):+ confidences, _, _, _ = self.sample(40, seed)+ self.assertAlmostEqual(audit.std(confidences),+ float(self.np.std(confidences)), places=12)+++class TestReadmeQuotesTheArchive(unittest.TestCase):+ """A wrong number in the prose is the failure this benchmark exists to catch.++ Every value is checked for presence, not for position, so reformatting either table is+ fine but editing a number without regenerating the archive is not. It is a net, not a+ proof: a value that also occurs in another row satisfies it.+ """++ def setUp(self):+ self.doc = audit.load_json(os.path.join(ARCHIVE, "report.json"))+ self.pages = {}+ for name in ("README.md", "README.zh-CN.md"):+ with open(os.path.join(BENCH, name), encoding="utf-8") as f:+ self.pages[name] = f.read()++ def test_accuracy_and_confusion_fractions_are_quoted(self):+ for config, block in self.doc["report"].items():+ records = [r for r in self.doc["cases"] if r["config"] == config]+ fraction = "%d/%d" % (sum(r["correct"] for r in records), len(records))+ for name, text in self.pages.items():+ self.assertIn(fraction, text, "%s %s" % (name, config))+ self.assertIn("%.4f" % block["accuracy"], text, "%s %s" % (name, config))++ def test_dimension_accuracies_are_quoted(self):+ for config in prompts.NOUL_CONFIGS:+ for dim, value in self.doc["report"][config]["accuracy_by_dimension"].items():+ for name, text in self.pages.items():+ self.assertIn("%.3f" % value, text, "%s %s/%s" % (name, config, dim))++ def test_choice_macro_f1_and_ece_are_quoted(self):+ for config in prompts.CHOICE_CONFIGS:+ block = self.doc["report"][config]+ for key in ("macro_f1", "ece"):+ for name, text in self.pages.items():+ self.assertIn("%.4f" % block[key], text, "%s %s/%s" % (name, config, key))++ def test_the_weight_hash_is_quoted_whole(self):+ digest = self.doc["config"]["checkpoint"]["weight_sha256"]+ for name, text in self.pages.items():+ self.assertIn(digest, text, name)+++class TestArchiveFilesArePresent(unittest.TestCase):+ def test_expected_files(self):+ for name in ("audit.py", "prompts.py", "run.py", "README.md", "README.zh-CN.md",+ "data/cases.jsonl", "data/manifest.json",+ "results/v1/laya-multilingual/report.json"):+ self.assertTrue(os.path.exists(os.path.join(BENCH, name)), name)++ def test_no_stray_output_committed(self):+ for _dirpath, _dirnames, filenames in os.walk(BENCH):+ _dirnames[:] = [d for d in _dirnames if d != "__pycache__"]+ for name in filenames:+ self.assertFalse(name.endswith((".pyc", ".tmp")), name)++ def test_every_file_is_lf(self):+ """`.gitattributes` keeps these LF, because audit.py pins the sha256 of the raw+ bytes of two of them. This catches a file written on Windows before that, or a+ report.json regenerated without newline="\\n"."""+ for name in ("data/cases.jsonl", "data/manifest.json", "prompts.py", "run.py",+ "audit.py", "README.md", "README.zh-CN.md",+ "results/v1/laya-multilingual/report.json"):+ with open(os.path.join(BENCH, name), "rb") as f:+ self.assertNotIn(b"\r\n", f.read(), name)++ def test_readme_links_the_language_switch(self):+ for name, other in (("README.md", "README.zh-CN.md"),+ ("README.zh-CN.md", "README.md")):+ with open(os.path.join(BENCH, name), encoding="utf-8") as f:+ self.assertIn("(%s)" % other, f.read(), name)+++if __name__ == "__main__":+ unittest.main()diff --git a/research/benchmarks/zh_short_commands/tests/test_benchmark.py b/research/benchmarks/zh_short_commands/tests/test_benchmark.pynew file mode 100644index 00000000..c03a61c0--- /dev/null+++ b/research/benchmarks/zh_short_commands/tests/test_benchmark.py@@ -0,0 +1,264 @@+"""Offline tests for the Chinese short-command benchmark: no checkpoint, no network.++Run from the repository root:++ python -m unittest discover -s research/benchmarks/zh_short_commands/tests -v++The pipeline tests drive ``run.py --stub``, whose scorer is a fixed pseudo-random+vector, so they assert the plumbing (record shapes, counts, re-derivation) rather+than any model behaviour.+"""+import contextlib+import io+import json+import os+import sys+import tempfile+import unittest++HERE = os.path.dirname(os.path.abspath(__file__))+BENCH = os.path.dirname(HERE)+ROOT = os.path.dirname(os.path.dirname(os.path.dirname(BENCH)))+for path in (BENCH, ROOT):+ if path not in sys.path:+ sys.path.insert(0, path)++import prompts # noqa: E402+import run as runner # noqa: E402++JSON_STATE_CONFIGS = ("choice_json_state", "noul_json_state")+SCENARIO_CONFIGS = ("choice_scenario", "noul_scenario")+++def load_cases():+ cases = []+ with open(os.path.join(BENCH, "data", "cases.jsonl"), encoding="utf-8") as f:+ for line in f:+ if line.strip():+ cases.append(json.loads(line))+ return cases+++class TestData(unittest.TestCase):+ def setUp(self):+ self.cases = load_cases()++ def test_cases_parse_and_are_unique(self):+ self.assertTrue(self.cases, "cases.jsonl is empty")+ ids = [c["id"] for c in self.cases]+ self.assertEqual(len(ids), len(set(ids)), "duplicate case ids")++ def test_every_case_has_a_known_gold_label(self):+ for case in self.cases:+ self.assertIn(case["gold"], prompts.LABELS, case["id"])+ self.assertTrue(case["text"].strip(), case["id"])+ self.assertTrue(case["family"].strip(), case["id"])++ def test_every_label_is_covered(self):+ covered = {c["gold"] for c in self.cases}+ self.assertEqual(covered, set(prompts.LABELS), "a label has no case")+++class TestPolicy(unittest.TestCase):+ def test_choice_options_are_the_label_policy(self):+ self.assertEqual(tuple(prompts.CHOICE_CRITERIA), tuple(prompts.LABELS))++ def test_four_noul_dimensions(self):+ self.assertEqual(len(prompts.NOUL_DIMENSIONS), 4)+ self.assertEqual(set(prompts.NOUL_INSTRUCTIONS), set(prompts.NOUL_DIMENSIONS))+ self.assertEqual(set(prompts.NOUL_CRITERIA), set(prompts.NOUL_DIMENSIONS))++ def test_noul_criteria_use_the_two_documented_keys(self):+ """`criteria` on a noul is keyed true/false: those keys ARE the option text."""+ for dim, criteria in prompts.NOUL_CRITERIA.items():+ self.assertEqual(sorted(criteria), ["false", "true"], dim)+ self.assertTrue(criteria["true"].strip() and criteria["false"].strip(), dim)+ self.assertNotEqual(criteria["true"], criteria["false"], dim)++ def test_noul_gold_policy(self):+ """Which dimensions a gold label should fire. `none` fires nothing: it is not+ a command, so all four dimensions are negative for it. Direction labels fire+ `is_command` only — the four dimensions cover speed and stop, not direction,+ which is why task A (six-way choice) exists alongside task B."""+ expectations = {+ "faster": ["wants_faster", "is_command"],+ "slower": ["wants_slower", "is_command"],+ "stop": ["wants_stop", "is_command"],+ "left": ["is_command"],+ "right": ["is_command"],+ "none": [],+ }+ self.assertEqual(sorted(expectations), sorted(prompts.LABELS))+ for gold, expected in expectations.items():+ fired = [d for d in prompts.NOUL_DIMENSIONS if prompts.NOUL_GOLD[d](gold)]+ self.assertEqual(fired, expected, gold)++ def test_only_the_none_label_is_not_a_command(self):+ for gold in prompts.LABELS:+ self.assertEqual(prompts.NOUL_GOLD["is_command"](gold), gold != "none", gold)+++class TestLadder(unittest.TestCase):+ """The ablation rungs must actually differ in the request they build."""++ def setUp(self):+ self.texts = ["快一点", "停下"]++ def test_every_config_builds_one_pair_per_text(self):+ for config in prompts.CONFIGS:+ self.assertEqual(len(prompts.cases_for(config, self.texts)), len(self.texts), config)++ def test_ladder_shape(self):+ self.assertEqual(list(prompts.CHOICE_CONFIGS),+ ["choice_criteria", "choice_scenario", "choice_json_state"])+ self.assertEqual(list(prompts.NOUL_CONFIGS),+ ["noul_plain", "noul_criteria", "noul_scenario", "noul_json_state"])+ self.assertEqual(list(prompts.CONFIGS),+ list(prompts.CHOICE_CONFIGS) + list(prompts.NOUL_CONFIGS))++ def test_noul_plain_omits_criteria_and_the_higher_rungs_add_them(self):+ plain = prompts.noul_questions("noul_plain")+ for dim, qdef in plain.items():+ self.assertNotIn("criteria", qdef, dim)+ for config in ("noul_criteria", "noul_scenario", "noul_json_state"):+ for dim, qdef in prompts.noul_questions(config).items():+ self.assertEqual(sorted(qdef["criteria"]), ["false", "true"], (config, dim))++ def test_scenario_rungs_prepend_the_scenario(self):+ for base_config, scenario_config in (("choice_criteria", "choice_scenario"),+ ("noul_criteria", "noul_scenario")):+ if base_config in prompts.CHOICE_CONFIGS:+ before = prompts.choice_question(base_config)["intent"]["instructions"]+ after = prompts.choice_question(scenario_config)["intent"]["instructions"]+ else:+ before = prompts.noul_questions(base_config)["wants_stop"]["instructions"]+ after = prompts.noul_questions(scenario_config)["wants_stop"]["instructions"]+ self.assertNotEqual(before, after)+ self.assertTrue(after.startswith(prompts.SCENARIO_PREFIX))+ self.assertEqual(after, prompts.SCENARIO_PREFIX + before)++ def test_json_state_is_the_only_wrapped_state(self):+ for config in prompts.CONFIGS:+ state = prompts.state_for(config, "停下")+ if config in JSON_STATE_CONFIGS:+ self.assertIsInstance(state, dict, config)+ self.assertEqual(state["text"], "停下")+ else:+ self.assertEqual(state, "停下", config)++ def test_noul_rungs_ask_the_four_dimensions_in_order(self):+ for config in prompts.NOUL_CONFIGS:+ questions = prompts.cases_for(config, self.texts)[0][1]+ self.assertEqual(tuple(questions), prompts.NOUL_DIMENSIONS, config)+ for qdef in questions.values():+ self.assertEqual(qdef["type"], "noul", config)++ def test_choice_rungs_ask_one_six_option_question(self):+ for config in prompts.CHOICE_CONFIGS:+ questions = prompts.cases_for(config, self.texts)[0][1]+ self.assertEqual(len(questions), 1, config)+ qdef = next(iter(questions.values()))+ self.assertEqual(qdef["type"], "choice")+ self.assertEqual(len(qdef["criteria"]), len(prompts.LABELS))++ def test_unknown_config_is_rejected(self):+ with self.assertRaises(ValueError):+ prompts.cases_for("nope", self.texts)+++class TestStubRun(unittest.TestCase):+ """Drive the whole runner offline and check what it wrote."""++ @classmethod+ def setUpClass(cls):+ cls._tmp = tempfile.TemporaryDirectory()+ # The runner prints its own table; keep the test log clean.+ with contextlib.redirect_stdout(io.StringIO()):+ rc = runner.main(["--stub", "--out", cls._tmp.name])+ if rc != 0:+ raise AssertionError("stub run exited %d" % rc)+ with open(os.path.join(cls._tmp.name, "report.json"), encoding="utf-8") as f:+ cls.doc = json.load(f)+ cls.cases = load_cases()++ @classmethod+ def tearDownClass(cls):+ cls._tmp.cleanup()++ def test_document_parts(self):+ self.assertEqual(sorted(self.doc), ["cases", "config", "report", "summary"])+ for config in prompts.CONFIGS:+ self.assertIn(config, self.doc["report"])+ self.assertIn(config, self.doc["summary"])++ def test_record_counts(self):+ n = len(self.cases)+ per_config = {}+ for record in self.doc["cases"]:+ per_config[record["config"]] = per_config.get(record["config"], 0) + 1+ for config in prompts.NOUL_CONFIGS:+ self.assertEqual(per_config[config], n * len(prompts.NOUL_DIMENSIONS), config)+ for config in prompts.CHOICE_CONFIGS:+ self.assertEqual(per_config[config], n, config)++ def test_every_case_is_recorded_once_per_role(self):+ choice = [c for c in self.doc["cases"] if c["config"] == "choice_criteria"]+ self.assertEqual([c["case_id"] for c in choice], [c["id"] for c in self.cases])++ def test_probabilities_are_finite_and_normalised(self):+ for record in self.doc["cases"]:+ if "p_gold" in record:+ for key in ("probability", "p_gold", "confidence"):+ self.assertGreaterEqual(record[key], 0.0, record["case_id"])+ self.assertLessEqual(record[key], 1.0, record["case_id"])+ self.assertEqual(record["pred_label"], prompts.LABELS[record["pred_index"]])+ else:+ self.assertGreaterEqual(record["p_true"], 0.0)+ self.assertLessEqual(record["p_true"], 1.0)+ self.assertEqual(record["pred_bool"], record["p_true"] >= 0.5)++ def test_every_record_carries_the_request_it_was_asked(self):+ """The record must say what was actually sent, not just what was scored."""+ by_id = {c["id"]: c for c in self.cases}+ for record in self.doc["cases"]:+ case = by_id[record["case_id"]]+ self.assertEqual(record["text"], case["text"], record["case_id"])+ self.assertEqual(record["state"],+ prompts.state_for(record["config"], case["text"]),+ record["case_id"])+ self.assertEqual(record["gold_label"], case["gold"], record["case_id"])+ if "dimension" in record:+ expected = prompts.noul_questions(record["config"])[record["dimension"]]["instructions"]+ else:+ expected = prompts.choice_question(record["config"])["intent"]["instructions"]+ self.assertEqual(record["instructions"], expected, record["case_id"])++ def test_accuracy_re_derives_from_the_cases(self):+ """The laya_eval property: report can be recomputed from cases alone."""+ for config in prompts.CHOICE_CONFIGS:+ records = [c for c in self.doc["cases"] if c["config"] == config]+ acc = sum(r["correct"] for r in records) / len(records)+ self.assertAlmostEqual(+ round(acc, 4), round(self.doc["report"][config]["accuracy"], 4), places=4)++ def test_noul_accuracy_re_derives_per_dimension(self):+ for config in prompts.NOUL_CONFIGS:+ records = [c for c in self.doc["cases"] if c["config"] == config]+ for dim, value in self.doc["report"][config]["accuracy_by_dimension"].items():+ subset = [r for r in records if r["dimension"] == dim]+ self.assertAlmostEqual(+ round(sum(r["correct"] for r in subset) / len(subset), 4),+ round(value, 4), places=4, msg="%s/%s" % (config, dim))++ def test_hashes_of_the_frozen_inputs_are_recorded(self):+ self.assertEqual(self.doc["config"]["cases_sha256"],+ runner.sha256_file(os.path.join(BENCH, "data", "cases.jsonl")))+ self.assertEqual(self.doc["config"]["prompts_sha256"],+ runner.sha256_file(os.path.join(BENCH, "prompts.py")))++ def test_stub_run_records_no_checkpoint(self):+ self.assertTrue(self.doc["config"]["checkpoint"]["stub"])+++if __name__ == "__main__":+ unittest.main()diff --git a/research/eval/__pycache__/__init__.cpython-312.pyc b/research/eval/__pycache__/__init__.cpython-312.pycnew file mode 100644index 00000000..1bfe9807Binary files /dev/null and b/research/eval/__pycache__/__init__.cpython-312.pyc differdiff --git a/research/eval/__pycache__/laya_eval.cpython-312.pyc b/research/eval/__pycache__/laya_eval.cpython-312.pycnew file mode 100644index 00000000..f071903aBinary files /dev/null and b/research/eval/__pycache__/laya_eval.cpython-312.pyc differ