Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(benchmarks): add Chinese reliability evaluation
Fanrito7c8b8316410d
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 39843b20..ef8a8415 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -148,6 +148,7 @@ jobs:python tests/test_langchain.pypython tests/test_portability.pypython tests/test_training.py+ python -m unittest discover -s tests -p 'test_zh_*.py' -vpython tests/test_example_server_limits.pypython tests/test_blank_lang_routing.pypython tests/test_export_onnx_safety.py@@ -355,9 +356,9 @@ jobs:python-version: "3.11"- run: pip install ruff- name: Ruff- run: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120+ run: ruff check laya/ research/benchmarks/zh_reliability tests/test_zh_*.py --select=E9,F63,F7,F82,F401,F811 --line-length=120- name: Byte-compile every module- run: python -m compileall -q laya/ tests/+ run: python -m compileall -q laya/ tests/ research/benchmarks/zh_reliabilitytypescript-sdk:name: TypeScript SDK (node${{ matrix.node }})diff --git a/.gitignore b/.gitignoreindex 75442a6f..42c9377a 100644--- a/.gitignore+++ b/.gitignore@@ -80,3 +80,6 @@ research/*_benchmark_results.json# local checkpoint weights, kept out of the tree# (fetch them with: python verify/checkpoints.py --fetch)models/++# Local Chinese reliability runs (model copies, data, diagnostics)+/runs/diff --git a/research/benchmarks/zh_reliability/README.md b/research/benchmarks/zh_reliability/README.mdnew file mode 100644index 00000000..45f6aa7d--- /dev/null+++ b/research/benchmarks/zh_reliability/README.md@@ -0,0 +1,183 @@+# Chinese alarm-intent evaluation and temperature calibration++This optional research benchmark measures `choice` and `noul` decisions through Laya's+existing local Agent, sequence builder and multilingual checkpoint. E0 uses the checkpoint's+runtime temperatures; E1 fits one bounded scalar temperature per question type on a separate+calibration partition. It does not update model weights or change the public runtime API.++The contribution is deliberately narrower than a general Chinese benchmark: a three-intent+MASSIVE alarm subset, grouping checks, auditable calibration/test separation, and paired+before/after reliability metrics. It complements the broader Chinese evaluation in+[#557](https://github.com/NandhaKishorM/laya/pull/557), short-command prompt ablations in+[#364](https://github.com/NandhaKishorM/laya/pull/364), and the generic evaluation harness in+[#382](https://github.com/NandhaKishorM/laya/pull/382).++## Inspect and recompute the historical result on CPU++[The result table](results/massive_alarm_2026-09-24.md) is generated from+[compact decision evidence](results/massive_alarm_2026-09-24.json). The JSON contains 128+calibration and 186 test records with source IDs, group IDs, labels, option order, raw logits,+actual temperatures and source hashes. Test records retain both runs' raw logits so the verifier+checks that calibration changed only temperature. It includes original file hashes, model+identity, selected dependency versions and numerical summaries. Source text, weights, local+absolute paths, GPU UUIDs and credentials are omitted.++**These are historical measurements from 2026-09-24**, executed on base commit `23a1752`+with the contribution subsequently recorded in `37cc584`. The execution worktree was dirty;+its recorded source-file hashes identify the measured implementation. The results were not+measured against the later upstream rebase or this evaluation-only refactor.++From the repository root, with the project's Python dependencies installed:++```bash+python -m research.benchmarks.zh_reliability.aggregate_massive \+ --artifact research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json \+ --report /tmp/massive-alarm-result.md+```++This performs no inference or downloads and needs no CUDA/model checkpoint. It verifies record+integrity, partition/group/source pairing and original temperatures; refits temperatures from+calibration logits; rebuilds probabilities and metrics; recomputes 1,000 paired group bootstrap+draws; and compares all numerical summaries with the saved values. Hashes bind the archived+artifacts; they are not independent authentication of data provenance. Auditing original text+and annotation declarations additionally requires the prepared public dataset.++| Condition | Accuracy | Macro F1 | NLL | Brier | ECE |+|---|---:|---:|---:|---:|---:|+| E0 original temperatures | 0.532258 | 0.440840 | 1.383747 | 0.730071 | 0.366488 |+| E1 calibrated temperatures | 0.532258 | 0.440840 | 0.785033 | 0.506898 | 0.197529 |++Accuracy and F1 are unchanged because positive scalar temperature preserves argmax. The NLL+difference is −0.598713, with a group bootstrap 95% interval [−0.865774, −0.369099]. This+supports a calibration result on this subset, not general Chinese capability improvement.+Choice temperature is 3.135504; noul reaches the permitted upper bound of 5.0. Risk/coverage+does not improve at every coverage.++## Prepare public data++Amazon's MASSIVE 1.1 is human-localized virtual-assistant text, licensed CC BY 4.0+([pinned publisher notice](https://github.com/alexa/massive/blob/f966f21846043aabef9b0f974fa7970027f43738/NOTICE.md)).+It is not naturally occurring Chinese customer-service logs. The derived tasks use the original+`alarm_set`, `alarm_query`, and `alarm_remove` labels: choice selects the operation; noul asks+whether the intent is setting an alarm. No paraphrases or new semantic labels are generated.++```bash+# Optional dependency for the small Parquet download route only.+uv pip install --python .venv/bin/python pyarrow+.venv/bin/python -m research.benchmarks.zh_reliability.prepare_massive \+ --download-parquet --run-dir runs/massive-alarm-data+.venv/bin/python -m research.benchmarks.zh_reliability import-data \+ --data runs/massive-alarm-data/data.jsonl --formal \+ --split-manifest runs/massive-alarm-data/split_manifest.json \+ --run-dir runs/massive-import+```++The Parquet route downloads six Chinese/English files, about 2 MB, from the publisher's+`AmazonScience/massive` repository at conversion revision+`ed58ac423a2f4121720918bf5301577edce4ffd3`. It executes no dataset loading script. Embedded+ClassLabel indices are decoded, and column-oriented annotation lists are transposed without+changing text or labels. Saved manifests identify transport, decoding rules, reader version,+file hashes and pinned attribution/legal text. Decoded source-row hashes are not archive-byte+hashes. Alternatively, `--download` uses the publisher's approximately 40 MB archive and+records its hash; only selected regular members are read. `--source-dir PATH` converts an+already downloaded, hash-checked source directory offline. Every output directory must be new.++The fixed admission policy requires at least three distinct upstream reviewers whose intent+scores all equal 1. It does not imply unanimous grammar/spelling/slot approval. Provenance+retains these upstream judgments and `human_verified=false`; no independent local review is+claimed. Other records remain in `diagnostic.jsonl`, with reasons. No model scores determine+admission, grouping or labels.++Connected groups link identical source IDs, normalized Chinese text and delexicalized English+source templates. Cross-partition groups keep the highest-priority original partition+(test > dev > train); lower-priority members are quarantined. Conflicting-label groups are+quarantined. Calibration uses 20% of the remaining training groups with seed `20260924`.+Grouping is heuristic and can miss semantic dependencies.++The retained 471 source utterances generate 942 correlated decisions. Fixed partition decision+counts are train 522, dev 106, calibration 128 and test 186, with 247/53/62/89 groups.+**This evaluation consumes only calibration and test**; the other source partitions remain+reserved. Test has 93 utterances, each represented by both question types.++## Run a new evaluation++Use a local copy of `convaiinnovations/laya-multilingual` revision+`e4e9ddf21a7b1903b7acffd8814ad4307bf63a67`, including the recorded `zh_lineage.json` identifying+that original revision. The script verifies fixed data/group counts and records paths/hashes+before predictions. Set `LAYA_ZH_PYTHON` if the interpreter is not `.venv/bin/python`.++On a clean clone, explicitly download that pinned revision and record its original-checkpoint+lineage. This downloads only runtime files to the Hugging Face cache, copies them into a new+local model directory, and performs no inference. It does not fall back to `main`:++```bash+.venv/bin/python - <<'PY_DOWNLOAD'+from pathlib import Path+from research.benchmarks.zh_reliability.data import atomic_json+from research.benchmarks.zh_reliability.runtime import resolve_model++revision = 'e4e9ddf21a7b1903b7acffd8814ad4307bf63a67'+work = Path('runs/massive-model-download')+work.mkdir(parents=True) # Refuse an existing output directory.+model, identity = resolve_model('convaiinnovations/laya-multilingual', revision, False, work)+if identity['revision'] != revision:+ raise ValueError('download resolved to an unexpected revision')+atomic_json(model / 'zh_lineage.json', {+ 'model_id': identity['model_id'], 'base_revision': revision,+ 'trained': False, 'weights_hash': identity['files']['model.safetensors']})+print(model)+PY_DOWNLOAD+```++The resulting path is `runs/massive-model-download/model_input`:++```bash+bash research/benchmarks/zh_reliability/reproduce_massive.sh \+ runs/new-massive-calibration runs/massive-model-download/model_input \+ runs/massive-alarm-data/data.jsonl runs/massive-alarm-data/split_manifest.json+```++The fixed recipe runs only `calibrate`, baseline `eval`, calibrated `eval`, and CPU aggregation.+It defaults to process-visible `cuda:0`, FP16, max_len=512, and head_max_len=192. Set+`LAYA_ZH_DEVICE` to a visible device selector, for example `LAYA_ZH_DEVICE="RTX 2080 Ti"`,+to use another device. The selected value is recorded before predictions in the generated+protocol; the historical measurements used RTX 2080 Ti. Selectors refer to visible CUDA devices; no device mask or driver is changed. Downloading models is+not implicit. A failed stage returns nonzero and records its stage in a FAILED manifest.+Run directories hold local predictions, environment, split, calibration and model identities;+private tokenizer copies protect the input checkpoint from compatibility patches.++Standalone commands are available with `python -m research.benchmarks.zh_reliability --help`.+`eval` and `calibrate` require consistent data, `--formal`, `--split-manifest`, model, dtype and+length settings. Calibration defaults to at least 30 samples per type. Artifact validation+rejects a different model/data/split or unapproved smoke-only fit. A new completed recipe can+be checked and its report rebuilt without GPU inference:++```bash+python -m research.benchmarks.zh_reliability.aggregate_massive runs/new-massive-calibration+```++The short agent-authored examples in `fixtures.jsonl` remain unverified regression diagnostics;+passing fixture tests is not a measured public-data result.++## Definitions and limits++- Only valid decision logits are used, never action outputs or rounded public probabilities.+ Noul option order is `[false,true]`; confidence is max(p). Existing temperatures are not+ applied twice. The bounded fit minimizes calibration NLL in the runtime range [0.5,5.0].+- NLL uses float64 log-softmax. Brier is the mean sum of squared class errors. Macro F1 averages+ task-local macro F1 over the full option set. ECE has 15 equal bins, the last including 1.+ Risk/coverage breaks confidence ties by lexical sample ID. Type/task/truncation slices and+ official-Agent probability parity are retained by standalone evaluation.+- Bootstrap resamples the 89 test groups together across both conditions and question types.+ Its 1,000 draws use seed `20260924`. Intervals condition on this checkpoint and fitted+ temperatures, omit calibration uncertainty, and make no corrected multiple-testing claim.+- This is one task subset and one split/seed. Checkpoint exposure to MASSIVE/SLURP is unknown.+ It does not establish full MASSIVE performance, general Chinese improvement, or unchanged+ behavior in other domains. Action outputs are outside scope.+- Historical test sequences were at most 60 tokens with no truncation. max_len=512 is a limit,+ not a 512-token stress test. Optional timings synchronize CUDA, distinguish loading and+ inference memory, and report single-decision batch p50/p95; GPUs are not exclusively reserved.++Offline checks use `python -m unittest discover -s tests -p 'test_zh_*.py' -v`. The archive+recomputation command above additionally verifies the submitted numerical evidence. Repository+Ruff and compileall gates apply; no hosted inference service or new runtime dependency is added.diff --git a/research/benchmarks/zh_reliability/__init__.py b/research/benchmarks/zh_reliability/__init__.pynew file mode 100644index 00000000..b634e621--- /dev/null+++ b/research/benchmarks/zh_reliability/__init__.py@@ -0,0 +1 @@+"""Opt-in local Chinese decision evaluation and calibration; not a runtime API."""diff --git a/research/benchmarks/zh_reliability/__main__.py b/research/benchmarks/zh_reliability/__main__.pynew file mode 100644index 00000000..c7568bf7--- /dev/null+++ b/research/benchmarks/zh_reliability/__main__.py@@ -0,0 +1,297 @@+"""Reproducible opt-in experiments: python -m research.benchmarks.zh_reliability --help."""+import argparse+from datetime import datetime, timezone+import json+from pathlib import Path+import sys+import traceback+import uuid++from .data import FIXTURES, SPLITS, atomic_json, partition, read_data+from .runtime import environment, select_device+++def parser():+ p = argparse.ArgumentParser(description=__doc__)+ sub = p.add_subparsers(dest='command', required=True)+ for name in ('check-env', 'import-data', 'smoke', 'eval', 'calibrate', 'report'):+ s = sub.add_parser(name)+ s.add_argument('--run-dir', type=Path, help='fresh output directory; existing runs are never overwritten')+ s.add_argument('--data', type=Path, default=FIXTURES)+ s.add_argument('--formal', action='store_true', help='require licensed data, documented human annotation and fixed splits')+ s.add_argument('--split-manifest', type=Path)+ s.add_argument('--seed', type=int, default=20260924)+ if name in ('smoke', 'eval', 'calibrate'):+ s.add_argument('--model', default='convaiinnovations/laya-multilingual')+ s.add_argument('--revision', default='main', help='resolved to immutable HF SHA before download')+ s.add_argument('--download', action='store_true', help='allow target-model-only download; default offline')+ s.add_argument('--device', default='cuda:0', help='unique visible GPU name, UUID or process cuda:index')+ s.add_argument('--dtype', choices=('fp16', 'fp32'), default='fp16')+ s.add_argument('--max-len', type=int, default=256 if name == 'smoke' else 512)+ s.add_argument('--head-max-len', type=int, default=192)+ if name in ('eval', 'smoke'):+ s.add_argument('--warmup', type=int, default=10)+ s.add_argument('--batches', type=int, default=30)+ if name == 'eval':+ s.add_argument('--split', choices=SPLITS, default='test')+ s.add_argument('--calibration', type=Path)+ s.add_argument('--allow-smoke-calibration', action='store_true')+ s.add_argument('--measure', action='store_true')+ if name in ('calibrate', 'smoke'):+ s.add_argument('--minimum', type=int, default=1 if name == 'smoke' else 30)+ if name == 'calibrate':+ s.add_argument('--smoke-only', action='store_true')+ if name == 'smoke':+ s.add_argument('--gpu', action='store_true', help='opt in to actual Laya inference, calibration and precision measurements')+ if name == 'report':+ s.add_argument('--source', type=Path, required=True, help='existing result directory (read only)')+ return p+++def write_predictions(path, rows):+ # Runs are private until manifest finalization; each row retains raw unrounded decision logits.+ with Path(path).open('w', encoding='utf-8') as f:+ for row in rows:+ f.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + '\n')+++def report(run, destination=None):+ run = Path(run)+ manifest = json.loads((run / 'manifest.json').read_text())+ metrics = json.loads((run / 'metrics.json').read_text())+ text = '# Chinese decision reliability run\n\n'+ text += f"Status: **{manifest['status']}**. Run: `{run.name}`.\n\n"+ data_info = manifest.get('data_provenance', manifest.get('source_manifest', {}).get('data_provenance', {}))+ if 'public_human_annotated' in data_info.get('kinds', []):+ text += 'Public human-localized data with retained upstream annotation evidence; no independent local human review. '+ text += 'This derived task/subset is not the full source benchmark or evidence of general Chinese improvement. '+ elif data_info.get('formal'):+ text += 'Source/annotation declarations and fixed splits were validated; this does not establish representative sampling or independent label correctness. '+ else:+ text += 'Agent-authored fixtures are unverified diagnostics, not a standard evaluation or evidence of general Chinese improvement. '+ text += 'Action outputs are outside this evaluation. No test-set temperature selection.\n\n'+ text += 'Full slices, per-sample evidence and configuration: `metrics.json`, `predictions.jsonl`, '+ text += '`split_manifest.json`, `environment.json`, `manifest.json`.\n\n'+ conditions = [(k, v['overall']) for k, v in metrics.items() if isinstance(v, dict) and 'overall' in v]+ if 'overall' in metrics:+ conditions = [('evaluated split', metrics['overall'])]+ if conditions:+ text += '| Condition | n | Accuracy | Task macro F1 | NLL | Brier | ECE |\n'+ text += '|---|---:|---:|---:|---:|---:|---:|\n'+ for k, m in conditions:+ values = [str(m['n'])] + [f"{m[x]:.6f}" if m[x] is not None else 'null'+ for x in ('accuracy', 'macro_f1', 'nll', 'brier', 'ece')]+ text += '| ' + k + ' | ' + ' | '.join(values) + ' |\n'+ text += '\n'+ timing = [(k, v) for k, v in metrics.items() if isinstance(v, dict) and 'end_to_end' in v]+ if timing:+ text += '| Device / dtype | E2E batch p50 ms | E2E batch p95 ms | state/s | Model p50 ms | Peak allocated MiB |\n'+ text += '|---|---:|---:|---:|---:|---:|\n'+ for k, v in timing:+ e, m = v['end_to_end'], v['model_stage']+ text += (f"| {k} | {e['p50_batch_ms']:.3f} | {e['p95_batch_ms']:.3f} | "+ f"{e['state_per_s']:.2f} | {m['p50_batch_ms']:.3f} | {e['max_memory_allocated']/2**20:.1f} |\n")+ text += '\nLatency is per batch (one state/decision); model stage excludes H2D/D2H.\n\n'+ details = {k: v for k, v in metrics.items() if k not in {x[0] for x in conditions + timing}}+ text += '## Verification and numerical diagnostics\n\n```json\n'+ text += json.dumps(details, ensure_ascii=False, indent=2, allow_nan=False) + '\n```\n'+ text += '\n## Execution\n\n```json\n' + json.dumps(manifest, ensure_ascii=False, indent=2) + '\n```\n'+ target = Path(destination) if destination else run / 'report.md'+ target.write_text(text, encoding='utf-8')+++def load_adapter(args, env, run):+ import torch+ from laya.agent import Agent+ from .runtime import Adapter, ensure_device, resolve_model+ device = select_device(args.device, env)+ model_input, identity = resolve_model(args.model, args.revision, not args.download, run)+ torch.cuda.synchronize(device)+ torch.cuda.reset_peak_memory_stats(device)+ agent = Agent(str(model_input), device=device, fast=False, compile=False)+ ensure_device(agent, device)+ atomic_json(Path(run) / 'loading_memory.json', {'device': device,+ 'max_memory_allocated': torch.cuda.max_memory_allocated(device),+ 'max_memory_reserved': torch.cuda.max_memory_reserved(device), 'scope': 'model loading only'})+ return Adapter(agent, identity, args.max_len, args.head_max_len, args.dtype), model_input+++def evaluate(adapter, rows, split, calibration=None, smoke=False, selection='test'):+ import copy+ from .calibration import validate_calibration+ from .metrics import summarize+ adapter.agent.lang_temperatures = (validate_calibration(calibration, adapter.identity, split, smoke)+ if calibration else copy.deepcopy(adapter.original['lang_temperatures']))+ condition = 'E1' if calibration else 'E0'+ selected = [r for r in rows if split['assignments'][r['id']] == selection]+ predictions = adapter.predict(selected, condition, split)+ return predictions, summarize(predictions)+++def gpu_smoke(args, env, run, rows, split, manifest):+ import copy+ import numpy as np+ import torch+ from .calibration import calibrate+ from .metrics import paired_bootstrap+ from .runtime import measure+ device = select_device(args.device, env)+ results = {'devices': {args.device: {'status': 'AVAILABLE', 'device': device}}}+ manifest['gpu_experiment'] = 'PARTIAL'+ base_dir = run / 'base'+ base_dir.mkdir()+ adapter, _ = load_adapter(args, env, base_dir)+ manifest['model'] = adapter.identity+ subset = [+ r for kind in ('choice', 'noul') for r in [x for x in rows if x['question']['type'] == kind][:4]]+ results['official_parity'] = adapter.parity(subset)+ cal_rows = [r for r in rows if split['assignments'][r['id']] == 'calibration']+ base_preds, results['E0'] = evaluate(adapter, rows, split)+ cal_preds = adapter.predict(cal_rows, 'E0', split)+ artifact = calibrate(cal_preds, adapter.identity, split, adapter.original,+ run / 'calibration.json', args.minimum, True)+ results['calibration_E1'] = artifact['fitted']+ if any(x['status'] == 'SKIPPED' for x in artifact['fitted'].values()):+ manifest['status'] = 'PARTIAL'+ fitted_preds, results['E1'] = evaluate(adapter, rows, split, artifact, True)+ all_predictions = base_preds + fitted_preds + cal_preds+ results['E1-E0'] = (paired_bootstrap(base_preds, fitted_preds) if args.formal+ else {'status': 'SKIPPED', 'reason': 'unverified fixtures cannot support population inference'})+ manifest['completed'].append('E0/E1')+ write_predictions(run / 'predictions.jsonl', all_predictions)+ atomic_json(run / 'metrics.json', results)+ atomic_json(run / 'manifest.json', manifest)+ # Precision comparisons use the original checkpoint temperatures and fixed samples.+ adapter.agent.lang_temperatures = copy.deepcopy(adapter.original['lang_temperatures'])+ numeric = {}+ for dtype in ('fp16', 'fp32'):+ key = args.device + '/' + dtype+ adapter.dtype = dtype+ adapter.agent.amp_enabled = dtype == 'fp16'+ adapter.agent.dtype = torch.float16 if dtype == 'fp16' else torch.float32+ numeric[dtype] = adapter.predict(subset, 'numeric-' + dtype, split)+ results[key] = measure(adapter, subset, args.warmup, args.batches)+ all_predictions += numeric[dtype]+ write_predictions(run / 'predictions.jsonl', all_predictions)+ atomic_json(run / 'metrics.json', results)+ delta = np.concatenate([np.abs(np.array(x['probabilities']) - y['probabilities'])+ for x, y in zip(numeric['fp16'], numeric['fp32'])])+ results['numerical_comparisons'] = {args.device + '/fp16 vs ' + args.device + '/fp32': {+ 'n': len(subset), 'argmax_agreement': float(np.mean([x['predicted'] == y['predicted']+ for x, y in zip(numeric['fp16'], numeric['fp32'])])),+ 'probability_abs_mean': float(delta.mean()), 'probability_abs_max': float(delta.max()),+ 'probability_abs_p50_p95_p99': np.quantile(delta, [.5, .95, .99]).tolist()}}+ manifest['completed'].append('FP16/FP32 measurements on requested device')+ return results+++def main(argv=None):+ args = parser().parse_args(argv)+ run = args.run_dir or Path('runs') / (datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ') + '-' + uuid.uuid4().hex[:8])+ if run.exists():+ print('refusing existing run directory: ' + str(run), file=sys.stderr)+ return 2+ run.mkdir(parents=True)+ manifest = {'run_id': run.name, 'status': 'RUNNING', 'command': args.command, 'completed': [],+ 'config': {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()},+ 'smoke_only': args.command == 'smoke' or getattr(args, 'smoke_only', False),+ 'gpu_experiment': 'NOT_RUN'}+ atomic_json(run / 'manifest.json', manifest)+ atomic_json(run / 'metrics.json', {'status': 'NOT_RUN'})+ write_predictions(run / 'predictions.jsonl', [])+ stage = 'environment'+ try:+ env = environment()+ atomic_json(run / 'environment.json', env)+ if args.command == 'report':+ import shutil+ stage = 'report'+ manifest['source_manifest'] = json.loads((args.source / 'manifest.json').read_text())+ for name in ('metrics.json', 'predictions.jsonl', 'split_manifest.json'):+ shutil.copyfile(args.source / name, run / name)+ shutil.copyfile(args.source / 'environment.json', run / 'source_environment.json')+ manifest['completed'].append('report generated from ' + str(args.source))+ else:+ stage = 'data'+ rows = read_data(args.data, args.formal)+ manifest['data_provenance'] = {+ 'formal': args.formal,+ 'kinds': sorted({r['provenance']['kind'] for r in rows}),+ 'licenses': sorted({r['provenance'].get('license', 'unspecified') for r in rows}),+ 'human_verified_rows': sum(r['provenance']['human_verified'] for r in rows),+ 'upstream_review_rows': sum('upstream_annotation' in r['provenance'] for r in rows)}+ split = partition(rows, args.split_manifest, args.seed, args.formal)+ atomic_json(run / 'split_manifest.json', split)+ manifest['completed'].append('data/schema/group partitions')+ atomic_json(run / 'metrics.json', {'data_counts': split['counts'], 'groups': split['group_counts']})+ if args.command == 'check-env':+ manifest['completed'].append('CUDA execution probes')+ if env['gpu_status'] == 'FAILED':+ raise RuntimeError('CUDA execution probe failed; see environment.json')+ elif args.command == 'smoke' and not args.gpu:+ import subprocess+ stage = 'offline tests'+ completed = subprocess.run([sys.executable, '-m', 'unittest', 'discover', '-s', 'tests',+ '-p', 'test_zh_*.py', '-v'], capture_output=True, text=True)+ (run / 'offline-tests.log').write_text(completed.stdout + completed.stderr)+ if completed.returncode:+ raise RuntimeError('offline tests failed; see offline-tests.log')+ atomic_json(run / 'metrics.json', {'offline_tests': 'SUCCESS', 'gpu': 'NOT_RUN',+ 'reason': 'default smoke is offline; opt in using --gpu', 'data_counts': split['counts']})+ manifest['completed'].append('offline tests (tiny/mock inference only)')+ manifest['status'] = 'PARTIAL'+ elif args.command == 'smoke':+ stage = 'GPU smoke'+ if not env['gpus']:+ manifest['status'] = 'SKIPPED'+ manifest['reason'] = 'CUDA unavailable; run default offline smoke'+ else:+ results = gpu_smoke(args, env, run, rows, split, manifest)+ atomic_json(run / 'metrics.json', results)+ manifest['gpu_experiment'] = ('SUCCESS' if manifest['status'] == 'RUNNING'+ else manifest['status'])+ elif args.command not in ('check-env', 'import-data'):+ stage = 'model loading'+ adapter, _ = load_adapter(args, env, run)+ manifest['model'] = adapter.identity+ stage = args.command+ if args.command == 'eval':+ calibration = json.loads(args.calibration.read_text()) if args.calibration else None+ preds, metrics = evaluate(adapter, rows, split, calibration, args.allow_smoke_calibration, args.split)+ metrics['official_parity'] = adapter.parity([r for r in rows if split['assignments'][r['id']] == args.split][:4])+ if args.measure:+ from .runtime import measure+ metrics['measurement'] = measure(adapter, [r for r in rows if split['assignments'][r['id']] == args.split],+ args.warmup, args.batches)+ write_predictions(run / 'predictions.jsonl', preds)+ elif args.command == 'calibrate':+ from .calibration import calibrate+ preds, metrics = evaluate(adapter, rows, split, selection='calibration')+ artifact = calibrate(preds, adapter.identity, split, adapter.original, run / 'calibration.json',+ args.minimum, args.smoke_only)+ metrics['fitted'] = artifact['fitted']+ if any(x['status'] == 'SKIPPED' for x in artifact['fitted'].values()):+ manifest['status'] = 'PARTIAL'+ write_predictions(run / 'predictions.jsonl', preds)+ atomic_json(run / 'metrics.json', metrics)+ manifest['completed'].append(args.command)+ manifest['gpu_experiment'] = 'SUCCESS'+ if manifest['status'] == 'RUNNING':+ manifest['status'] = 'SUCCESS'+ atomic_json(run / 'manifest.json', manifest)+ report(run)+ print(str(run.resolve()))+ return 0+ except Exception as e:+ if manifest['gpu_experiment'] != 'NOT_RUN':+ manifest['gpu_experiment'] = 'FAILED'+ manifest.update(status='FAILED', error={'stage': stage, 'type': type(e).__name__, 'reason': str(e)})+ atomic_json(run / 'manifest.json', manifest)+ (run / 'error.log').write_text(traceback.format_exc())+ report(run)+ print(f'{stage}: {e}; diagnostics: {run}', file=sys.stderr)+ return 1+++if __name__ == '__main__':+ raise SystemExit(main())diff --git a/research/benchmarks/zh_reliability/aggregate_massive.py b/research/benchmarks/zh_reliability/aggregate_massive.pynew file mode 100644index 00000000..afadf689--- /dev/null+++ b/research/benchmarks/zh_reliability/aggregate_massive.py@@ -0,0 +1,320 @@+"""Audit baseline/calibrated runs, or recompute the compact historical evidence on CPU."""+import argparse+from collections import Counter+from datetime import datetime, timezone+import json+import math+from pathlib import Path+import sys+import traceback++sys.path.insert(0, str(Path(__file__).resolve().parents[3]))+from research.benchmarks.zh_reliability.calibration import fit_temperature, validate_calibration+from research.benchmarks.zh_reliability.data import atomic_json, digest, file_hash, partition, read_data+from research.benchmarks.zh_reliability.metrics import paired_bootstrap, probability_record, summarize++CHILDREN = ('base-calibration', 'E0', 'E1')+CONDITIONS = ('E0', 'E1')+SCHEMA = 'massive-alarm-calibration-evidence-v1'+++def require(value, message):+ if not value:+ raise ValueError(message)+++def read_json(path):+ return json.loads(Path(path).read_text(encoding='utf-8'))+++def read_jsonl(path):+ return [json.loads(line) for line in Path(path).read_text(encoding='utf-8').splitlines() if line.strip()]+++def numerically_equal(a, b):+ if isinstance(a, dict) and isinstance(b, dict):+ return a.keys() == b.keys() and all(numerically_equal(a[k], b[k]) for k in a)+ if isinstance(a, list) and isinstance(b, list):+ return len(a) == len(b) and all(numerically_equal(x, y) for x, y in zip(a, b))+ if isinstance(a, float) and isinstance(b, (int, float)):+ return math.isclose(a, b, rel_tol=1e-10, abs_tol=1e-10)+ return a == b+++def evidence_prediction(record, condition):+ """Rebuild probabilities without source text, a tokenizer, model weights or CUDA."""+ row = {'id': record['id'], 'group_id': record['group_id'], 'task_id': record['task_id'],+ 'question': {'type': record['type'], 'criteria': dict.fromkeys(record['option_order'], '')},+ 'label': record['gold'], 'tags': ['public_localized', 'alarm']}+ logits = record['raw_logits'] if condition == 'E0' else record['calibrated_raw_logits']+ temperature = record['temperature'] if condition == 'E0' else record['calibrated_temperature']+ result = probability_record(row, logits, temperature)+ result.update(sample_hash=record['sample_hash'], split=record['split'],+ tokens={'input_tokens': record['input_tokens'], 'truncated': record['truncated']})+ return result+++def compact_summary(rows):+ result = summarize(rows)+ return {key: result[key] for key in ('overall', 'by_type', 'definitions')}+++def recompute(artifact, bootstrap=True):+ """Recompute evidence and validate pairing/fit; do not trust stored summary values."""+ require(artifact['schema'] == SCHEMA, 'unsupported evidence schema')+ records = artifact['records']+ model = artifact['provenance']['model']+ require(model['content_hash'] == digest(model['files']), 'model identity hash mismatch')+ require(model['model_id'] == 'convaiinnovations/laya-multilingual'+ and model['revision'] == 'e4e9ddf21a7b1903b7acffd8814ad4307bf63a67'+ and model['trained'] is False, 'expected pinned original multilingual checkpoint')+ require(artifact['calibration']['minimum_per_type'] == 30 and artifact['calibration']['bounds'] == [.5, 5.],+ 'calibration protocol mismatch')+ from laya.common import QTYPES, temp_bucket+ original = artifact['calibration']['original']+ temperature_config = original['lang_temperatures'].get('zh', original)+ require(artifact['records_sha256'] == digest(records), 'decision record hash mismatch')+ require(len({r['id'] for r in records}) == len(records), 'duplicate decision IDs')+ require(all(r['split'] in ('calibration', 'test') for r in records), 'unexpected evidence split')+ groups = {}+ source_types = {}+ source_bindings = {}+ for rec in records:+ require(rec['type'] in ('choice', 'noul'), 'unsupported question type')+ require(rec['gold'] in rec['option_order'] and len(set(rec['option_order'])) == len(rec['option_order']),+ 'invalid gold/options')+ require(len(rec['raw_logits']) == len(rec['option_order']), 'logit/options mismatch')+ require(len(rec['sample_hash']) == 64 and len(rec['source_row_hash']) == 64, 'invalid source hash')+ require(rec['group_id'] not in groups or groups[rec['group_id']] == rec['split'], 'group leakage')+ groups[rec['group_id']] = rec['split']+ source = (rec['split'], rec['source_id'])+ binding = (rec['split'], rec['group_id'], rec['source_row_hash'])+ require(rec['source_id'] not in source_bindings or source_bindings[rec['source_id']] == binding,+ 'source utterance split/group/hash mismatch')+ source_bindings[rec['source_id']] = binding+ qt = QTYPES[rec['type']]+ expected_temperature = temperature_config['temperature_by_options'].get(+ temp_bucket(qt, len(rec['option_order'])), temperature_config['temperature'][qt])+ require(rec['temperature'] == expected_temperature, 'baseline temperature mismatch')+ source_types.setdefault(source, []).append(rec['type'])+ if rec['split'] == 'test':+ require(rec['calibrated_raw_logits'] == rec['raw_logits'], 'calibration changed raw decision logits')+ require(all(sorted(types) == ['choice', 'noul'] for types in source_types.values()),+ 'source utterances must keep both question types together')+ counts = Counter(r['split'] for r in records)+ group_counts = Counter(groups.values())+ require(counts == {'calibration': 128, 'test': 186}, 'fixed subset decision counts mismatch')+ require(group_counts == {'calibration': 62, 'test': 89}, 'fixed subset group counts mismatch')+ calibration = [evidence_prediction(r, 'E0') for r in records if r['split'] == 'calibration']+ fitted = {}+ for kind in ('choice', 'noul'):+ selected = [r for r in calibration if r['question']['type'] == kind]+ require(len(selected) == 64, 'calibration type count mismatch')+ fitted[kind] = fit_temperature(selected, minimum=30)+ require(numerically_equal(fitted[kind], artifact['calibration']['fitted'][kind]),+ 'fitted temperature/objective mismatch: ' + kind)+ test = [r for r in records if r['split'] == 'test']+ for rec in test:+ require(rec['calibrated_temperature'] == artifact['calibration']['fitted'][rec['type']]['temperature'],+ 'test calibration temperature mismatch')+ predictions = {c: [evidence_prediction(r, c) for r in test] for c in CONDITIONS}+ require(all(a['predicted'] == b['predicted'] for a, b in zip(predictions['E0'], predictions['E1'])),+ 'positive scalar temperature changed argmax')+ result = {c: compact_summary(predictions[c]) for c in CONDITIONS}+ result['fitted'] = fitted+ result['counts'] = {'calibration_decisions': 128, 'calibration_groups': 62,+ 'test_decisions': 186, 'test_groups': 89, 'test_source_utterances': 93}+ if bootstrap:+ config = artifact['analysis']['paired_group_bootstrap']+ require(config == {'repeats': 1000, 'seed': 20260924, 'min_groups': 20}, 'bootstrap protocol mismatch')+ result['paired'] = {'E1-E0': paired_bootstrap(predictions['E0'], predictions['E1'], **config)}+ return result+++def verify_artifact(artifact, bootstrap=True):+ """Return recomputed results; reject changed records, fit, pairing or stored metrics."""+ result = recompute(artifact, bootstrap)+ for key, value in result.items():+ require(numerically_equal(value, artifact['summary'][key]), 'stored summary mismatch: ' + key)+ return result+++def render_report(artifact, result):+ provenance = artifact['provenance']+ lines = ['# MASSIVE Chinese alarm calibration evaluation', '',+ f"Measurement date: **{provenance['measurement_date']}**. "+ f"Execution base: `{provenance['execution_base_commit']}`; "+ f"contribution: `{provenance['contribution_commit']}`.", '',+ provenance['version_note'], '',+ 'E0 and E1 use the same multilingual checkpoint and raw decision logits. E1 fits one '+ 'temperature per question type on calibration data. No model weights are updated.', '',+ '| Condition | Slice | n | Accuracy | Macro F1 | NLL | Brier | ECE |',+ '|---|---|---:|---:|---:|---:|---:|---:|']+ for condition in CONDITIONS:+ for kind, scores in [('overall', result[condition]['overall']), *result[condition]['by_type'].items()]:+ values = ' | '.join(f'{scores[k]:.6f}' for k in ('accuracy', 'macro_f1', 'nll', 'brier', 'ece'))+ lines.append(f"| {condition} | {kind} | {scores['n']} | {values} |")+ lines += ['', 'Positive scalar temperature preserves argmax: accuracy and F1 are unchanged.', '',+ '| Type | Calibration n | Temperature | Bound reached |', '|---|---:|---:|---|']+ for kind, fit in result['fitted'].items():+ lines.append(f"| {kind} | {fit['n']} | {fit['temperature']:.6f} | {fit['boundary_hit']} |")+ if 'paired' in result:+ lines += ['', 'Paired differences are E1 minus E0; negative NLL, Brier and ECE differences are better.', '',+ '| Metric | Difference | 95% group bootstrap interval |', '|---|---:|---|']+ for name, value in result['paired']['E1-E0']['after_minus_before'].items():+ lo, hi = value['ci95']+ lines.append(f"| {name} | {value['delta']:+.6f} | [{lo:+.6f}, {hi:+.6f}] |")+ lines += ['', '## Scope and limits', '',+ '- Test: 93 source utterances, 89 heuristic groups, 186 correlated decisions. Calibration: '+ '64 source utterances, 62 groups, 128 decisions. The paired question types are not independent samples.',+ '- This is a derived MASSIVE 1.1 alarm subset with upstream human intent-review evidence, '+ 'not the full benchmark, naturally occurring customer-service logs, or independent local human review.',+ '- One fixed split/seed; 1,000 group bootstrap draws. Intervals condition on the fixed checkpoint '+ 'and fitted temperatures, omit calibration uncertainty, and are descriptive without multiplicity correction.',+ '- Unknown checkpoint exposure to MASSIVE/SLURP; grouping may miss semantic dependencies. '+ 'These results do not establish general Chinese improvement or unchanged behavior on other tasks.',+ '- The baseline noul fit reaches T=5, the permitted upper bound. Calibration settings were fixed '+ 'without choosing on test. Risk/coverage does not improve at every coverage.',+ '- Configured max_len=512, head_max_len=192, FP16. Historical test inputs were at most 60 tokens; '+ 'this is not a 512-token stress measurement.',+ '- The compact evidence omits source text and retains source IDs/hashes. It can recompute numerical '+ 'results and check pairing; full source-schema/annotation checks require the prepared public dataset.', '']+ return '\n'.join(lines)+++def verify_run_predictions(predictions, rows, split, model, selection, condition):+ expected = {r['id']: r for r in rows if split['assignments'][r['id']] == selection}+ require(len(predictions) == len(expected) == len({r['id'] for r in predictions}), 'prediction count/duplicate mismatch')+ require({r['id'] for r in predictions} == set(expected), 'prediction membership mismatch')+ for rec in predictions:+ row = expected[rec['id']]+ require(rec['split'] == selection and rec['condition'] == condition, 'prediction partition/condition mismatch')+ require(rec['model'] == model and rec['sample_hash'] == digest(row) == split['row_hashes'][rec['id']],+ 'prediction model/source mismatch')+ require(all(rec.get(k) == v for k, v in row.items()), 'prediction source row changed')+ rebuilt = probability_record(row, rec['raw_logits'], rec['temperature'])+ require(all(numerically_equal(rec[k], rebuilt[k]) for k in+ ('option_order', 'gold', 'predicted', 'probabilities', 'log_probabilities', 'confidence', 'p_true')),+ 'probability reconstruction mismatch')+++def collect_run(root):+ """Read only the three evaluation children; never modify the supplied run."""+ root = Path(root)+ protocol = read_json(root / 'protocol.json')+ manifest = read_json(root / 'manifest.json')+ require(file_hash(root / 'protocol.json') == manifest['protocol_sha256'], 'protocol hash mismatch')+ rows = read_data(protocol['data'], formal=True)+ require(file_hash(protocol['data']) == protocol['data_file_sha256'], 'source file hash mismatch')+ require(file_hash(protocol['split_manifest']) == protocol['split_file_sha256'], 'split file hash mismatch')+ split = partition(rows, protocol['split_manifest'], 20260924, True)+ children, predictions, hashes = {}, {}, {}+ for name in CHILDREN:+ directory = root / name+ child = read_json(directory / 'manifest.json')+ require(child['status'] == 'SUCCESS' and child['gpu_experiment'] == 'SUCCESS' and not child['smoke_only'],+ name + ': unsuccessful/formally ineligible run')+ expected = {'formal': True, 'seed': 20260924, 'max_len': 512, 'head_max_len': 192, 'dtype': 'fp16'}+ require(all(child['config'].get(k) == v for k, v in expected.items()), name + ': config mismatch')+ require(child['command'] == ('calibrate' if name == 'base-calibration' else 'eval'), 'wrong command')+ require(read_json(directory / 'split_manifest.json') == split, name + ': split mismatch')+ require(read_json(directory / 'model_identity.json') == child['model'], name + ': model metadata mismatch')+ children[name] = child+ predictions[name] = read_jsonl(directory / 'predictions.jsonl')+ hashes.update({name + '/' + file: file_hash(directory / file) for file in+ ('manifest.json', 'predictions.jsonl', 'metrics.json', 'split_manifest.json',+ 'model_identity.json', 'environment.json')})+ model = children['E0']['model']+ require(all(child['model'] == model for child in children.values()), 'model differs between conditions')+ require(model['content_hash'] == digest(model['files']), 'model identity content hash mismatch')+ require(model['trained'] is False, 'expected original checkpoint')+ for name in CHILDREN:+ verify_run_predictions(predictions[name], rows, split, model,+ 'calibration' if name == 'base-calibration' else 'test',+ 'E0' if name == 'base-calibration' else name)+ calibration = read_json(root / 'base-calibration/calibration.json')+ validate_calibration(calibration, model, split)+ require(calibration['minimum_per_type'] == 30 and calibration['smoke_only'] is False, 'invalid calibration minimum/mode')+ require(calibration['calibration_ids'] == [r['id'] for r in predictions['base-calibration']], 'calibration IDs differ')+ hashes['base-calibration/calibration.json'] = file_hash(root / 'base-calibration/calibration.json')+ after = {r['id']: r for r in predictions['E1']}+ records = []+ for rec in predictions['base-calibration'] + predictions['E0']:+ upstream = rec['provenance']['upstream_annotation']+ compact = {key: rec[key] for key in ('id', 'group_id', 'task_id', 'split', 'sample_hash', 'option_order',+ 'gold', 'raw_logits', 'temperature')}+ compact.update(type=rec['question']['type'], source_id=upstream['source_id'],+ source_row_hash=upstream['source_row_hash'], input_tokens=rec['tokens']['input_tokens'],+ truncated=rec['tokens']['truncated'])+ if rec['split'] == 'test':+ paired = after[rec['id']]+ require(paired['gold'] == rec['gold'] and paired['group_id'] == rec['group_id'], 'pairing changed')+ compact.update(calibrated_raw_logits=paired['raw_logits'], calibrated_temperature=paired['temperature'])+ records.append(compact)+ environment = read_json(root / 'E0/environment.json')+ safe_environment = {k: environment[k] for k in ('python', 'os', 'dependencies', 'cuda', 'cudnn')}+ source_hashes = {Path(k).name: v for k, v in environment.get('source_hashes', {}).items()+ if Path(k).name in ('__main__.py', 'data.py', 'metrics.py', 'calibration.py', 'runtime.py', 'prepare_massive.py')}+ artifact = {'schema': SCHEMA, 'provenance': {+ 'measurement_date': protocol['created_utc'][:10], 'execution_base_commit': environment['commit'],+ 'contribution_commit': 'unrecorded', 'version_note': 'Source version recorded by the execution environment; '+ 'recomputing this artifact does not rerun model inference.', 'execution_worktree_dirty': environment['dirty'],+ 'execution_source_hashes': source_hashes, 'original_file_sha256': hashes,+ 'dataset': 'MASSIVE 1.1 zh-CN alarm subset', 'dataset_license': 'CC-BY-4.0',+ 'publisher_notice': 'https://github.com/alexa/massive/blob/f966f21846043aabef9b0f974fa7970027f43738/NOTICE.md',+ 'data_hash': split['data_hash'], 'split_hash': split['split_hash'],+ 'data_file_sha256': protocol['data_file_sha256'], 'split_file_sha256': protocol['split_file_sha256'],+ 'model': model, 'environment': safe_environment,+ 'evaluation_config': {k: children['E0']['config'][k] for k in+ ('seed', 'max_len', 'head_max_len', 'dtype', 'device')}},+ 'analysis': {'paired_group_bootstrap': {'repeats': 1000, 'seed': 20260924, 'min_groups': 20}},+ 'calibration': {'minimum_per_type': 30, 'bounds': [.5, 5.], 'fitted': calibration['fitted'],+ 'original': calibration['original']}, 'records_sha256': digest(records), 'records': records}+ artifact['summary'] = recompute(artifact)+ for name in CONDITIONS:+ source = read_json(root / name / 'metrics.json')+ require(numerically_equal(artifact['summary'][name], {k: source[k] for k in artifact['summary'][name]}),+ 'recomputed metrics differ from measured child: ' + name)+ return artifact+++def main(argv=None):+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument('run_dir', type=Path, nargs='?')+ parser.add_argument('--artifact', type=Path, help='verify compact evidence without model/data downloads or CUDA')+ parser.add_argument('--report', type=Path, help='write Markdown rebuilt from verified values')+ args = parser.parse_args(argv)+ if (args.run_dir is None) == (args.artifact is None):+ parser.error('provide either RUN_DIR or --artifact')+ try:+ artifact = read_json(args.artifact) if args.artifact else collect_run(args.run_dir)+ result = verify_artifact(artifact) if args.artifact else artifact['summary']+ if args.artifact:+ if args.report:+ args.report.write_text(render_report(artifact, result), encoding='utf-8')+ else:+ root = args.run_dir+ atomic_json(root / 'calibration_evidence.json', artifact)+ atomic_json(root / 'metrics.json', result)+ (args.report or root / 'report.md').write_text(render_report(artifact, result), encoding='utf-8')+ manifest = read_json(root / 'manifest.json')+ manifest.update(status='SUCCESS', gpu_experiment='SUCCESS', active_stage='complete', completed=list(CHILDREN),+ completed_utc=datetime.now(timezone.utc).isoformat(),+ evidence_sha256=file_hash(root / 'calibration_evidence.json'))+ atomic_json(root / 'manifest.json', manifest)+ print('SUCCESS: E0/E1 pairing, fitted temperatures, metrics and 1,000 group bootstrap draws verified')+ return 0+ except Exception as error:+ if args.run_dir:+ manifest_path = args.run_dir / 'manifest.json'+ manifest = read_json(manifest_path) if manifest_path.exists() else {}+ manifest.update(status='FAILED', gpu_experiment='FAILED', active_stage='aggregation',+ error={'stage': 'aggregation', 'type': type(error).__name__, 'reason': str(error)})+ atomic_json(manifest_path, manifest)+ (args.run_dir / 'aggregation-error.log').write_text(traceback.format_exc(), encoding='utf-8')+ print('verification failed: ' + str(error), file=sys.stderr)+ return 1+++if __name__ == '__main__':+ raise SystemExit(main())diff --git a/research/benchmarks/zh_reliability/calibration.py b/research/benchmarks/zh_reliability/calibration.pynew file mode 100644index 00000000..b015a44c--- /dev/null+++ b/research/benchmarks/zh_reliability/calibration.py@@ -0,0 +1,86 @@+"""Fit bounded scalar temperatures from unscaled calibration logits only."""+import copy+import math+from .data import atomic_json, digest+from .metrics import log_probs+++def fit_temperature(rows, minimum=30):+ from laya.common import TEMP_MIN, TEMP_MAX+ if len(rows) < minimum:+ return {'status': 'SKIPPED', 'n': len(rows), 'reason': 'insufficient calibration samples',+ 'temperature': None, 'boundary_hit': False}+ def loss(beta):+ return sum(-log_probs(r['raw_logits'], 1 / beta)[r['option_order'].index(r['gold'])]+ for r in rows) / len(rows)+ # NLL is convex in inverse temperature. Bounded golden-section, no new dependency.+ lo, hi = 1 / TEMP_MAX, 1 / TEMP_MIN+ ratio = (math.sqrt(5) - 1) / 2+ for _ in range(80):+ x, y = hi - ratio * (hi - lo), lo + ratio * (hi - lo)+ if loss(x) < loss(y):+ hi = y+ else:+ lo = x+ candidates = [TEMP_MIN, TEMP_MAX, 1., 1 / ((lo + hi) / 2)]+ t = min(candidates, key=lambda v: (loss(1 / v), abs(v - 1)))+ return {'status': 'SUCCESS', 'n': len(rows), 'temperature': t,+ 'boundary_hit': t in (TEMP_MIN, TEMP_MAX), 'nll_t1': loss(1.), 'nll_fitted': loss(1 / t)}+++def calibrate(rows, model, split, original, path=None, minimum=30, smoke=False):+ if minimum < 30 and not smoke:+ raise ValueError('minimum < 30 is smoke_only')+ if not rows or any(r['split'] != 'calibration' for r in rows):+ raise ValueError('fit only on calibration partition')+ if len({r['id'] for r in rows}) != len(rows):+ raise ValueError('duplicate calibration predictions')+ for r in rows:+ if (split['assignments'].get(r['id']) != 'calibration' or r.get('model') != model+ or r.get('sample_hash') != split['row_hashes'].get(r['id'])):+ raise ValueError('calibration model/partition mismatch')+ cfg = copy.deepcopy(original)+ base = cfg.get('lang_temperatures', {}).get('zh', {+ 'temperature': cfg['temperature'], 'temperature_by_options': cfg['temperature_by_options']})+ override = copy.deepcopy(base)+ fitted = {}+ for kind, index in [('choice', 0), ('noul', 2)]:+ fitted[kind] = fit_temperature([r for r in rows if r['question']['type'] == kind], minimum)+ if fitted[kind]['status'] == 'SUCCESS':+ override['temperature'][index] = fitted[kind]['temperature']+ override['temperature_by_options'] = {+ k: v for k, v in override['temperature_by_options'].items() if not k.startswith(kind + ':')}+ out = {'model': model, 'data_hash': split['data_hash'], 'split_hash': split['split_hash'],+ 'calibration_hash': digest([split['row_hashes'][r['id']] for r in rows]),+ 'calibration_ids': [r['id'] for r in rows], 'smoke_only': smoke,+ 'minimum_per_type': minimum, 'fitted': fitted, 'original': original,+ 'lang_temperatures': {**cfg.get('lang_temperatures', {}), 'zh': override}}+ if path:+ atomic_json(path, out)+ return out+++def validate_calibration(artifact, model, split, allow_smoke=False):+ from laya.common import clamp_temperature+ if artifact['model'] != model or artifact['data_hash'] != split['data_hash']:+ raise ValueError('calibration model or data hash mismatch')+ ids = artifact['calibration_ids']+ if (artifact['split_hash'] != split['split_hash'] or len(set(ids)) != len(ids)+ or any(split['assignments'].get(i) != 'calibration' for i in ids)+ or artifact['calibration_hash'] != digest([split['row_hashes'][i] for i in ids])):+ raise ValueError('calibration partition/hash mismatch')+ if artifact['smoke_only'] and not allow_smoke:+ raise ValueError('smoke calibration requires explicit --allow-smoke-calibration')+ for cfg in artifact['lang_temperatures'].values():+ if len(cfg['temperature']) != 3:+ raise ValueError('invalid temperature vector')+ for t in list(cfg['temperature']) + list(cfg['temperature_by_options'].values()):+ if not isinstance(t, (int, float)) or not math.isfinite(t) or clamp_temperature(t) != t:+ raise ValueError('illegal exported temperature')+ for kind, index in [('choice', 0), ('noul', 2)]:+ fit = artifact['fitted'][kind]+ cfg = artifact['lang_temperatures']['zh']+ if fit['status'] == 'SUCCESS' and (cfg['temperature'][index] != fit['temperature'] or+ any(k.startswith(kind + ':') for k in cfg['temperature_by_options'])):+ raise ValueError('fitted temperature shadowed by stale option bucket')+ return artifact['lang_temperatures']diff --git a/research/benchmarks/zh_reliability/data.py b/research/benchmarks/zh_reliability/data.pynew file mode 100644index 00000000..60ad8a55--- /dev/null+++ b/research/benchmarks/zh_reliability/data.py@@ -0,0 +1,189 @@+"""Strict fixture/real-data validation and immutable group partitions."""+import hashlib+import json+import os+from pathlib import Path+import random+import re+import tempfile+from collections import Counter++SPLITS = ('train', 'dev', 'calibration', 'test')+FIXTURES = Path(__file__).with_name('fixtures.jsonl')+++def digest(value):+ # Preserve criteria insertion order: order is part of the actual model input.+ return hashlib.sha256(json.dumps(value, ensure_ascii=False, separators=(',', ':')).encode()).hexdigest()+++def file_hash(path):+ h = hashlib.sha256()+ with open(path, 'rb') as f:+ for chunk in iter(lambda: f.read(1024 * 1024), b''):+ h.update(chunk)+ return h.hexdigest()+++def atomic_json(path, value):+ path = Path(path)+ path.parent.mkdir(parents=True, exist_ok=True)+ fd, tmp = tempfile.mkstemp(dir=path.parent, prefix='.' + path.name)+ try:+ with os.fdopen(fd, 'w', encoding='utf-8') as f:+ json.dump(value, f, ensure_ascii=False, indent=2, allow_nan=False)+ f.write('\n')+ os.replace(tmp, path)+ finally:+ if os.path.exists(tmp):+ os.unlink(tmp)+++def options(row):+ return ['false', 'true'] if row['question']['type'] == 'noul' else list(row['question']['criteria'])+++def upstream_annotation_verified(provenance):+ """Validate supplied source-review evidence, not a local review of the derived task."""+ if provenance.get('kind') != 'public_human_annotated' or provenance.get('human_verified') is not False:+ return False+ evidence = provenance.get('upstream_annotation')+ if not isinstance(evidence, dict):+ return False+ if (evidence.get('dataset') != 'MASSIVE' or evidence.get('version') != '1.1'+ or not isinstance(evidence.get('source_id'), str) or not evidence['source_id'].strip()+ or not isinstance(provenance.get('source_intent'), str) or not provenance['source_intent'].strip()+ or evidence.get('intent') != provenance['source_intent']):+ return False+ source_hash = evidence.get('source_row_hash')+ evidence_url = evidence.get('evidence_url')+ if (not isinstance(source_hash, str) or not re.fullmatch(r'[0-9a-f]{64}', source_hash)+ or not isinstance(evidence_url, str) or not re.fullmatch(+ r'https://(?:github\.com/alexa/massive/blob/|raw\.githubusercontent\.com/alexa/massive/)'+ r'[0-9a-f]{40}/README\.md(?:#[A-Za-z0-9_-]+)?', evidence_url)):+ return False+ judgments = evidence.get('judgments')+ if not isinstance(judgments, list) or len(judgments) < 3:+ return False+ workers = set()+ for judgment in judgments:+ if (not isinstance(judgment, dict) or not isinstance(judgment.get('worker_id'), str)+ or not judgment['worker_id'].strip() or judgment['worker_id'] in workers+ or type(judgment.get('intent_score')) is not int or judgment['intent_score'] != 1):+ return False+ workers.add(judgment['worker_id'])+ return True+++def read_data(path, formal=False):+ rows = [json.loads(line) for line in Path(path).read_text(encoding='utf-8').splitlines() if line.strip()]+ if not rows:+ raise ValueError('empty dataset')+ seen, tasks, families, relations = set(), {}, {}, {}+ for r in rows:+ for key in ('id', 'group_id', 'task_id', 'state'):+ if not isinstance(r.get(key), str) or not r[key].strip():+ raise ValueError('nonempty string required: ' + key)+ if r['id'] in seen:+ raise ValueError('duplicate id: ' + r['id'])+ seen.add(r['id'])+ q = r.get('question', {})+ if q.get('type') not in ('choice', 'noul') or not isinstance(q.get('instructions'), str):+ raise ValueError('only choice/noul with string instructions are supported')+ c = q.get('criteria')+ if not isinstance(c, dict) or len(c) < 2 or any(not isinstance(v, str) for v in c.values()):+ raise ValueError('criteria must map at least two stable option IDs to string descriptions')+ if any(not isinstance(k, str) or not k.strip() for k in c):+ raise ValueError('invalid option_id')+ if q['type'] == 'noul' and set(c) != {'false', 'true'}:+ raise ValueError('noul criteria must be [false, true]')+ if 'labels' in q:+ labels = q['labels']+ if (q['type'] != 'noul' or not isinstance(labels, dict) or set(labels) != {'false', 'true'}+ or any(not isinstance(x, str) or not x.strip() for x in labels.values())+ or len(set(labels.values())) != 2):+ raise ValueError('invalid noul display labels')+ if r.get('label') not in options(r):+ raise ValueError('label not in semantic options')+ if not isinstance(r.get('tags'), list) or any(not isinstance(t, str) for t in r['tags']):+ raise ValueError('tags must be a list of strings')+ p = r.get('provenance', {})+ if not isinstance(p.get('kind'), str) or type(p.get('human_verified')) is not bool:+ raise ValueError('provenance kind and human_verified required')+ if p['kind'].startswith('agent_authored') and p['human_verified']:+ raise ValueError('agent fixtures cannot claim human verification')+ upstream_verified = upstream_annotation_verified(p)+ if p['kind'] == 'public_human_annotated' and not upstream_verified:+ raise ValueError('public human annotations require uncontested pinned source-review evidence')+ if formal and (p['kind'].startswith('agent_authored') or not (p['human_verified'] or upstream_verified)+ or any(not isinstance(p.get(k), str) or not p[k].strip()+ for k in ('source', 'license', 'annotation_status'))+ or r.get('split') not in SPLITS):+ raise ValueError('formal data requires source, license, verified annotation and fixed split')+ task = (q['type'], tuple(sorted(c.items())), q['instructions'])+ if r['task_id'] in tasks and tasks[r['task_id']] != task:+ raise ValueError('task_id must have consistent semantic criteria/instructions')+ tasks[r['task_id']] = task+ family = p.get('template_family', r['group_id'])+ if family in families and families[family] != r['group_id']:+ raise ValueError('template family crosses groups')+ families[family] = r['group_id']+ rel = r.get('relation')+ if rel:+ if rel.get('kind') not in ('preserve', 'flip') or not isinstance(rel.get('pair_id'), str):+ raise ValueError('invalid relation')+ relations.setdefault(rel['pair_id'], []).append(r)+ for pair in relations.values():+ if len(pair) != 2 or len({r['group_id'] for r in pair}) != 1:+ raise ValueError('each relation must pair exactly two rows in the same group')+ if len({r['relation']['kind'] for r in pair}) != 1 or len({r['task_id'] for r in pair}) != 1:+ raise ValueError('inconsistent pair')+ if (pair[0]['label'] == pair[1]['label']) != (pair[0]['relation']['kind'] == 'preserve'):+ raise ValueError('relation contradicts labels')+ return rows+++def partition(rows, path=None, seed=20260924, formal=False):+ hashes = {r['id']: digest(r) for r in rows}+ content_hash = digest(hashes)+ groups = sorted({r['group_id'] for r in rows})+ if path and Path(path).exists():+ out = json.loads(Path(path).read_text())+ if out['data_hash'] != content_hash or out['row_hashes'] != hashes:+ raise ValueError('split manifest data hash mismatch')+ else:+ if formal:+ mapping = {}+ for r in rows:+ if r['group_id'] in mapping and mapping[r['group_id']] != r['split']:+ raise ValueError('fixed partitions split a group')+ mapping[r['group_id']] = r['split']+ else:+ random.Random(seed).shuffle(groups)+ n = len(groups)+ cuts = [round(n * .6), round(n * .7), round(n * .85), n]+ mapping, start = {}, 0+ for split, end in zip(SPLITS, cuts):+ mapping.update({g: split for g in groups[start:end]})+ start = end+ out = {'seed': seed, 'formal': formal, 'data_hash': content_hash, 'row_hashes': hashes,+ 'groups': mapping, 'assignments': {r['id']: mapping[r['group_id']] for r in rows}}+ if set(out['groups']) != set(groups) or set(out['assignments']) != set(hashes):+ raise ValueError('split manifest membership mismatch')+ for r in rows:+ s = out['assignments'][r['id']]+ if s not in SPLITS or s != out['groups'][r['group_id']] or (formal and s != r['split']):+ raise ValueError('group leakage or fixed split mismatch')+ if out['formal'] != formal:+ raise ValueError('split mode mismatch')+ out['counts'] = dict(Counter(out['assignments'].values()))+ out['group_counts'] = dict(Counter(out['groups'].values()))+ split_hash = digest(out['assignments'])+ if 'split_hash' in out and out['split_hash'] != split_hash:+ raise ValueError('split manifest assignment hash mismatch')+ out['split_hash'] = split_hash+ if any(not out['counts'].get(s) for s in SPLITS):+ raise ValueError('all four partitions must be nonempty')+ if path and not Path(path).exists():+ atomic_json(path, out)+ return outdiff --git a/research/benchmarks/zh_reliability/fixtures.jsonl b/research/benchmarks/zh_reliability/fixtures.jsonlnew file mode 100644index 00000000..dc37971e--- /dev/null+++ b/research/benchmarks/zh_reliability/fixtures.jsonl@@ -0,0 +1,80 @@+{"id": "explicit_refund_01_1", "group_id": "explicit_refund_01", "task_id": "explicit_refund", "state": "包裹还在路上,我决定不要了,请给我退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_01"}}+{"id": "explicit_refund_01_2", "group_id": "explicit_refund_01", "task_id": "explicit_refund", "state": "包裹还在路上,我不想等了,麻烦退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_01"}}+{"id": "explicit_refund_02_1", "group_id": "explicit_refund_02", "task_id": "explicit_refund", "state": "别退款,我只想知道维修进度。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_02"}}+{"id": "explicit_refund_02_2", "group_id": "explicit_refund_02", "task_id": "explicit_refund", "state": "维修进度帮我查查,钱先别退。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_02"}}+{"id": "explicit_refund_03_1", "group_id": "explicit_refund_03", "task_id": "explicit_refund", "state": "如果明天还没发货,我才申请退款,现在先等。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_03"}}+{"id": "explicit_refund_03_2", "group_id": "explicit_refund_03", "task_id": "explicit_refund", "state": "明天还没发货再退,今天不要操作。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_03"}}+{"id": "explicit_refund_04_1", "group_id": "explicit_refund_04", "task_id": "explicit_refund", "state": "我不是不想退款,请现在办理。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_04", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_04"}}+{"id": "explicit_refund_04_2", "group_id": "explicit_refund_04", "task_id": "explicit_refund", "state": "别误会,我确实要把钱退回来。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_04", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_04"}}+{"id": "explicit_refund_05_1", "group_id": "explicit_refund_05", "task_id": "explicit_refund", "state": "这单不要啦,退下款呗。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_05", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_05"}}+{"id": "explicit_refund_05_2", "group_id": "explicit_refund_05", "task_id": "explicit_refund", "state": "这单不要啦,不用退款,送给朋友。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_05", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_05"}}+{"id": "explicit_refund_06_1", "group_id": "explicit_refund_06", "task_id": "explicit_refund", "state": "Please refund 这笔重复扣款,谢谢。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_06", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_06"}}+{"id": "explicit_refund_06_2", "group_id": "explicit_refund_06", "task_id": "explicit_refund", "state": "Please check 这笔重复扣款,暂时不退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_06", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_06"}}+{"id": "explicit_refund_07_1", "group_id": "explicit_refund_07", "task_id": "explicit_refund", "state": "天气真热,快递员也辛苦,但买错尺寸了,请退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_07", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_07"}}+{"id": "explicit_refund_07_2", "group_id": "explicit_refund_07", "task_id": "explicit_refund", "state": "天气真热,快递员也辛苦,尺寸刚好,不用退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_07", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_07"}}+{"id": "explicit_refund_08_1", "group_id": "explicit_refund_08", "task_id": "explicit_refund", "state": "退款按钮在哪里?我只是看看,还没决定。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_08"}}+{"id": "explicit_refund_08_2", "group_id": "explicit_refund_08", "task_id": "explicit_refund", "state": "退款按钮在哪里?我决定现在退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_08"}}+{"id": "explicit_refund_09_1", "group_id": "explicit_refund_09", "task_id": "explicit_refund", "state": "客服说可以退款,但我选择换货,不退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_09", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_09"}}+{"id": "explicit_refund_09_2", "group_id": "explicit_refund_09", "task_id": "explicit_refund", "state": "客服说可以退款,我选择退钱,不换货。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_09", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_09"}}+{"id": "explicit_refund_10_1", "group_id": "explicit_refund_10", "task_id": "explicit_refund", "state": "我没有取消退款的打算,继续退吧。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_10", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_10"}}+{"id": "explicit_refund_10_2", "group_id": "explicit_refund_10", "task_id": "explicit_refund", "state": "我打算取消退款,别继续退了。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_10", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_10"}}+{"id": "explicit_refund_11_1", "group_id": "explicit_refund_11", "task_id": "explicit_refund", "state": "发票开错了,只重开发票,不涉及退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_11", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_11"}}+{"id": "explicit_refund_11_2", "group_id": "explicit_refund_11", "task_id": "explicit_refund", "state": "发票不用重开了,这笔费用请直接退回。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_11", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_11"}}+{"id": "explicit_refund_12_1", "group_id": "explicit_refund_12", "task_id": "explicit_refund", "state": "定金已经交了,活动也取消了,麻烦退定金。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_12"}}+{"id": "explicit_refund_12_2", "group_id": "explicit_refund_12", "task_id": "explicit_refund", "state": "活动取消了,把我交的定金退回来吧。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_12"}}+{"id": "explicit_refund_13_1", "group_id": "explicit_refund_13", "task_id": "explicit_refund", "state": "上次说退款是气话,这次只查物流。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_13"}}+{"id": "explicit_refund_13_2", "group_id": "explicit_refund_13", "task_id": "explicit_refund", "state": "别按上次的气话退款,现在帮我查物流就好。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_13"}}+{"id": "explicit_refund_14_1", "group_id": "explicit_refund_14", "task_id": "explicit_refund", "state": "会员自动续费我没用过,申清退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["typo"], "relation": {"pair_id": "explicit_refund_14", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_14"}}+{"id": "explicit_refund_14_2", "group_id": "explicit_refund_14", "task_id": "explicit_refund", "state": "会员续费没用过,申请把费用退回。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["typo"], "relation": {"pair_id": "explicit_refund_14", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_14"}}+{"id": "explicit_refund_15_1", "group_id": "explicit_refund_15", "task_id": "explicit_refund", "state": "不排除以后退款,但今天没有这个要求。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_15"}}+{"id": "explicit_refund_15_2", "group_id": "explicit_refund_15", "task_id": "explicit_refund", "state": "以后也许会退款,今天先不退。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_15"}}+{"id": "explicit_refund_16_1", "group_id": "explicit_refund_16", "task_id": "explicit_refund", "state": "你说不能退款,我不同意,仍要求退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_16", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_16"}}+{"id": "explicit_refund_16_2", "group_id": "explicit_refund_16", "task_id": "explicit_refund", "state": "你说不能退款,我同意了,不再要求退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_16", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_16"}}+{"id": "explicit_refund_17_1", "group_id": "explicit_refund_17", "task_id": "explicit_refund", "state": "订单 A 保留;订单 B 重复买了,请退 B。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_17", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_17"}}+{"id": "explicit_refund_17_2", "group_id": "explicit_refund_17", "task_id": "explicit_refund", "state": "订单 A 和 B 都保留,重复那件送人。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_17", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_17"}}+{"id": "explicit_refund_18_1", "group_id": "explicit_refund_18", "task_id": "explicit_refund", "state": "refund status 显示处理中,查一下即可,别新建申请。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_18", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_18"}}+{"id": "explicit_refund_18_2", "group_id": "explicit_refund_18", "task_id": "explicit_refund", "state": "查一下已有 refund 的进度,不是要再申请。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_18", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_18"}}+{"id": "explicit_refund_19_1", "group_id": "explicit_refund_19", "task_id": "explicit_refund", "state": "不退不是我的意思,我的意思就是退。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_19", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_19"}}+{"id": "explicit_refund_19_2", "group_id": "explicit_refund_19", "task_id": "explicit_refund", "state": "我的意思就是不退,别理解反了。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_19", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_19"}}+{"id": "explicit_refund_20_1", "group_id": "explicit_refund_20", "task_id": "explicit_refund", "state": "东西破了,赔个配件就行,不必退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_20"}}+{"id": "explicit_refund_20_2", "group_id": "explicit_refund_20", "task_id": "explicit_refund", "state": "东西破了,不要配件了,请退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_20"}}+{"id": "ticket_category_01_1", "group_id": "ticket_category_01", "task_id": "ticket_category", "state": "支付成功后网页又扣了一次钱,请核对账单。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_01"}}+{"id": "ticket_category_01_2", "group_id": "ticket_category_01", "task_id": "ticket_category", "state": "账单上同一笔支付记了两遍,帮忙核对。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_01"}}+{"id": "ticket_category_02_1", "group_id": "ticket_category_02", "task_id": "ticket_category", "state": "应用一点登录就闪退,换过网络也一样。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_02"}}+{"id": "ticket_category_02_2", "group_id": "ticket_category_02", "task_id": "ticket_category", "state": "切了网络仍然登录闪退,看看程序哪里坏了。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_02"}}+{"id": "ticket_category_03_1", "group_id": "ticket_category_03", "task_id": "ticket_category", "state": "新买的键盘一周没送到,查一下运输位置。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_03"}}+{"id": "ticket_category_03_2", "group_id": "ticket_category_03", "task_id": "ticket_category", "state": "键盘寄出一周了还没到,帮我追踪包裹。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_03"}}+{"id": "ticket_category_04_1", "group_id": "ticket_category_04", "task_id": "ticket_category", "state": "不是软件坏了,是账单金额不对。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_04", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_04"}}+{"id": "ticket_category_04_2", "group_id": "ticket_category_04", "task_id": "ticket_category", "state": "不是账单问题,是软件打开就报错。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_04", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_04"}}+{"id": "ticket_category_05_1", "group_id": "ticket_category_05", "task_id": "ticket_category", "state": "如果包裹今天到就不用查;现在请处理无法启动的程序。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_05", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_05"}}+{"id": "ticket_category_05_2", "group_id": "ticket_category_05", "task_id": "ticket_category", "state": "程序无法启动请先处理,物流等今天过后再说。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_05", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_05"}}+{"id": "ticket_category_06_1", "group_id": "ticket_category_06", "task_id": "ticket_category", "state": "不是没收到包裹,我收到了;现在问如何更改昵称。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_06", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_06"}}+{"id": "ticket_category_06_2", "group_id": "ticket_category_06", "task_id": "ticket_category", "state": "包裹已经签收,和物流无关,昵称怎么改?", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_06", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_06"}}+{"id": "ticket_category_07_1", "group_id": "ticket_category_07", "task_id": "ticket_category", "state": "俺的快递跑哪去啦,好几天莫动静。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["colloquial", "option_order"], "relation": {"pair_id": "ticket_category_07", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_07"}}+{"id": "ticket_category_07_2", "group_id": "ticket_category_07", "task_id": "ticket_category", "state": "快递好几天没动静,帮忙找找在哪。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["colloquial", "option_order"], "relation": {"pair_id": "ticket_category_07", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_07"}}+{"id": "ticket_category_08_1", "group_id": "ticket_category_08", "task_id": "ticket_category", "state": "App 报 error 503,无法加载首页。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_08"}}+{"id": "ticket_category_08_2", "group_id": "ticket_category_08", "task_id": "ticket_category", "state": "App 正常,只是 invoice 金额算错。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_08"}}+{"id": "ticket_category_09_1", "group_id": "ticket_category_09", "task_id": "ticket_category", "state": "昨天喝了咖啡,今天天气不错,顺便问你们周日几点营业?", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_09", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_09"}}+{"id": "ticket_category_09_2", "group_id": "ticket_category_09", "task_id": "ticket_category", "state": "天气不错,想问周日营业时间。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_09", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_09"}}+{"id": "ticket_category_10_1", "group_id": "ticket_category_10", "task_id": "ticket_category", "state": "地址写错了,请联系承运方改配送地址。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_10", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_10"}}+{"id": "ticket_category_10_2", "group_id": "ticket_category_10", "task_id": "ticket_category", "state": "配送地址填错,麻烦帮我联系快递改一下。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_10", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_10"}}+{"id": "ticket_category_11_1", "group_id": "ticket_category_11", "task_id": "ticket_category", "state": "优惠券抵扣没有显示在最终付款账单中。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_11", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_11"}}+{"id": "ticket_category_11_2", "group_id": "ticket_category_11", "task_id": "ticket_category", "state": "付款账单没有减去优惠券金额。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_11", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_11"}}+{"id": "ticket_category_12_1", "group_id": "ticket_category_12", "task_id": "ticket_category", "state": "导出文件时程序卡死,重装也不行。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_12"}}+{"id": "ticket_category_12_2", "group_id": "ticket_category_12", "task_id": "ticket_category", "state": "重装后导出还是卡死,需要排查程序。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_12"}}+{"id": "ticket_category_13_1", "group_id": "ticket_category_13", "task_id": "ticket_category", "state": "我想了解你们公司的招聘渠道。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_13"}}+{"id": "ticket_category_13_2", "group_id": "ticket_category_13", "task_id": "ticket_category", "state": "请告诉我应聘岗位应该去哪里看。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_13"}}+{"id": "ticket_category_14_1", "group_id": "ticket_category_14", "task_id": "ticket_category", "state": "快递没丢,不用找;发票税额有误,请核算。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_14", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_14"}}+{"id": "ticket_category_14_2", "group_id": "ticket_category_14", "task_id": "ticket_category", "state": "税额没错,不用算;快递丢了,请查找。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_14", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_14"}}+{"id": "ticket_category_15_1", "group_id": "ticket_category_15", "task_id": "ticket_category", "state": "包裹若被退回再联系我,目前页面白屏才是问题。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_15"}}+{"id": "ticket_category_15_2", "group_id": "ticket_category_15", "task_id": "ticket_category", "state": "页面白屏需要修复,包裹退回是还没发生的假设。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_15"}}+{"id": "ticket_category_16_1", "group_id": "ticket_category_16", "task_id": "ticket_category", "state": "不能说系统没有故障,每次保存都会报错。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_16", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_16"}}+{"id": "ticket_category_16_2", "group_id": "ticket_category_16", "task_id": "ticket_category", "state": "系统保存每次报错,确实有故障。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_16", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_16"}}+{"id": "ticket_category_17_1", "group_id": "ticket_category_17", "task_id": "ticket_category", "state": "物流显是签收,可我根本没拿到呀。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["typo", "option_order"], "relation": {"pair_id": "ticket_category_17", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_17"}}+{"id": "ticket_category_17_2", "group_id": "ticket_category_17", "task_id": "ticket_category", "state": "物流显示已签收,我却没有收到。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["typo", "option_order"], "relation": {"pair_id": "ticket_category_17", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_17"}}+{"id": "ticket_category_18_1", "group_id": "ticket_category_18", "task_id": "ticket_category", "state": "Tracking number 查不到,帮忙确认是否发货。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_18", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_18"}}+{"id": "ticket_category_18_2", "group_id": "ticket_category_18", "task_id": "ticket_category", "state": "Tracking 正常,VAT invoice 少了一项费用。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_18", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_18"}}+{"id": "ticket_category_19_1", "group_id": "ticket_category_19", "task_id": "ticket_category", "state": "我看了三篇教程也问过同事,仍想知道客服几点下班。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_19", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_19"}}+{"id": "ticket_category_19_2", "group_id": "ticket_category_19", "task_id": "ticket_category", "state": "教程与同事都没告诉我,客服几点下班?", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_19", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_19"}}+{"id": "ticket_category_20_1", "group_id": "ticket_category_20", "task_id": "ticket_category", "state": "不用改付款记录,也不用修程序,我是来提个配色建议。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_20"}}+{"id": "ticket_category_20_2", "group_id": "ticket_category_20", "task_id": "ticket_category", "state": "配色挺好,不用改,我要反馈付款记录金额错误。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_20"}}diff --git a/research/benchmarks/zh_reliability/massive_protocol.json b/research/benchmarks/zh_reliability/massive_protocol.jsonnew file mode 100644index 00000000..43e65b09--- /dev/null+++ b/research/benchmarks/zh_reliability/massive_protocol.json@@ -0,0 +1,74 @@+{+ "version": "massive-alarm-calibration-v1",+ "model_revision": "e4e9ddf21a7b1903b7acffd8814ad4307bf63a67",+ "seed": 20260924,+ "max_len": 512,+ "head_max_len": 192,+ "dtype": "fp16",+ "evaluation_device": "RTX 2080 Ti",+ "calibration": {+ "minimum_per_type": 30,+ "n_per_type": 64,+ "bounds": [+ 0.5,+ 5.0+ ],+ "objective": "calibration partition NLL only",+ "smoke_only": false+ },+ "conditions": {+ "E0": "original checkpoint with runtime-clamped original temperatures",+ "E1": "original checkpoint + calibration-fitted temperatures"+ },+ "analysis": {+ "primary_metrics": [+ "nll",+ "accuracy"+ ],+ "secondary_metrics": [+ "macro_f1",+ "brier",+ "ece"+ ],+ "paired_group_bootstrap": {+ "repeats": 1000,+ "seed": 20260924,+ "interval": "percentile 95%",+ "min_groups": 20+ },+ "comparisons": [+ "E1-E0"+ ],+ "interpretation": "Exploratory fixed-checkpoint estimates on 89 heuristic source groups; intervals condition on the fitted temperatures and exclude calibration uncertainty; descriptive intervals without multiplicity correction.",+ "test_access": "Temperatures and evaluation settings locked before E0/E1 test evaluation; no settings selected using test."+ },+ "timing": {+ "conditions": [+ "E0"+ ],+ "warmup_batches": 10,+ "measured_batches": 186,+ "batch_decisions": 1,+ "note": "all 186 test decisions once per timed stage; short utterances under max_len512, not 512-token stress. GPUs not exclusively reserved."+ },+ "limits": [+ "public human-localized alarm subset, not naturally occurring customer-service logs",+ "upstream intent-match evidence, no independent local human review",+ "choice/noul share source text and group; not 942 independent utterances",+ "base model exposure to MASSIVE/SLURP unknown",+ "single seed, single task, no general Chinese improvement claim"+ ],+ "purpose": "Evaluation-only fixed settings. The runner records paths, hashes and time before predictions; train/dev partitions are retained for source split compatibility and are not consumed.",+ "expected_split_counts": {+ "train": 522,+ "dev": 106,+ "calibration": 128,+ "test": 186+ },+ "expected_group_counts": {+ "train": 247,+ "dev": 53,+ "calibration": 62,+ "test": 89+ }+}diff --git a/research/benchmarks/zh_reliability/metrics.py b/research/benchmarks/zh_reliability/metrics.pynew file mode 100644index 00000000..e323b2f5--- /dev/null+++ b/research/benchmarks/zh_reliability/metrics.py@@ -0,0 +1,126 @@+"""Metrics use valid decision probabilities, never action/entropy confidence."""+import math+from collections import defaultdict+import numpy as np+++def log_probs(logits, temperature=1.0, mask=None):+ from laya.common import TEMP_MIN, TEMP_MAX+ if not math.isfinite(temperature) or not TEMP_MIN <= temperature <= TEMP_MAX:+ raise ValueError('temperature outside runtime range')+ z = np.asarray(logits, dtype=np.float64)+ if mask is not None:+ z = z[np.asarray(mask, dtype=bool)]+ if z.ndim != 1 or not len(z) or not np.isfinite(z).all():+ raise ValueError('invalid decision logits')+ z = z / temperature+ z -= z.max()+ return z - np.log(np.exp(z).sum())+++def probability_record(row, logits, temperature):+ from .data import digest, options+ order = options(row)+ if len(logits) != len(order):+ raise ValueError('logits/options mismatch')+ lp = log_probs(logits, temperature)+ p = np.exp(lp)+ return {**row, 'sample_hash': digest(row), 'option_order': order, 'gold': row['label'], 'raw_logits': list(logits),+ 'temperature': temperature, 'probabilities': p.tolist(), 'log_probabilities': lp.tolist(),+ 'predicted': order[int(p.argmax())], 'confidence': float(p.max()),+ 'p_true': float(p[1]) if row['question']['type'] == 'noul' else None}+++def quality(rows):+ if not rows:+ return {'n': 0, 'accuracy': None, 'macro_f1': None, 'nll': None, 'brier': None, 'ece': None,+ 'reason': 'empty group'}+ correct = np.array([r['predicted'] == r['gold'] for r in rows], dtype=float)+ conf = np.array([max(r['probabilities']) for r in rows])+ nll, brier = [], []+ task_rows = defaultdict(list)+ for r in rows:+ task_rows[r['task_id']].append(r)+ g = r['option_order'].index(r['gold'])+ nll.append(-r['log_probabilities'][g])+ brier.append(sum((p - (j == g)) ** 2 for j, p in enumerate(r['probabilities'])))+ f1 = []+ for task in task_rows.values():+ labels = sorted({x for r in task for x in r['option_order']})+ scores = []+ for label in labels:+ tp = sum(r['predicted'] == r['gold'] == label for r in task)+ fp = sum(r['predicted'] == label and r['gold'] != label for r in task)+ fn = sum(r['predicted'] != label and r['gold'] == label for r in task)+ scores.append(2 * tp / (2 * tp + fp + fn) if 2 * tp + fp + fn else 0.0)+ f1.append(float(np.mean(scores)))+ bucket = np.minimum(14, (conf * 15).astype(int))+ ece = sum(np.mean(bucket == b) * abs(correct[bucket == b].mean() - conf[bucket == b].mean())+ for b in range(15) if (bucket == b).any())+ # Ties broken lexically by stable ID, independently of labels/correctness.+ ordered = sorted(rows, key=lambda r: (-max(r['probabilities']), r['id']))+ risk = []+ for c in (.25, .5, .75, .9, 1.):+ k = max(1, math.ceil(c * len(rows)))+ risk.append({'target_coverage': c, 'coverage': k / len(rows), 'n': k,+ 'risk': sum(r['predicted'] != r['gold'] for r in ordered[:k]) / k})+ return {'n': len(rows), 'accuracy': float(correct.mean()), 'macro_f1': float(np.mean(f1)),+ 'nll': float(np.mean(nll)), 'brier': float(np.mean(brier)), 'ece': float(ece),+ 'risk_coverage': risk}+++def summarize(rows):+ result = {'overall': quality(rows), 'definitions': {+ 'macro_f1': 'equal mean of task-local macro F1; all task options, zero for undefined F1',+ 'brier': 'mean sum of squared errors over all valid classes (including both noul classes)',+ 'ece': '15 equal bins [i/15,(i+1)/15), last bin includes 1; confidence=max(p)',+ 'risk_ties': 'descending max(p), then lexical sample ID; ceil target count; diagnostic only'}}+ for name, getter in [('task', lambda r: [r['task_id']]),+ ('type', lambda r: [r['question']['type']]),+ ('option_count', lambda r: [str(len(r['option_order']))]),+ ('phenomenon', lambda r: r['tags']),+ ('truncation', lambda r: [str(r.get('tokens', {}).get('truncated', False))])]:+ groups = defaultdict(list)+ for r in rows:+ for key in getter(r):+ groups[key].append(r)+ if name == 'type':+ for key in ('choice', 'noul'):+ groups.setdefault(key, [])+ result['by_' + name] = {k: quality(v) for k, v in sorted(groups.items())}+ pairs = defaultdict(list)+ for r in rows:+ if r.get('relation'):+ pairs[r['relation']['pair_id']].append(r)+ result['relations'] = {}+ for kind in ('preserve', 'flip'):+ selected = [v for v in pairs.values() if len(v) == 2 and v[0]['relation']['kind'] == kind]+ result['relations'][kind] = {+ 'n_pairs': len(selected),+ 'consistent_or_changed': (sum((v[0]['predicted'] == v[1]['predicted']) == (kind == 'preserve')+ for v in selected) / len(selected)) if selected else None,+ 'both_correct': (sum(all(r['predicted'] == r['gold'] for r in v) for v in selected)+ / len(selected)) if selected else None}+ return result+++def paired_bootstrap(before, after, repeats=1000, seed=20260924, min_groups=20):+ a, b = {r['id']: r for r in before}, {r['id']: r for r in after}+ if a.keys() != b.keys() or any(a[k]['group_id'] != b[k]['group_id'] or a[k]['gold'] != b[k]['gold'] for k in a):+ raise ValueError('paired bootstrap requires identical samples/groups/labels')+ groups = sorted({r['group_id'] for r in before})+ if len(groups) < min_groups:+ return {'status': 'SKIPPED', 'reason': 'too few independent groups', 'n_groups': len(groups)}+ ids = {g: [k for k in a if a[k]['group_id'] == g] for g in groups}+ rng = np.random.default_rng(seed)+ keys = ('accuracy', 'macro_f1', 'nll', 'brier', 'ece')+ deltas = {k: [] for k in keys}+ for _ in range(repeats):+ selected = [i for g in rng.choice(groups, len(groups), replace=True) for i in ids[g]]+ qa, qb = quality([a[i] for i in selected]), quality([b[i] for i in selected])+ for k in keys:+ deltas[k].append(qb[k] - qa[k])+ qa, qb = quality(before), quality(after)+ return {'status': 'SUCCESS', 'n_groups': len(groups), 'repeats': repeats, 'seed': seed,+ 'after_minus_before': {k: {'delta': qb[k] - qa[k], 'ci95': np.quantile(v, [.025, .975]).tolist()}+ for k, v in deltas.items()}}diff --git a/research/benchmarks/zh_reliability/prepare_massive.py b/research/benchmarks/zh_reliability/prepare_massive.pynew file mode 100644index 00000000..e31c180e--- /dev/null+++ b/research/benchmarks/zh_reliability/prepare_massive.py@@ -0,0 +1,357 @@+"""Prepare a licensed, source-reviewed MASSIVE 1.1 Chinese alarm subset (no model calls)."""+import argparse+from collections import Counter, defaultdict+from datetime import datetime, timezone+import json+from pathlib import Path+import random+import re+import shutil+import sys+import tarfile+import unicodedata+import urllib.request++from .data import atomic_json, digest, file_hash, partition, read_data++PUBLISHER_REVISION = 'f966f21846043aabef9b0f974fa7970027f43738'+ARCHIVE_URL = 'https://amazon-massive-nlu-dataset.s3.amazonaws.com/amazon-massive-dataset-1.1.tar.gz'+EVIDENCE_URL = f'https://github.com/alexa/massive/blob/{PUBLISHER_REVISION}/README.md'+RAW_DOCS = f'https://raw.githubusercontent.com/alexa/massive/{PUBLISHER_REVISION}/'+PARQUET_REVISION = 'ed58ac423a2f4121720918bf5301577edce4ffd3'+PARQUET_BASE = f'https://huggingface.co/datasets/AmazonScience/massive/resolve/{PARQUET_REVISION}/'+PARQUET_FILES = tuple(f'{locale}-{split}.parquet' for locale in ('zh-CN', 'en-US')+ for split in ('train', 'validation', 'test'))+# The publisher NOTICE establishes that this license applies to MASSIVE.+LICENSE_TEXT_URL = 'https://raw.githubusercontent.com/creativecommons/cc-legal-tools-data/b40ad648826d12d38207b111d66a708bfa6c5ce4/docs/licenses/by/4.0/legalcode.txt'+INTENTS = {'alarm_set': '设置闹钟', 'alarm_query': '查询闹钟', 'alarm_remove': '删除或取消闹钟'}+SOURCE_FILES = ('zh-CN.jsonl', 'en-US.jsonl', 'DATA_LICENSE', 'README.md', 'NOTICE.md', 'THIRD-PARTY.md', 'LICENSE.txt')+++def fetch_source(directory):+ directory = Path(directory)+ directory.mkdir()+ archive = directory / 'massive-1.1.tar.gz'+ with urllib.request.urlopen(ARCHIVE_URL, timeout=60) as source, archive.with_suffix('.part').open('wb') as out:+ shutil.copyfileobj(source, out)+ archive.with_suffix('.part').replace(archive)+ with tarfile.open(archive, 'r:gz') as tar:+ for member in tar:+ name = member.name+ target = None+ if name.endswith('/data/zh-CN.jsonl'):+ target = 'zh-CN.jsonl'+ elif name.endswith('/data/en-US.jsonl'):+ target = 'en-US.jsonl'+ elif name.endswith('/LICENSE'):+ target = 'DATA_LICENSE'+ if target:+ if not member.isfile() or (directory / target).exists():+ raise ValueError('unexpected/duplicate archive member: ' + name)+ # Read only exact members; never extract arbitrary archive paths/symlinks.+ with tar.extractfile(member) as src, (directory / target).open('wb') as out:+ shutil.copyfileobj(src, out)+ for name in SOURCE_FILES[3:]:+ with urllib.request.urlopen(RAW_DOCS + name, timeout=30) as src:+ (directory / name).write_bytes(src.read())+ metadata = {'archive_url': ARCHIVE_URL, 'archive_sha256': file_hash(archive),+ 'publisher_revision': PUBLISHER_REVISION,+ 'files': {name: file_hash(directory / name) for name in SOURCE_FILES}}+ atomic_json(directory / 'source_manifest.json', metadata)+ return directory+++def check_source(directory):+ directory = Path(directory)+ metadata = json.loads((directory / 'source_manifest.json').read_text())+ if metadata['archive_url'] != ARCHIVE_URL or metadata['publisher_revision'] != PUBLISHER_REVISION:+ raise ValueError('unexpected dataset version or documentation revision')+ transport = metadata.get('transport', 'publisher_archive')+ if transport not in ('publisher_archive', 'hf_parquet'):+ raise ValueError('unexpected source transport')+ if transport == 'hf_parquet' and (metadata.get('mirror_revision') != PARQUET_REVISION+ or metadata.get('mirror_base_url') != PARQUET_BASE+ or metadata.get('license_text_url') != LICENSE_TEXT_URL):+ raise ValueError('unexpected parquet source revision or license URL')+ source_files = SOURCE_FILES + (PARQUET_FILES if transport == 'hf_parquet' else ())+ for name in source_files:+ if metadata['files'].get(name) != file_hash(directory / name):+ raise ValueError('source file hash mismatch: ' + name)+ archive = directory / 'massive-1.1.tar.gz'+ if archive.exists() and file_hash(archive) != metadata['archive_sha256']:+ raise ValueError('source archive hash mismatch')+ if 'CC BY 4.0' not in (directory / 'NOTICE.md').read_text():+ raise ValueError('missing publisher dataset-license notice')+ if 'Attribution 4.0 International' not in (directory / 'DATA_LICENSE').read_text():+ raise ValueError('unexpected dataset license')+ return metadata+++def decode_parquet_row(row, features, locale, split):+ """Decode HF label indices and column-oriented sequences without changing source text."""+ row = dict(row)+ if row.get('locale') != locale or row.get('partition') != ('dev' if split == 'validation' else split):+ raise ValueError('parquet locale/partition mismatch')+ for field in ('scenario', 'intent'):+ feature, value = features[field], row[field]+ names = feature.get('names', [])+ if feature.get('_type') != 'ClassLabel' or type(value) is not int or not 0 <= value < len(names):+ raise ValueError('invalid parquet class label: ' + field)+ row[field] = names[value]+ for field in ('judgments', 'slot_method'):+ value = row.get(field)+ if not isinstance(value, dict) or not value or any(not isinstance(v, list) for v in value.values()):+ raise ValueError('invalid parquet sequence: ' + field)+ if len({len(v) for v in value.values()}) != 1:+ raise ValueError('unequal parquet sequence lengths: ' + field)+ row[field] = [dict(zip(value, values)) for values in zip(*value.values())]+ return row+++def prepare_parquet_source(directory):+ """Decode six pinned files and record artifact and decoded hashes; no network calls."""+ import pyarrow+ import pyarrow.parquet as pq+ directory = Path(directory)+ for locale in ('zh-CN', 'en-US'):+ rows = []+ for split in ('train', 'validation', 'test'):+ table = pq.read_table(directory / f'{locale}-{split}.parquet')+ features = json.loads(table.schema.metadata[b'huggingface'])['info']['features']+ rows.extend(decode_parquet_row(row, features, locale, split) for row in table.to_pylist())+ write_jsonl(directory / f'{locale}.jsonl', rows)+ metadata = {'archive_url': ARCHIVE_URL, 'publisher_revision': PUBLISHER_REVISION,+ 'transport': 'hf_parquet', 'mirror_revision': PARQUET_REVISION, 'mirror_base_url': PARQUET_BASE,+ 'license_text_url': LICENSE_TEXT_URL, 'pyarrow_version': pyarrow.__version__,+ 'decoding': 'ClassLabel indices use embedded feature names; column-oriented judgments/slot_method are transposed to lists. Row hashes cover decoded JSON, not original archive bytes.',+ 'files': {name: file_hash(directory / name) for name in SOURCE_FILES + PARQUET_FILES}}+ atomic_json(directory / 'source_manifest.json', metadata)+ return directory+++def fetch_parquet_source(directory):+ directory = Path(directory)+ directory.mkdir()+ downloads = {f'{locale}-{split}.parquet': f'{PARQUET_BASE}{locale}/{split}/0000.parquet'+ for locale in ('zh-CN', 'en-US') for split in ('train', 'validation', 'test')}+ downloads.update({name: RAW_DOCS + name for name in SOURCE_FILES[3:]})+ downloads['DATA_LICENSE'] = LICENSE_TEXT_URL+ for name, url in downloads.items():+ target = directory / name+ temporary = target.with_suffix(target.suffix + '.part')+ with urllib.request.urlopen(url, timeout=60) as src, temporary.open('wb') as out:+ shutil.copyfileobj(src, out)+ temporary.replace(target)+ return prepare_parquet_source(directory)+++def normalized(text):+ text = unicodedata.normalize('NFKC', text).casefold()+ return ''.join(c for c in text if not c.isspace() and not unicodedata.category(c).startswith('P'))+++def source_template(row):+ # Slot values in English source annotations must not create independent groups.+ return normalized(re.sub(r'\[\s*([^:\]]+?)\s*:\s*[^\]]*\]', r'[\1]', row['annot_utt']))+++def records(path):+ rows = [json.loads(line) for line in Path(path).read_text(encoding='utf-8').splitlines() if line.strip()]+ if len({r['id'] for r in rows}) != len(rows):+ raise ValueError('duplicate source ID')+ return rows+++def confirmed(row):+ judgments = row.get('judgments', [])+ return (isinstance(judgments, list) and len(judgments) >= 3+ and all(isinstance(j, dict) and isinstance(j.get('worker_id'), str) and j['worker_id'].strip()+ and type(j.get('intent_score')) is int and j['intent_score'] == 1 for j in judgments)+ and len({j['worker_id'] for j in judgments}) == len(judgments))+++def convert(chinese, english, source, seed=20260924, calibration_fraction=.20):+ if not 0 < calibration_fraction < 1:+ raise ValueError('calibration fraction must be between zero and one')+ en = {r['id']: r for r in english}+ selected = [r for r in chinese if r.get('scenario') == 'alarm' and r.get('intent') in INTENTS]+ if not selected or len({r['id'] for r in selected}) != len(selected):+ raise ValueError('empty alarm selection or duplicate source IDs')+ parent = {r['id']: r['id'] for r in selected}++ def root(i):+ while parent[i] != i:+ parent[i] = parent[parent[i]]+ i = parent[i]+ return i++ anchors, labels = {}, defaultdict(set)+ for r in selected:+ e = en.get(r['id'])+ if (r.get('locale') != 'zh-CN' or r.get('partition') not in ('train', 'dev', 'test')+ or not isinstance(r.get('utt'), str) or not normalized(r['utt']) or not e+ or e.get('locale') != 'en-US' or e['partition'] != r['partition'] or e['intent'] != r['intent']):+ raise ValueError('Chinese/English source mismatch: ' + str(r['id']))+ chinese_key = normalized(r['utt'])+ labels[chinese_key].add(r['intent'])+ for key in (('zh', chinese_key), ('en_template', source_template(e))):+ if key in anchors:+ parent[root(r['id'])] = root(anchors[key])+ else:+ anchors[key] = r['id']+ groups = defaultdict(list)+ for r in selected:+ groups[root(r['id'])].append(r)+ decisions, diagnostic = [], []+ priority = {'train': 0, 'dev': 1, 'test': 2}+ for members in groups.values():+ owner = max((r['partition'] for r in members), key=priority.get)+ group_id = 'massive-alarm-' + digest(sorted(r['id'] for r in members))[:20]+ conflict = any(len(labels[normalized(r['utt'])]) > 1 for r in members)+ for r in members:+ reasons = []+ if r['partition'] != owner:+ reasons.append('shares_source_template_or_text_with_' + owner)+ if conflict:+ reasons.append('conflicting_intents_for_identical_normalized_chinese')+ if not confirmed(r):+ reasons.append('upstream_intent_review_not_unanimous_or_missing')+ if reasons:+ diagnostic.append({'source_id': r['id'], 'group_id': group_id, 'reasons': reasons,+ 'original_partition': r['partition'], 'source_row': r})+ else:+ decisions.append((r, group_id, owner))+ train_groups = sorted({g for _, g, s in decisions if s == 'train'})+ if len(train_groups) < 2:+ raise ValueError('insufficient independent train groups after source-leakage quarantine')+ random.Random(seed).shuffle(train_groups)+ cal_groups = set(train_groups[:max(1, min(len(train_groups) - 1, round(len(train_groups) * calibration_fraction)))])+ output = []+ for r, group_id, original_split in sorted(decisions, key=lambda x: x[0]['id']):+ split = 'calibration' if group_id in cal_groups else original_split+ provenance = {'kind': 'public_human_annotated', 'human_verified': False,+ 'source': ARCHIVE_URL, 'license': 'CC-BY-4.0', 'source_intent': r['intent'],+ 'annotation_status': 'Unanimous upstream human intent-match judgments; derived mapping has no independent local human review',+ 'local_human_review': False, 'template_family': group_id,+ 'source_partition': original_split, 'source_manifest_hash': digest(source),+ 'source_transport': source.get('transport', 'publisher_archive'),+ 'upstream_annotation': {'dataset': 'MASSIVE', 'version': '1.1', 'source_id': r['id'],+ 'intent': r['intent'], 'source_row_hash': digest(r), 'evidence_url': EVIDENCE_URL,+ 'judgments': r['judgments']}}+ if 'archive_sha256' in source:+ provenance['source_archive_sha256'] = source['archive_sha256']+ if source.get('transport') == 'hf_parquet':+ provenance['source_mirror_revision'] = source['mirror_revision']+ common = {'group_id': group_id, 'state': r['utt'], 'split': split,+ 'tags': ['public_localized', 'alarm'], 'provenance': provenance}+ output.append({**common, 'id': 'massive-' + r['id'] + '-choice', 'task_id': 'massive_alarm_choice_v1',+ 'question': {'type': 'choice', 'instructions': '这句话的主要意图是哪一种闹钟操作?', 'criteria': dict(INTENTS)},+ 'label': r['intent']})+ output.append({**common, 'id': 'massive-' + r['id'] + '-noul', 'task_id': 'massive_alarm_set_noul_v1',+ 'question': {'type': 'noul', 'instructions': '这句话的主要意图是否是设置闹钟?',+ 'criteria': {'false': '意图是查询、删除或取消闹钟', 'true': '意图是设置闹钟'}},+ 'label': 'true' if r['intent'] == 'alarm_set' else 'false'})+ summary = {'source_zh_rows': len(chinese), 'selected_alarm_source_rows': len(selected),+ 'retained_source_rows': len(decisions), 'diagnostic_source_rows': len(diagnostic),+ 'source_groups_before_quarantine': len(groups), 'derived_decisions': len(output),+ 'source_partition_counts': dict(Counter(r['partition'] for r in selected)),+ 'excluded_reasons': dict(Counter(reason for r in diagnostic for reason in r['reasons'])),+ 'seed': seed, 'calibration_fraction_of_remaining_train_groups': calibration_fraction,+ 'local_human_review': False, 'scope': 'MASSIVE-derived source-reviewed alarm subset; not official full benchmark or natural customer-service logs',+ 'grouping_limit': 'source IDs, exact normalized Chinese and delexicalized English templates; undisclosed semantic/template relations may remain',+ 'base_model_training_overlap': 'unknown; no unseen-data claim',+ 'by_split': {s: {'decisions': sum(r['split'] == s for r in output),+ 'source_rows': sum(('calibration' if g in cal_groups else t) == s for _, g, t in decisions),+ 'groups': len({r['group_id'] for r in output if r['split'] == s}),+ 'source_intents': dict(Counter(r['label'] for r in output if r['split'] == s and r['question']['type'] == 'choice'))}+ for s in ('train', 'dev', 'calibration', 'test')}}+ return output, diagnostic, summary+++def write_jsonl(path, rows):+ with Path(path).open('w', encoding='utf-8') as f:+ for row in rows:+ f.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + '\n')+++def main(argv=None):+ p = argparse.ArgumentParser(description=__doc__)+ p.add_argument('--source-dir', type=Path, help='existing source with source_manifest.json; default offline')+ p.add_argument('--download', action='store_true', help='download publisher release into this new run')+ p.add_argument('--download-parquet', action='store_true', help='download only zh-CN/en-US from pinned publisher HF mirror; requires pyarrow')+ p.add_argument('--run-dir', type=Path, required=True, help='new output directory')+ p.add_argument('--seed', type=int, default=20260924)+ p.add_argument('--calibration-fraction', type=float, default=.20)+ args = p.parse_args(argv)+ if sum((bool(args.source_dir), args.download, args.download_parquet)) != 1:+ p.error('choose exactly one of --source-dir, --download or --download-parquet')+ run = args.run_dir+ if run.exists():+ print('refusing existing output directory: ' + str(run), file=sys.stderr)+ return 2+ run.mkdir(parents=True)+ manifest = {'status': 'RUNNING', 'command': 'prepare-massive', 'gpu_experiment': 'NOT_RUN',+ 'created_utc': datetime.now(timezone.utc).isoformat(), 'completed': [],+ 'converter_sha256': file_hash(__file__),+ 'validation_sha256': file_hash(Path(__file__).with_name('data.py')),+ 'config': {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()}}+ atomic_json(run / 'manifest.json', manifest)+ stage = 'source'+ try:+ if args.download:+ source_dir = fetch_source(run / 'source')+ elif args.download_parquet:+ source_dir = fetch_parquet_source(run / 'source')+ else:+ source_dir = args.source_dir+ source = check_source(source_dir)+ atomic_json(run / 'source_manifest.json', source)+ license_dir = run / 'attribution'+ license_dir.mkdir()+ for name in SOURCE_FILES[2:]:+ shutil.copyfile(source_dir / name, license_dir / name)+ (license_dir / 'DERIVATION.md').write_text(+ '# Derived MASSIVE alarm subset\n\n'+ 'MASSIVE Copyright Amazon.com, Inc. or its affiliates. Source: ' + ARCHIVE_URL + '\n\n'+ 'Dataset and underlying SLURP text: CC BY 4.0; see NOTICE.md and DATA_LICENSE.\n\n'+ 'Changes: select three alarm intents with unanimous upstream intent-match reviews; '+ 'quarantine template overlaps and conflicting normalized utterances; derive choice and noul '+ 'questions; split calibration groups from original train. Original Chinese text and '+ 'intent labels remain unchanged. No local human review was performed.\n\n'+ 'Transport and decoding: see ../source_manifest.json. For Parquet, DATA_LICENSE is '+ 'the CC legal text fetched from its recorded official URL, not archive/LICENSE bytes.\n',+ encoding='utf-8')+ manifest['completed'].append('source hashes and publisher attribution')+ stage = 'conversion'+ rows, diagnostics, summary = convert(records(source_dir / 'zh-CN.jsonl'), records(source_dir / 'en-US.jsonl'),+ source, args.seed, args.calibration_fraction)+ write_jsonl(run / 'data.jsonl', rows)+ write_jsonl(run / 'diagnostic.jsonl', diagnostics)+ # Full normal importer validation is a gate, not just converter-specific assertions.+ validated = read_data(run / 'data.jsonl', formal=True)+ split = partition(validated, run / 'split_manifest.json', args.seed, formal=True)+ summary.update(data_hash=split['data_hash'], split_hash=split['split_hash'])+ atomic_json(run / 'metrics.json', summary)+ manifest.update(status='SUCCESS', data_sha256=file_hash(run / 'data.jsonl'),+ diagnostic_sha256=file_hash(run / 'diagnostic.jsonl'))+ manifest['completed'].append('source-reviewed conversion and fixed group partition validation')+ atomic_json(run / 'manifest.json', manifest)+ (run / 'report.md').write_text('# MASSIVE Chinese alarm data preparation\n\n'+ 'Public human-localized data; no local human review or model evaluation has been performed.\n\n'+ 'The original Chinese utterances and publisher intent labels are preserved. '+ 'Choice and noul are deterministic views of the same source and share a group. '+ 'Official test/dev membership is retained for eligible rows; lower-partition template overlaps '+ 'and contested/missing human reviews remain in diagnostic.jsonl. This is a filtered subset, '+ 'not the full MASSIVE benchmark.\n\n```json\n' + json.dumps(summary, ensure_ascii=False, indent=2)+ + '\n```\n\nAttribution: Amazon MASSIVE and SLURP, CC BY 4.0; see attribution/.\n', encoding='utf-8')+ print(str(run.resolve()))+ return 0+ except Exception as e:+ manifest.update(status='FAILED', error={'stage': stage, 'type': type(e).__name__, 'reason': str(e)})+ atomic_json(run / 'manifest.json', manifest)+ print(f'{stage}: {e}; see {run / "manifest.json"}', file=sys.stderr)+ return 1+++if __name__ == '__main__':+ raise SystemExit(main())diff --git a/research/benchmarks/zh_reliability/reproduce_massive.sh b/research/benchmarks/zh_reliability/reproduce_massive.shnew file mode 100644index 00000000..dae116a1--- /dev/null+++ b/research/benchmarks/zh_reliability/reproduce_massive.sh@@ -0,0 +1,77 @@+#!/usr/bin/env bash+set -Eeuo pipefail+# Relative paths are interpreted from the repository root. Outputs must be new.+BENCH_REPO=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../../.." && pwd)+cd "$BENCH_REPO"+EXP_DIR=${1:?Usage: bash research/benchmarks/zh_reliability/reproduce_massive.sh OUTPUT_DIR [MODEL_DIR [DATA_JSONL [SPLIT_JSON]]]}+BENCH_PY=${LAYA_ZH_PYTHON:-.venv/bin/python}+BENCH_BASE=${2:-${LAYA_ZH_MODEL:-runs/zh-download/e4e9ddf21a7b1903b7acffd8814ad4307bf63a67}}+BENCH_DATA=${3:-${LAYA_ZH_DATA:-runs/massive-alarm-data/data.jsonl}}+BENCH_SPLIT=${4:-${LAYA_ZH_SPLIT:-runs/massive-alarm-data/split_manifest.json}}+BENCH_REVISION=e4e9ddf21a7b1903b7acffd8814ad4307bf63a67+BENCH_DEVICE=${LAYA_ZH_DEVICE:-cuda:0}+mkdir -- "$EXP_DIR"+BENCH_STAGE=initialization+mark_failed() {+ BENCH_EXIT_CODE=$?+ "$BENCH_PY" - "$EXP_DIR" "$BENCH_STAGE" "$BENCH_EXIT_CODE" <<'PY_FAIL'+import json+from pathlib import Path+import sys+from research.benchmarks.zh_reliability.data import atomic_json+root, stage, code = Path(sys.argv[1]), sys.argv[2], int(sys.argv[3])+path = root / 'manifest.json'+manifest = json.loads(path.read_text()) if path.exists() else {}+manifest.update(status='FAILED', gpu_experiment='FAILED', active_stage=stage,+ error={'stage': stage, 'exit_code': code})+atomic_json(path, manifest)+PY_FAIL+ return "$BENCH_EXIT_CODE"+}+trap mark_failed ERR+"$BENCH_PY" - "$EXP_DIR" "$BENCH_BASE" "$BENCH_DATA" "$BENCH_SPLIT" "$BENCH_DEVICE" <<'PY_INIT'+from datetime import datetime, timezone+import json+from pathlib import Path+import sys+from research.benchmarks.zh_reliability.data import atomic_json, file_hash, partition, read_data++root, model, data, split_path = (Path(value).resolve() for value in sys.argv[1:5])+template = Path('research/benchmarks/zh_reliability/massive_protocol.json')+protocol = json.loads(template.read_text(encoding='utf-8'))+rows = read_data(data, formal=True)+split = partition(rows, split_path, protocol['seed'], formal=True)+if split['counts'] != protocol['expected_split_counts'] or split['group_counts'] != protocol['expected_group_counts']:+ raise ValueError('data partitions do not match the fixed MASSIVE alarm protocol')+lineage = json.loads((model / 'zh_lineage.json').read_text(encoding='utf-8'))+if (lineage.get('model_id') != 'convaiinnovations/laya-multilingual'+ or lineage.get('base_revision') != protocol['model_revision'] or lineage.get('trained') is not False):+ raise ValueError('use the pinned original multilingual checkpoint with its zh_lineage.json')+protocol['evaluation_device'] = sys.argv[5]+protocol.update(created_utc=datetime.now(timezone.utc).isoformat(), status='FIXED_BEFORE_MODEL_PREDICTIONS',+ data=str(data), data_file_sha256=file_hash(data), split_manifest=str(split_path),+ split_file_sha256=file_hash(split_path), base_model=str(model),+ base_model_weights_sha256=file_hash(model / 'model.safetensors'),+ base_model_lineage_sha256=file_hash(model / 'zh_lineage.json'),+ template_sha256=file_hash(template))+atomic_json(root / 'protocol.json', protocol)+atomic_json(root / 'manifest.json', {+ 'status': 'RUNNING', 'gpu_experiment': 'RUNNING', 'command': 'formal-massive-calibration-replay',+ 'completed': [], 'protocol_sha256': file_hash(root / 'protocol.json')})+PY_INIT+run_benchmark() {+ "$BENCH_PY" -m research.benchmarks.zh_reliability "$@" --data "$BENCH_DATA" --formal \+ --split-manifest "$BENCH_SPLIT" --max-len 512 --head-max-len 192 --dtype fp16 \+ --seed 20260924 --revision "$BENCH_REVISION"+}+BENCH_STAGE=base-calibration+run_benchmark calibrate --model "$BENCH_BASE" --device "$BENCH_DEVICE" --minimum 30 \+ --run-dir "$EXP_DIR/base-calibration"+BENCH_STAGE=E0+run_benchmark eval --model "$BENCH_BASE" --device "$BENCH_DEVICE" --split test \+ --measure --warmup 10 --batches 186 --run-dir "$EXP_DIR/E0"+BENCH_STAGE=E1+run_benchmark eval --model "$BENCH_BASE" --device "$BENCH_DEVICE" --split test \+ --calibration "$EXP_DIR/base-calibration/calibration.json" --run-dir "$EXP_DIR/E1"+BENCH_STAGE=aggregation+"$BENCH_PY" -m research.benchmarks.zh_reliability.aggregate_massive "$EXP_DIR"diff --git a/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.jsonnew file mode 100644index 00000000..35377d4b--- /dev/null+++ b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json@@ -0,0 +1,776 @@+{+ "schema": "massive-alarm-calibration-evidence-v1",+ "provenance": {+ "measurement_date": "2026-09-24",+ "execution_base_commit": "23a17522aa4942da6cce53a995a275760320b691",+ "contribution_commit": "37cc58465da5cdabddf1ff543234397d666ad546",+ "version_note": "Historical inference was executed on 2026-09-24 from base commit 23a17522aa4942da6cce53a995a275760320b691 with a dirty contribution worktree subsequently recorded in 37cc58465da5cdabddf1ff543234397d666ad546. The recorded execution source-file hashes are authoritative for the measured code. These measurements were not rerun on the later upstream rebase or evaluation-only refactor.",+ "execution_worktree_dirty": true,+ "execution_source_hashes": {+ "__main__.py": "2db327517919084fa3257659a168442024952fd36c649a37869cd460275cf692",+ "calibration.py": "0bd68229d859cf55b0826a43f1a43789edc9cb9c07ee849e2144e6ad747263c7",+ "data.py": "baf8061a097c0993fe31d8c7d69db8ccddd6d714075f3fc717b8ddf12ce11c66",+ "metrics.py": "8c0be190b7c9e1d55ce5ebe7a5b8aff6ab8c8ae24183ecb59f7838cf7867925f",+ "prepare_massive.py": "a83c5f74285b6ccc03851efebbbc9a77bf220d891d78ff8e4faddf9a22944e95",+ "runtime.py": "8c28e4d8113d3fc25ad1ca8f06d1d8254a4334fbeaff23e93c8098db0b25cfa9"+ },+ "original_file_sha256": {+ "base-calibration/manifest.json": "106c40af31bdfbc436ce68c6482ec36cef747566ff6d428f5335cdc2061da0ac",+ "base-calibration/predictions.jsonl": "c763309e5f81a8cc97656c89d8a7ee945914ca601a9c9cc6402ce77235315e35",+ "base-calibration/metrics.json": "07d3ea93222c202d1bc95757fee99c064c9448bc324eaaa759f5d06429426384",+ "base-calibration/split_manifest.json": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",+ "base-calibration/model_identity.json": "144087e7a313e8b144e2dc5296d502750c6f1d2223d45910038ccdddcfcabf19",+ "base-calibration/environment.json": "0b8b2a2d2cc8b2fdeef997c608b92309fce3fe8bdb2fde2b2fdaa1a29dd362b2",+ "E0/manifest.json": "e3c9a4791c3c2f152944d2245e6cc05a8b78e881de1283f90fe50f438722fb78",+ "E0/predictions.jsonl": "6524bdd74d10b513e6b6cb468f1993a8ff1d74abfda69eb767efdd41213c6f96",+ "E0/metrics.json": "b5bd3a28d867745e4f44c25bc1fdf4e395cf328b95b384f7610567f9635bd933",+ "E0/split_manifest.json": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",+ "E0/model_identity.json": "144087e7a313e8b144e2dc5296d502750c6f1d2223d45910038ccdddcfcabf19",+ "E0/environment.json": "0b8b2a2d2cc8b2fdeef997c608b92309fce3fe8bdb2fde2b2fdaa1a29dd362b2",+ "E1/manifest.json": "b60a71273d9f3a5d48debe5040107a36af045d8ed7447612e3b576242bf6f284",+ "E1/predictions.jsonl": "393009f41ba9e9d44f0ad88926b85f95896f40c922d1bfbf24759669a162b763",+ "E1/metrics.json": "b6d6cc8bfe1a3376648b17d79ac25b7706a0c98ca586a2c3b6e6d5a891cf7451",+ "E1/split_manifest.json": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",+ "E1/model_identity.json": "144087e7a313e8b144e2dc5296d502750c6f1d2223d45910038ccdddcfcabf19",+ "E1/environment.json": "0b8b2a2d2cc8b2fdeef997c608b92309fce3fe8bdb2fde2b2fdaa1a29dd362b2",+ "base-calibration/calibration.json": "b5e8b409cb5fb340a4911109327f166c85437307a5684235509ca0c1ada92ae2"+ },+ "dataset": "MASSIVE 1.1 zh-CN alarm subset",+ "dataset_license": "CC-BY-4.0",+ "publisher_notice": "https://github.com/alexa/massive/blob/f966f21846043aabef9b0f974fa7970027f43738/NOTICE.md",+ "data_hash": "dd800e8bc62366cbcf06907e63a1cf225c9aa1add35aa9892fe567e07d470b16",+ "split_hash": "9abde4aa5ecbc90de2929617b72f380ab18f51b5bac38659f7c5641c7739c707",+ "data_file_sha256": "e97c52a226d202a6a3a654d6645b7caff5b3a48952258e8d67149778faa4939a",+ "split_file_sha256": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",+ "model": {+ "model_id": "convaiinnovations/laya-multilingual",+ "revision": "e4e9ddf21a7b1903b7acffd8814ad4307bf63a67",+ "content_hash": "27d6023d6a2033c2c8e504d88ede242c0f7faf7ffd85781540d5fb935bbde855",+ "files": {+ "model.safetensors": "9d628fd971b700382ac6f65920a86f149777b2e748e0c955fb3b19695aa8f204",+ "rl_agent_config.json": "25061739243b617ad88d1219ba6f8a9c86c5881ca28df024fa2d9b3b2fcc30c6",+ "encoder/config.json": "83f6916d13ef0f556ac461f28308dc2bffa7ebeadee8ec9e2db5812020ea5bb4",+ "tokenizer/tokenizer.json": "609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f",+ "tokenizer/tokenizer_config.json": "424b69444bf7b5809dc2cd2e36d0bd71b8055124dd24274d6db3c655d38205e7"+ },+ "trained": false+ },+ "environment": {+ "python": "3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]",+ "os": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.39",+ "dependencies": {+ "torch": "2.7.1+cu126",+ "transformers": "4.57.6",+ "numpy": "2.5.3",+ "safetensors": "0.8.0",+ "huggingface_hub": "0.36.2",+ "tokenizers": "0.22.2"+ },+ "cuda": "12.6",+ "cudnn": 90501+ },+ "evaluation_config": {+ "seed": 20260924,+ "max_len": 512,+ "head_max_len": 192,+ "dtype": "fp16",+ "device": "RTX 2080 Ti"+ }+ },+ "analysis": {+ "paired_group_bootstrap": {+ "repeats": 1000,+ "seed": 20260924,+ "min_groups": 20+ }+ },+ "calibration": {+ "minimum_per_type": 30,+ "bounds": [+ 0.5,+ 5.0+ ],+ "fitted": {+ "choice": {+ "status": "SUCCESS",+ "n": 64,+ "temperature": 3.1355035588563744,+ "boundary_hit": false,+ "nll_t1": 1.21606373270805,+ "nll_fitted": 0.771914452059509+ },+ "noul": {+ "status": "SUCCESS",+ "n": 64,+ "temperature": 5.0,+ "boundary_hit": true,+ "nll_t1": 1.5203078336357765,+ "nll_fitted": 0.7411119749553794+ }+ },+ "original": {+ "temperature_raw": [+ 1.0,+ 1.0,+ 1.0+ ],+ "temperature_by_options_raw": {},+ "temperature": [+ 1.0,+ 1.0,+ 1.0+ ],+ "temperature_by_options": {},+ "lang_temperatures": {}+ }+ },+ "records_sha256": "570f24fe7110855ac974d12903e45579c496d853625a45e916e20f2381e28852",+ "summary": {+ "E0": {+ "overall": {+ "n": 186,+ "accuracy": 0.532258064516129,+ "macro_f1": 0.4408400882085093,+ "nll": 1.3837465414314734,+ "brier": 0.7300712607122152,+ "ece": 0.36648789161804096,+ "risk_coverage": [+ {+ "target_coverage": 0.25,+ "coverage": 0.25268817204301075,+ "n": 47,+ "risk": 0.1702127659574468+ },+ {+ "target_coverage": 0.5,+ "coverage": 0.5,+ "n": 93,+ "risk": 0.27956989247311825+ },+ {+ "target_coverage": 0.75,+ "coverage": 0.7526881720430108,+ "n": 140,+ "risk": 0.4+ },+ {+ "target_coverage": 0.9,+ "coverage": 0.9032258064516129,+ "n": 168,+ "risk": 0.43452380952380953+ },+ {+ "target_coverage": 1.0,+ "coverage": 1.0,+ "n": 186,+ "risk": 0.46774193548387094+ }+ ]+ },+ "by_type": {+ "choice": {+ "n": 93,+ "accuracy": 0.6344086021505376,+ "macro_f1": 0.5809282967177705,+ "nll": 1.2960431695747896,+ "brier": 0.5208253347319968,+ "ece": 0.2534260594442905,+ "risk_coverage": [+ {+ "target_coverage": 0.25,+ "coverage": 0.25806451612903225,+ "n": 24,+ "risk": 0.2916666666666667+ },+ {+ "target_coverage": 0.5,+ "coverage": 0.5053763440860215,+ "n": 47,+ "risk": 0.19148936170212766+ },+ {+ "target_coverage": 0.75,+ "coverage": 0.7526881720430108,+ "n": 70,+ "risk": 0.2571428571428571+ },+ {+ "target_coverage": 0.9,+ "coverage": 0.9032258064516129,+ "n": 84,+ "risk": 0.3333333333333333+ },+ {+ "target_coverage": 1.0,+ "coverage": 1.0,+ "n": 93,+ "risk": 0.3655913978494624+ }+ ]+ },+ "noul": {+ "n": 93,+ "accuracy": 0.43010752688172044,+ "macro_f1": 0.3007518796992481,+ "nll": 1.4714499132881573,+ "brier": 0.9393171866924332,+ "ece": 0.49461061636621556,+ "risk_coverage": [+ {+ "target_coverage": 0.25,+ "coverage": 0.25806451612903225,+ "n": 24,+ "risk": 0.20833333333333334+ },+ {+ "target_coverage": 0.5,+ "coverage": 0.5053763440860215,+ "n": 47,+ "risk": 0.3829787234042553+ },+ {+ "target_coverage": 0.75,+ "coverage": 0.7526881720430108,+ "n": 70,+ "risk": 0.4857142857142857+ },+ {+ "target_coverage": 0.9,+ "coverage": 0.9032258064516129,+ "n": 84,+ "risk": 0.5357142857142857+ },+ {+ "target_coverage": 1.0,+ "coverage": 1.0,+ "n": 93,+ "risk": 0.5698924731182796+ }+ ]+ }+ },+ "definitions": {+ "macro_f1": "equal mean of task-local macro F1; all task options, zero for undefined F1",+ "brier": "mean sum of squared errors over all valid classes (including both noul classes)",+ "ece": "15 equal bins [i/15,(i+1)/15), last bin includes 1; confidence=max(p)",+ "risk_ties": "descending max(p), then lexical sample ID; ceil target count; diagnostic only"+ }+ },+ "E1": {+ "overall": {+ "n": 186,+ "accuracy": 0.532258064516129,+ "macro_f1": 0.4408400882085093,+ "nll": 0.7850334112663782,+ "brier": 0.5068983072913132,+ "ece": 0.1975294208240641,+ "risk_coverage": [+ {+ "target_coverage": 0.25,+ "coverage": 0.25268817204301075,+ "n": 47,+ "risk": 0.19148936170212766+ },+ {+ "target_coverage": 0.5,+ "coverage": 0.5,+ "n": 93,+ "risk": 0.2903225806451613+ },+ {+ "target_coverage": 0.75,+ "coverage": 0.7526881720430108,+ "n": 140,+ "risk": 0.40714285714285714+ },+ {+ "target_coverage": 0.9,+ "coverage": 0.9032258064516129,+ "n": 168,+ "risk": 0.43452380952380953+ },+ {+ "target_coverage": 1.0,+ "coverage": 1.0,+ "n": 186,+ "risk": 0.46774193548387094+ }+ ]+ },+ "by_type": {+ "choice": {+ "n": 93,+ "accuracy": 0.6344086021505376,+ "macro_f1": 0.5809282967177705,+ "nll": 0.8372707851828709,+ "brier": 0.4740320871109113,+ "ece": 0.16990508653316772,+ "risk_coverage": [+ {+ "target_coverage": 0.25,+ "coverage": 0.25806451612903225,+ "n": 24,+ "risk": 0.2916666666666667+ },+ {+ "target_coverage": 0.5,+ "coverage": 0.5053763440860215,+ "n": 47,+ "risk": 0.19148936170212766+ },+ {+ "target_coverage": 0.75,+ "coverage": 0.7526881720430108,+ "n": 70,+ "risk": 0.24285714285714285+ },+ {+ "target_coverage": 0.9,+ "coverage": 0.9032258064516129,+ "n": 84,+ "risk": 0.32142857142857145+ },+ {+ "target_coverage": 1.0,+ "coverage": 1.0,+ "n": 93,+ "risk": 0.3655913978494624+ }+ ]+ },+ "noul": {+ "n": 93,+ "accuracy": 0.43010752688172044,+ "macro_f1": 0.3007518796992481,+ "nll": 0.7327960373498855,+ "brier": 0.5397645274717153,+ "ece": 0.2896348457994376,+ "risk_coverage": [+ {+ "target_coverage": 0.25,+ "coverage": 0.25806451612903225,+ "n": 24,+ "risk": 0.20833333333333334+ },+ {+ "target_coverage": 0.5,+ "coverage": 0.5053763440860215,+ "n": 47,+ "risk": 0.3829787234042553+ },+ {+ "target_coverage": 0.75,+ "coverage": 0.7526881720430108,+ "n": 70,+ "risk": 0.4857142857142857+ },+ {+ "target_coverage": 0.9,+ "coverage": 0.9032258064516129,+ "n": 84,+ "risk": 0.5357142857142857+ },+ {+ "target_coverage": 1.0,+ "coverage": 1.0,+ "n": 93,+ "risk": 0.5698924731182796+ }+ ]+ }+ },+ "definitions": {+ "macro_f1": "equal mean of task-local macro F1; all task options, zero for undefined F1",+ "brier": "mean sum of squared errors over all valid classes (including both noul classes)",+ "ece": "15 equal bins [i/15,(i+1)/15), last bin includes 1; confidence=max(p)",+ "risk_ties": "descending max(p), then lexical sample ID; ceil target count; diagnostic only"+ }+ },+ "fitted": {+ "choice": {+ "status": "SUCCESS",+ "n": 64,+ "temperature": 3.1355035588563744,+ "boundary_hit": false,+ "nll_t1": 1.21606373270805,+ "nll_fitted": 0.771914452059509+ },+ "noul": {+ "status": "SUCCESS",+ "n": 64,+ "temperature": 5.0,+ "boundary_hit": true,+ "nll_t1": 1.5203078336357765,+ "nll_fitted": 0.7411119749553794+ }+ },+ "counts": {+ "calibration_decisions": 128,+ "calibration_groups": 62,+ "test_decisions": 186,+ "test_groups": 89,+ "test_source_utterances": 93+ },+ "paired": {+ "E1-E0": {+ "status": "SUCCESS",+ "n_groups": 89,+ "repeats": 1000,+ "seed": 20260924,+ "after_minus_before": {+ "accuracy": {+ "delta": 0.0,+ "ci95": [+ 0.0,+ 0.0+ ]+ },+ "macro_f1": {+ "delta": 0.0,+ "ci95": [+ 0.0,+ 0.0+ ]+ },+ "nll": {+ "delta": -0.5987131301650952,+ "ci95": [+ -0.8657735567616283,+ -0.3690991941162156+ ]+ },+ "brier": {+ "delta": -0.22317295342090193,+ "ci95": [+ -0.2988262606403422,+ -0.15288959035634678+ ]+ },+ "ece": {+ "delta": -0.16895847079397686,+ "ci95": [+ -0.22499543687347137,+ -0.09854721867871374+ ]+ }+ }+ }+ }+ },+ "records": [+ {"id":"massive-1003-choice","group_id":"massive-alarm-9c31991bfab87db64771","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"e6fa805478662b9dbcc6c13d66010516a205998b1e55b8fa88718b9a5ca94f46","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.212890625,-3.9296875,1.6923828125],"temperature":1.0,"type":"choice","source_id":"1003","source_row_hash":"c139b76204b21a35e4e183738cdcebd381ba62ffd54b666f3265e945432e67a4","input_tokens":52,"truncated":false},+ {"id":"massive-1003-noul","group_id":"massive-alarm-9c31991bfab87db64771","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"6f5566a71b373d478175fad317696441493eb97aed759b090d2a2c48545ee861","option_order":["false","true"],"gold":"false","raw_logits":[-1.9501953125,0.350830078125],"temperature":1.0,"type":"noul","source_id":"1003","source_row_hash":"c139b76204b21a35e4e183738cdcebd381ba62ffd54b666f3265e945432e67a4","input_tokens":47,"truncated":false},+ {"id":"massive-1037-choice","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"bb5e6d977cbf55ef5e40bb73ea9baf10960578c334f3d536be4306644facb148","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.31640625,-1.30078125,-2.2265625],"temperature":1.0,"type":"choice","source_id":"1037","source_row_hash":"cc358f7ff6234a620435ded52f9c6458c9725655acb4e04af4631fd08e5a5204","input_tokens":54,"truncated":false},+ {"id":"massive-1037-noul","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"3e21af49ed7e7d8892eab2ba9ab022e2e3588c8b688c2341ba43c4b28c8864f9","option_order":["false","true"],"gold":"false","raw_logits":[-2.087890625,0.63623046875],"temperature":1.0,"type":"noul","source_id":"1037","source_row_hash":"cc358f7ff6234a620435ded52f9c6458c9725655acb4e04af4631fd08e5a5204","input_tokens":49,"truncated":false},+ {"id":"massive-1263-choice","group_id":"massive-alarm-63e247661f9a8287cd19","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"08d4385e1c618bb738acb9a5a596d85083a795736d25f99d24a4109c26bce6e1","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.66015625,1.666015625,-2.01171875],"temperature":1.0,"type":"choice","source_id":"1263","source_row_hash":"4c56c9a24f86097f01329c4a62df6b539d96202d275ca101ee455c86e1997ca1","input_tokens":51,"truncated":false},+ {"id":"massive-1263-noul","group_id":"massive-alarm-63e247661f9a8287cd19","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"6754f8f7870ad226a15315003e5c8c2d81894182d894561530bf9df030f82f52","option_order":["false","true"],"gold":"false","raw_logits":[-1.7666015625,0.31396484375],"temperature":1.0,"type":"noul","source_id":"1263","source_row_hash":"4c56c9a24f86097f01329c4a62df6b539d96202d275ca101ee455c86e1997ca1","input_tokens":46,"truncated":false},+ {"id":"massive-1436-choice","group_id":"massive-alarm-c583aafff75bc3841fef","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3424e9c88433013f65faaf5bd116ec0d597ba22d08776672379224d576f843f6","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[5.23828125,-2.333984375,-2.853515625],"temperature":1.0,"type":"choice","source_id":"1436","source_row_hash":"94eb7e687527f8f71b3feca63c7dbbd70e8a983b83278b2da3c6c151e6c788d7","input_tokens":52,"truncated":false},+ {"id":"massive-1436-noul","group_id":"massive-alarm-c583aafff75bc3841fef","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"3a9863e2c88dffd907da9f6044347c9f61de6505d1a9dce9417f406054ad0b2e","option_order":["false","true"],"gold":"false","raw_logits":[-1.951171875,1.1201171875],"temperature":1.0,"type":"noul","source_id":"1436","source_row_hash":"94eb7e687527f8f71b3feca63c7dbbd70e8a983b83278b2da3c6c151e6c788d7","input_tokens":47,"truncated":false},+ {"id":"massive-1604-choice","group_id":"massive-alarm-7aa04ea17df3cdddb3fd","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"c544d289a669972f9dd9ffc0abfbabbd6242e07c0777812794fa4cd915f376dd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.869140625,0.30810546875,-0.5908203125],"temperature":1.0,"type":"choice","source_id":"1604","source_row_hash":"11a0e6002c5740f2e54e2854cfc01f4b30c2d0d7e65eccfa97155aa39c654f4b","input_tokens":52,"truncated":false},+ {"id":"massive-1604-noul","group_id":"massive-alarm-7aa04ea17df3cdddb3fd","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"07d97cb090365c59b005f4f668e2e3fff1c297a02363218def7e349191debaf7","option_order":["false","true"],"gold":"false","raw_logits":[-1.8564453125,0.95654296875],"temperature":1.0,"type":"noul","source_id":"1604","source_row_hash":"11a0e6002c5740f2e54e2854cfc01f4b30c2d0d7e65eccfa97155aa39c654f4b","input_tokens":47,"truncated":false},+ {"id":"massive-1657-choice","group_id":"massive-alarm-8bd33166b8b9b2f13b07","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d7486a4cb8953ccc3bb3ddd9179299768df6d34c1e2e880e766e75efedfd7be5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.91015625,-1.087890625,-3.57421875],"temperature":1.0,"type":"choice","source_id":"1657","source_row_hash":"ef2da490978458ee2ef0ee0a0d3b0133c56588c1e26696bccf3e2aa4c8259058","input_tokens":53,"truncated":false},+ {"id":"massive-1657-noul","group_id":"massive-alarm-8bd33166b8b9b2f13b07","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"a560e880161b61f771ac3a6338ff7e841df824be6b274d90320b5785ce6d72b3","option_order":["false","true"],"gold":"false","raw_logits":[-1.30859375,1.0390625],"temperature":1.0,"type":"noul","source_id":"1657","source_row_hash":"ef2da490978458ee2ef0ee0a0d3b0133c56588c1e26696bccf3e2aa4c8259058","input_tokens":48,"truncated":false},+ {"id":"massive-1733-choice","group_id":"massive-alarm-57e77352e57177a49115","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a7c5c35f649bc0b1446ed1c39de844fbbe2bfedb8762971db11bff47397c568d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.76171875,-1.87890625,-0.54833984375],"temperature":1.0,"type":"choice","source_id":"1733","source_row_hash":"a1f5a6d541d3e0f93903c8eb255a58ef9734fe6e55a7407e4015d4ebe61f872a","input_tokens":50,"truncated":false},+ {"id":"massive-1733-noul","group_id":"massive-alarm-57e77352e57177a49115","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"444a0c395aed016e36c3ee3c22e2f4668c4d0012de718db84987c75b48deba4a","option_order":["false","true"],"gold":"false","raw_logits":[-0.98388671875,0.47119140625],"temperature":1.0,"type":"noul","source_id":"1733","source_row_hash":"a1f5a6d541d3e0f93903c8eb255a58ef9734fe6e55a7407e4015d4ebe61f872a","input_tokens":45,"truncated":false},+ {"id":"massive-233-choice","group_id":"massive-alarm-a76f37cef918fa476dce","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1edd77d48a208990431178b3f9ff0e81bba90fc424f207edfe4901049150c5c6","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.35546875,-3.86328125,-3.658203125],"temperature":1.0,"type":"choice","source_id":"233","source_row_hash":"d4a1830096b5b362cb2c8979d9bee17ce35e11662498301fcf7d424aba32f005","input_tokens":53,"truncated":false},+ {"id":"massive-233-noul","group_id":"massive-alarm-a76f37cef918fa476dce","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"bbac79120c3ff2705b203e60febc97ce48a4593d3ed949b4e48d407694ac163a","option_order":["false","true"],"gold":"true","raw_logits":[-2.30078125,1.5107421875],"temperature":1.0,"type":"noul","source_id":"233","source_row_hash":"d4a1830096b5b362cb2c8979d9bee17ce35e11662498301fcf7d424aba32f005","input_tokens":48,"truncated":false},+ {"id":"massive-2335-choice","group_id":"massive-alarm-2e28becb2c156092a49a","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1b380ecf765e790d39243a04e05a6a1a4e89dc1dcd3153c1351181a4f7f04dea","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.982421875,-4.1015625,-3.669921875],"temperature":1.0,"type":"choice","source_id":"2335","source_row_hash":"714a2094c1a038fba50aa33f971e26771226ce7e9e04ebb9d2c7efcec38d0310","input_tokens":56,"truncated":false},+ {"id":"massive-2335-noul","group_id":"massive-alarm-2e28becb2c156092a49a","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"2ab3d560c5bc5dafe905f62e711b23fa7294c813fbb83752e5d86531b0739ce4","option_order":["false","true"],"gold":"true","raw_logits":[-2.322265625,1.3525390625],"temperature":1.0,"type":"noul","source_id":"2335","source_row_hash":"714a2094c1a038fba50aa33f971e26771226ce7e9e04ebb9d2c7efcec38d0310","input_tokens":51,"truncated":false},+ {"id":"massive-238-choice","group_id":"massive-alarm-d0053508597b2d9fa47e","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d7642d01f72c4bf4414827571c80447d99b745af71a1fbb58350a576c27de378","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.7109375,-1.130859375,-1.6572265625],"temperature":1.0,"type":"choice","source_id":"238","source_row_hash":"b05a2c56edee6e6a1f6ad98e6e4db9b0ac80e73346236fb818961f2a8aad75eb","input_tokens":50,"truncated":false},+ {"id":"massive-238-noul","group_id":"massive-alarm-d0053508597b2d9fa47e","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"8901ba86edfac63a84581592485d396272294c4567e25620eecaf2ab26a0efeb","option_order":["false","true"],"gold":"false","raw_logits":[-1.6650390625,1.33203125],"temperature":1.0,"type":"noul","source_id":"238","source_row_hash":"b05a2c56edee6e6a1f6ad98e6e4db9b0ac80e73346236fb818961f2a8aad75eb","input_tokens":45,"truncated":false},+ {"id":"massive-2872-choice","group_id":"massive-alarm-28ce6af777b18d848cc4","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3266abead6625894002caf1ffbd1b89d1302d2cfda49d08401fd6188c04d450e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[5.796875,-4.578125,-4.578125],"temperature":1.0,"type":"choice","source_id":"2872","source_row_hash":"4702426e7fb1d8fa48b86077ff3fb58fa8999c52a880e860661019a7aff3a2ff","input_tokens":50,"truncated":false},+ {"id":"massive-2872-noul","group_id":"massive-alarm-28ce6af777b18d848cc4","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"a90d58891fd2025284bd565a1f8fa39b735090a012bd696054f66d8d81022dfb","option_order":["false","true"],"gold":"false","raw_logits":[-1.7099609375,1.833984375],"temperature":1.0,"type":"noul","source_id":"2872","source_row_hash":"4702426e7fb1d8fa48b86077ff3fb58fa8999c52a880e860661019a7aff3a2ff","input_tokens":45,"truncated":false},+ {"id":"massive-3150-choice","group_id":"massive-alarm-97d6527cc37656846b02","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"090b673551073f1ebfdf857a5bb68d8e62b08acfc82426883ea71e1709f3797e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.951171875,-2.08203125,-0.1639404296875],"temperature":1.0,"type":"choice","source_id":"3150","source_row_hash":"4d419b74038f055d4737cbf88b4723ee2a78b9ce112e8bf29e34e37d457b8022","input_tokens":53,"truncated":false},+ {"id":"massive-3150-noul","group_id":"massive-alarm-97d6527cc37656846b02","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c90f89f70a86da01043d5a5439c184efa63669f3f9a36adace7492118154db62","option_order":["false","true"],"gold":"false","raw_logits":[-2.302734375,1.0859375],"temperature":1.0,"type":"noul","source_id":"3150","source_row_hash":"4d419b74038f055d4737cbf88b4723ee2a78b9ce112e8bf29e34e37d457b8022","input_tokens":48,"truncated":false},+ {"id":"massive-3155-choice","group_id":"massive-alarm-1a05a193b7245896522f","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"bff7c384e9000bf1ca3d85672a1230b2e6b175b679b063b0be917adc3a0b6ae9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.4921875,-2.19140625,-2.087890625],"temperature":1.0,"type":"choice","source_id":"3155","source_row_hash":"ad03657b56afbfa502e69cf2bd0e29a98c83996c41222f4931b15bd39ae5c116","input_tokens":61,"truncated":false},+ {"id":"massive-3155-noul","group_id":"massive-alarm-1a05a193b7245896522f","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"761825a90cafa87fe6dc6a599248c59f4ac2296feb91e87a0b7ba56befc4453a","option_order":["false","true"],"gold":"true","raw_logits":[-2.064453125,1.5185546875],"temperature":1.0,"type":"noul","source_id":"3155","source_row_hash":"ad03657b56afbfa502e69cf2bd0e29a98c83996c41222f4931b15bd39ae5c116","input_tokens":56,"truncated":false},+ {"id":"massive-3180-choice","group_id":"massive-alarm-e4f7f3a520230de4d198","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1be584752d9da34d19148ed46c921d36f618e808f753b1b66487bab510798a0c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.625,-2.34375,-1.4287109375],"temperature":1.0,"type":"choice","source_id":"3180","source_row_hash":"48769eebc7610c81b81ad51c5266ea0166abeb999712ec3304969fda5c1136c6","input_tokens":55,"truncated":false},+ {"id":"massive-3180-noul","group_id":"massive-alarm-e4f7f3a520230de4d198","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"74856b6b33b3577a5dd45f8b2bf787504276b37df6b707eeffe9fae3efca17d7","option_order":["false","true"],"gold":"true","raw_logits":[-0.70849609375,0.75732421875],"temperature":1.0,"type":"noul","source_id":"3180","source_row_hash":"48769eebc7610c81b81ad51c5266ea0166abeb999712ec3304969fda5c1136c6","input_tokens":50,"truncated":false},+ {"id":"massive-3181-choice","group_id":"massive-alarm-513568a33433630f98d8","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"830d07369c5b2e5c39ee664f2fdcbf69533affad202f7d9a653a5494cf4b489e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.8046875,-4.19140625,-3.318359375],"temperature":1.0,"type":"choice","source_id":"3181","source_row_hash":"ac7f0b47957f7510837d1547341e088f0a4d09ed3a9701a61b95a7854d695e0e","input_tokens":55,"truncated":false},+ {"id":"massive-3181-noul","group_id":"massive-alarm-513568a33433630f98d8","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"3defc8360fec5e2ca53cce81386406fafd3a3537186fb44d3bd1e8dc5f2d474b","option_order":["false","true"],"gold":"true","raw_logits":[-1.85546875,1.07421875],"temperature":1.0,"type":"noul","source_id":"3181","source_row_hash":"ac7f0b47957f7510837d1547341e088f0a4d09ed3a9701a61b95a7854d695e0e","input_tokens":50,"truncated":false},+ {"id":"massive-3241-choice","group_id":"massive-alarm-b22b203130c5192a6423","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"f524b17f665cda002061c754e3fd41fd0e740c3018b24ba2d98ae591ceedb7c4","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.1953125,-3.646484375,-3.552734375],"temperature":1.0,"type":"choice","source_id":"3241","source_row_hash":"16f6792d8e8ccd3989e04e04b8148d3be61199c7e43d402bc6eeb08cbe3fd917","input_tokens":53,"truncated":false},+ {"id":"massive-3241-noul","group_id":"massive-alarm-b22b203130c5192a6423","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"fc6c5b8d2f44d273695f46d9a43719cadeb39c36eacba4524af59eb3c7f4767b","option_order":["false","true"],"gold":"true","raw_logits":[-2.30078125,1.1953125],"temperature":1.0,"type":"noul","source_id":"3241","source_row_hash":"16f6792d8e8ccd3989e04e04b8148d3be61199c7e43d402bc6eeb08cbe3fd917","input_tokens":48,"truncated":false},+ {"id":"massive-3271-choice","group_id":"massive-alarm-cc4399c7e1fe9bedaecc","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"b2fb2fd52dd7d6293ffb2ce75dfa9d145afb3da928c3f35cb90089a75a41290c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.455078125,-4.16015625,1.923828125],"temperature":1.0,"type":"choice","source_id":"3271","source_row_hash":"3d4b1654d3459ec3a62cc33209806f071eb0d809cc71d0fb51be19014a8b13d9","input_tokens":51,"truncated":false},+ {"id":"massive-3271-noul","group_id":"massive-alarm-cc4399c7e1fe9bedaecc","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"304b596915cb7afe22d5674667661a63184a832a5d2bd367bc730d199d2cd0ba","option_order":["false","true"],"gold":"false","raw_logits":[-2.5078125,-0.36181640625],"temperature":1.0,"type":"noul","source_id":"3271","source_row_hash":"3d4b1654d3459ec3a62cc33209806f071eb0d809cc71d0fb51be19014a8b13d9","input_tokens":46,"truncated":false},+ {"id":"massive-3313-choice","group_id":"massive-alarm-9624abc57fb5f9491c3c","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a63a16cad37a46fa428b72311b35fdecf3557e049597fe79ffbc0e23619e92ac","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.78125,-4.37890625,-4.58203125],"temperature":1.0,"type":"choice","source_id":"3313","source_row_hash":"25d5d82e30b677564c58a29305e0946447be07c0985598b902c051ef1fb1ea80","input_tokens":60,"truncated":false},+ {"id":"massive-3313-noul","group_id":"massive-alarm-9624abc57fb5f9491c3c","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"62a66e214884f42fa03ddc5d6e062bb3657c1418478a8e7b625c87853a0ee453","option_order":["false","true"],"gold":"true","raw_logits":[-2.55078125,1.4404296875],"temperature":1.0,"type":"noul","source_id":"3313","source_row_hash":"25d5d82e30b677564c58a29305e0946447be07c0985598b902c051ef1fb1ea80","input_tokens":55,"truncated":false},+ {"id":"massive-3344-choice","group_id":"massive-alarm-2124aad0708d0ec9337e","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"9748505b1055a8e4b9b580e4f065f1fb025a351d0316f2664fe9fb467aa20b01","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.60205078125,-1.8876953125,-0.740234375],"temperature":1.0,"type":"choice","source_id":"3344","source_row_hash":"503f2378950bb6b7542b8fd34d1781fa9a81f3da8e533bc3e5474ee2e90182a6","input_tokens":51,"truncated":false},+ {"id":"massive-3344-noul","group_id":"massive-alarm-2124aad0708d0ec9337e","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"40547d94b9391018d836f7c33cb83a9888d4a349795a9961a9c4d63bd0f9b699","option_order":["false","true"],"gold":"false","raw_logits":[-1.6083984375,0.9453125],"temperature":1.0,"type":"noul","source_id":"3344","source_row_hash":"503f2378950bb6b7542b8fd34d1781fa9a81f3da8e533bc3e5474ee2e90182a6","input_tokens":46,"truncated":false},+ {"id":"massive-3461-choice","group_id":"massive-alarm-ce0cb4819c1987861556","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"73431ea9804730e590ddc48be8d2c8035bb0dbeb45e6c1cfa2b74cddf9289334","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.64453125,-3.767578125,-3.158203125],"temperature":1.0,"type":"choice","source_id":"3461","source_row_hash":"0e09b286b7b15ee40d39a9dc76bc5d58004ddbae3898a25e3673897d633f6fd7","input_tokens":54,"truncated":false},+ {"id":"massive-3461-noul","group_id":"massive-alarm-ce0cb4819c1987861556","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"9c083d9ba471c24cd29e87a06ee702b6f461f167c9e5f0040a36737925acfc5b","option_order":["false","true"],"gold":"true","raw_logits":[-2.619140625,1.3740234375],"temperature":1.0,"type":"noul","source_id":"3461","source_row_hash":"0e09b286b7b15ee40d39a9dc76bc5d58004ddbae3898a25e3673897d633f6fd7","input_tokens":49,"truncated":false},+ {"id":"massive-3537-choice","group_id":"massive-alarm-1b97665270a474c9189b","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"58b2a286faf22ace00cb5b5322c9207f9304330fc3153fe79b6e308492cccfb8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.4990234375,-3.5234375,2.623046875],"temperature":1.0,"type":"choice","source_id":"3537","source_row_hash":"3bb0e1d5d6780212fc57f7c5bdc635e4f3b8af60e8f05a8864c0e0801b159e28","input_tokens":50,"truncated":false},+ {"id":"massive-3537-noul","group_id":"massive-alarm-1b97665270a474c9189b","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"27a1f6fa962a3cd311f0a1ded2227f879f7d3f63afd038278229abc437872012","option_order":["false","true"],"gold":"false","raw_logits":[-2.248046875,0.375244140625],"temperature":1.0,"type":"noul","source_id":"3537","source_row_hash":"3bb0e1d5d6780212fc57f7c5bdc635e4f3b8af60e8f05a8864c0e0801b159e28","input_tokens":45,"truncated":false},+ {"id":"massive-3696-choice","group_id":"massive-alarm-66bb02dcd4dfec9a31da","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"5c799de6dc9b83c02640969e26d2a2b4abd09017c1da008663ae386f27a420dc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.1328125,-3.171875,0.63134765625],"temperature":1.0,"type":"choice","source_id":"3696","source_row_hash":"50b44cd61dd0112bec73547e36d51b18ec64f7f88e01642a88fe65e66aaf8019","input_tokens":57,"truncated":false},+ {"id":"massive-3696-noul","group_id":"massive-alarm-66bb02dcd4dfec9a31da","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"83b6f74e7c01aed19a8a603c7b23d4c58da8a0282bc12b481546ed07d0dae1ca","option_order":["false","true"],"gold":"true","raw_logits":[-2.15625,1.0517578125],"temperature":1.0,"type":"noul","source_id":"3696","source_row_hash":"50b44cd61dd0112bec73547e36d51b18ec64f7f88e01642a88fe65e66aaf8019","input_tokens":52,"truncated":false},+ {"id":"massive-3760-choice","group_id":"massive-alarm-7f77ea61d28ffa0c4760","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a226efece8e4ca552b5c604d2c0e77f1e7ad8ef574882ccdf2abc1353914a530","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.5634765625,-1.9404296875,1.23046875],"temperature":1.0,"type":"choice","source_id":"3760","source_row_hash":"c58ac469fe7402eeee9373c32aed43cde55b7f6a576c78cf3e6cdbdcad81b62f","input_tokens":54,"truncated":false},+ {"id":"massive-3760-noul","group_id":"massive-alarm-7f77ea61d28ffa0c4760","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"a17c30741ef64b9a54cd5fa80638f0d977ace713acc174493612ac98af69da35","option_order":["false","true"],"gold":"true","raw_logits":[-2.052734375,1.4296875],"temperature":1.0,"type":"noul","source_id":"3760","source_row_hash":"c58ac469fe7402eeee9373c32aed43cde55b7f6a576c78cf3e6cdbdcad81b62f","input_tokens":49,"truncated":false},+ {"id":"massive-3948-choice","group_id":"massive-alarm-74196bc28bd235b5ebd4","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2244c648b7fdc8555facdb170e46c5ab59d024a693910e18c7b28b94ac0edb53","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.94140625,-4.47265625,-4.48828125],"temperature":1.0,"type":"choice","source_id":"3948","source_row_hash":"efdef6a7ca0afe5619c4c98377ee543a2c63c2f4a10c19d98cf9308324489779","input_tokens":52,"truncated":false},+ {"id":"massive-3948-noul","group_id":"massive-alarm-74196bc28bd235b5ebd4","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"72dd631c4fbc49146d876163d1ab74c4cb6eeb7076498c11a5a138878ca345bd","option_order":["false","true"],"gold":"true","raw_logits":[-2.017578125,1.310546875],"temperature":1.0,"type":"noul","source_id":"3948","source_row_hash":"efdef6a7ca0afe5619c4c98377ee543a2c63c2f4a10c19d98cf9308324489779","input_tokens":47,"truncated":false},+ {"id":"massive-3959-choice","group_id":"massive-alarm-1e2d1e22e2a448d0d4be","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3d05bb7983f196a0d3e77f5d1eda7f2de7d1eb9f65c25e6a5aa4985decca2ffc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.8291015625,-4.109375,-0.41552734375],"temperature":1.0,"type":"choice","source_id":"3959","source_row_hash":"7539f118e45280ccaa989666ca00e8b1765628ce7a944ffa526b60c3c58b91ab","input_tokens":52,"truncated":false},+ {"id":"massive-3959-noul","group_id":"massive-alarm-1e2d1e22e2a448d0d4be","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"0ca03992c0d8713a6590531ef8364a455a9ee6f7d7116d6a6262e593a496f42e","option_order":["false","true"],"gold":"true","raw_logits":[-1.388671875,1.0908203125],"temperature":1.0,"type":"noul","source_id":"3959","source_row_hash":"7539f118e45280ccaa989666ca00e8b1765628ce7a944ffa526b60c3c58b91ab","input_tokens":47,"truncated":false},+ {"id":"massive-3975-choice","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"e9c4661dc32b87b3f0762cd7a7254309cd7852b13a77514c93a4e4003111fe2f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.0711669921875,-2.5546875,-0.60888671875],"temperature":1.0,"type":"choice","source_id":"3975","source_row_hash":"c8c3d4b20f0116b687d7d684dba6fe8070f18bad470dc44ddbb7000c8c4ebaee","input_tokens":49,"truncated":false},+ {"id":"massive-3975-noul","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"df1f03cd870145a75b4e1403f5ec6464d4217ddf98f0e60c79cebc6dcf18f34c","option_order":["false","true"],"gold":"true","raw_logits":[-2.453125,1.7314453125],"temperature":1.0,"type":"noul","source_id":"3975","source_row_hash":"c8c3d4b20f0116b687d7d684dba6fe8070f18bad470dc44ddbb7000c8c4ebaee","input_tokens":44,"truncated":false},+ {"id":"massive-3985-choice","group_id":"massive-alarm-0eab4c88f4262e142df9","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a0ae587eb80ef7ddb6b9eb42df39de2690cd1df443b346b429870aa844881111","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.611328125,-4.12890625,-4.41015625],"temperature":1.0,"type":"choice","source_id":"3985","source_row_hash":"405084aa291615ce23004c5c83303d8418be167a30ba30ed939eb50f7b2e6b87","input_tokens":53,"truncated":false},+ {"id":"massive-3985-noul","group_id":"massive-alarm-0eab4c88f4262e142df9","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"ea1cfc0c358999cca571b97cb637ab166d0ca3225c812cf9c03f24cc95d8d626","option_order":["false","true"],"gold":"true","raw_logits":[-2.0390625,1.4208984375],"temperature":1.0,"type":"noul","source_id":"3985","source_row_hash":"405084aa291615ce23004c5c83303d8418be167a30ba30ed939eb50f7b2e6b87","input_tokens":48,"truncated":false},+ {"id":"massive-4016-choice","group_id":"massive-alarm-4be456711aa1993110b2","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"505ee765de12c09f3f828b85ca69cd4953e2191d4d8c4ee6a25189a83a760043","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.306640625,-4.37890625,4.83203125],"temperature":1.0,"type":"choice","source_id":"4016","source_row_hash":"eb5665c8fa5ae7c7dc5804205acfba089cd6c2936b540d0c86da0c3ee73ec92c","input_tokens":53,"truncated":false},+ {"id":"massive-4016-noul","group_id":"massive-alarm-4be456711aa1993110b2","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"d140649f39079e246dadf8169a7a97f6b2489328b4734f0c178f3de16660dba5","option_order":["false","true"],"gold":"false","raw_logits":[-1.740234375,0.7607421875],"temperature":1.0,"type":"noul","source_id":"4016","source_row_hash":"eb5665c8fa5ae7c7dc5804205acfba089cd6c2936b540d0c86da0c3ee73ec92c","input_tokens":48,"truncated":false},+ {"id":"massive-4108-choice","group_id":"massive-alarm-ba768fc0bd4d5e4b7d89","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"98479093baf492b301adaa3cc8ba52c0481115d2eca82b13c0b146b35bb9a417","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.076171875,1.240234375,-1.5771484375],"temperature":1.0,"type":"choice","source_id":"4108","source_row_hash":"442cf4064902074bab424f8a13b5da971239324a1ddb14ac9c1461459605890d","input_tokens":51,"truncated":false},+ {"id":"massive-4108-noul","group_id":"massive-alarm-ba768fc0bd4d5e4b7d89","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"85af1497981b267f36416626603b0919ffe18af189c37a8b46e7da2c551c5255","option_order":["false","true"],"gold":"false","raw_logits":[-1.4697265625,1.169921875],"temperature":1.0,"type":"noul","source_id":"4108","source_row_hash":"442cf4064902074bab424f8a13b5da971239324a1ddb14ac9c1461459605890d","input_tokens":46,"truncated":false},+ {"id":"massive-4122-choice","group_id":"massive-alarm-7b84a2782f58c91b2bae","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"acc9bcb43f252751dd15370f7a1f9f35ad22a6b1c088fb3e9b36ede8d965b270","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.61328125,-0.46728515625,-1.58203125],"temperature":1.0,"type":"choice","source_id":"4122","source_row_hash":"844fa5f915442e44ef3076cd0ce2150f380ac58e2b067d29b96f08c100a1d939","input_tokens":50,"truncated":false},+ {"id":"massive-4122-noul","group_id":"massive-alarm-7b84a2782f58c91b2bae","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"907aa9c5faa26b58d904e64f202bcce86d6c4f242c23186d9ee265ae4166ee9d","option_order":["false","true"],"gold":"false","raw_logits":[-1.4423828125,0.82763671875],"temperature":1.0,"type":"noul","source_id":"4122","source_row_hash":"844fa5f915442e44ef3076cd0ce2150f380ac58e2b067d29b96f08c100a1d939","input_tokens":45,"truncated":false},+ {"id":"massive-4144-choice","group_id":"massive-alarm-95d77d103fd5fab659a2","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3f8abfe30982bff102e8b189574ee14bbf601371a4c033c0ebc000c980e801f7","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[1.3994140625,-2.697265625,-0.350830078125],"temperature":1.0,"type":"choice","source_id":"4144","source_row_hash":"0fd0739e6946a24cc24d8339f712773ba73cdcb18125f66f4b1af0f6c56cb409","input_tokens":54,"truncated":false},+ {"id":"massive-4144-noul","group_id":"massive-alarm-95d77d103fd5fab659a2","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"bb33b9b4e1da7c9029b695525c094b5019ad962957ae1e33e14fe4d50ffc5925","option_order":["false","true"],"gold":"false","raw_logits":[-1.7705078125,0.91015625],"temperature":1.0,"type":"noul","source_id":"4144","source_row_hash":"0fd0739e6946a24cc24d8339f712773ba73cdcb18125f66f4b1af0f6c56cb409","input_tokens":49,"truncated":false},+ {"id":"massive-4200-choice","group_id":"massive-alarm-a56e43f47ec9a9386d07","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"6774a6320d42c290b80ddb390be36ac859ee6f15374b0bd6ff3fefd461823904","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.27734375,-4.15234375,1.6220703125],"temperature":1.0,"type":"choice","source_id":"4200","source_row_hash":"c355f735101213734ce53521f480d2713ee60fde1860e82542bfa9d357596c71","input_tokens":52,"truncated":false},+ {"id":"massive-4200-noul","group_id":"massive-alarm-a56e43f47ec9a9386d07","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c2d79b9b8cf8fe2e19bf991be938ccd39a5638a21c2c00469f612d6d93b32d30","option_order":["false","true"],"gold":"false","raw_logits":[-3.84375,0.4052734375],"temperature":1.0,"type":"noul","source_id":"4200","source_row_hash":"c355f735101213734ce53521f480d2713ee60fde1860e82542bfa9d357596c71","input_tokens":47,"truncated":false},+ {"id":"massive-4509-choice","group_id":"massive-alarm-d4f2a8b9df951f72e77a","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"31ad31c567248f3781353d8e1cc63b681d0d3e9c5bdda2d3fab14afc0a04b41a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.2138671875,-3.05078125,1.6796875],"temperature":1.0,"type":"choice","source_id":"4509","source_row_hash":"bf224c141975e01266dcf5ab02b2f22e5e961fe3acb83f2ff99c3f8f960284b2","input_tokens":51,"truncated":false},+ {"id":"massive-4509-noul","group_id":"massive-alarm-d4f2a8b9df951f72e77a","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"dbe036ea36070f090a10050a3982caa2a9e3a00a5640b7586cef8afc7276731c","option_order":["false","true"],"gold":"false","raw_logits":[-2.4609375,1.2763671875],"temperature":1.0,"type":"noul","source_id":"4509","source_row_hash":"bf224c141975e01266dcf5ab02b2f22e5e961fe3acb83f2ff99c3f8f960284b2","input_tokens":46,"truncated":false},+ {"id":"massive-4530-choice","group_id":"massive-alarm-c0492ef78107c7e23f86","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"4bcb70abe01980ab58b9e301d26da030dae938a5e9ae12f907c932fb5ffa09a8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.2783203125,-0.74560546875,-0.94482421875],"temperature":1.0,"type":"choice","source_id":"4530","source_row_hash":"3dc1bd3fdc52e2c231408ad4a522b21e4518cff1813ef183884a10d582345cee","input_tokens":55,"truncated":false},+ {"id":"massive-4530-noul","group_id":"massive-alarm-c0492ef78107c7e23f86","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"15ca7806e7ca42d6ce4a143c2712cc799fa9ccd81611e366e35693d2927483cb","option_order":["false","true"],"gold":"false","raw_logits":[-1.53125,0.697265625],"temperature":1.0,"type":"noul","source_id":"4530","source_row_hash":"3dc1bd3fdc52e2c231408ad4a522b21e4518cff1813ef183884a10d582345cee","input_tokens":50,"truncated":false},+ {"id":"massive-4585-choice","group_id":"massive-alarm-57ee6f9659a3c6835215","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"8e4c5a399d92bc0fee79a0b7c274ef67eb3dadc5c492cccc34f148aa6184c53d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.625,-3.06640625,-2.103515625],"temperature":1.0,"type":"choice","source_id":"4585","source_row_hash":"29cb3192dcfaf8dd3c91f8560e6956e6d62314b02d2c3a19b339b44e7f94808e","input_tokens":61,"truncated":false},+ {"id":"massive-4585-noul","group_id":"massive-alarm-57ee6f9659a3c6835215","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"5d41960f1abd888d11d48ec0e983393f7efd4ffdad5f9941815404c8ba47ba75","option_order":["false","true"],"gold":"false","raw_logits":[-1.8154296875,0.28125],"temperature":1.0,"type":"noul","source_id":"4585","source_row_hash":"29cb3192dcfaf8dd3c91f8560e6956e6d62314b02d2c3a19b339b44e7f94808e","input_tokens":56,"truncated":false},+ {"id":"massive-4593-choice","group_id":"massive-alarm-70c776b38b2c5f805dcd","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2b329bec09cbc6dbd489dba7bf5ba03c208c7c760c3b11dcae5beab03e9de1d9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.73828125,1.751953125,-0.728515625],"temperature":1.0,"type":"choice","source_id":"4593","source_row_hash":"573a25fd5fa0ccdcbdf4ebee81ca2c4bbaab53ba3cb1563b483549e39c6e6a7a","input_tokens":53,"truncated":false},+ {"id":"massive-4593-noul","group_id":"massive-alarm-70c776b38b2c5f805dcd","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c9d1252c93abe919faf36506ceee2a4d572863bd74e43eb30c6e82ad87f3f29e","option_order":["false","true"],"gold":"false","raw_logits":[-1.8310546875,0.95751953125],"temperature":1.0,"type":"noul","source_id":"4593","source_row_hash":"573a25fd5fa0ccdcbdf4ebee81ca2c4bbaab53ba3cb1563b483549e39c6e6a7a","input_tokens":48,"truncated":false},+ {"id":"massive-4744-choice","group_id":"massive-alarm-4328c171395268c31624","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d0d46e21b89af922fa963c14d2cecf2e2b20b70099999c4ad40900017de92075","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.828125,3.25,0.44384765625],"temperature":1.0,"type":"choice","source_id":"4744","source_row_hash":"a4566885577d6aab551b8f32aa9432224700f661cf64bebd0176e38f03274c83","input_tokens":54,"truncated":false},+ {"id":"massive-4744-noul","group_id":"massive-alarm-4328c171395268c31624","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"afae574d52cbd08a229a7819423ec0cbd438a7d7268bf06023f2f5e18648c7b9","option_order":["false","true"],"gold":"false","raw_logits":[-1.40234375,0.57763671875],"temperature":1.0,"type":"noul","source_id":"4744","source_row_hash":"a4566885577d6aab551b8f32aa9432224700f661cf64bebd0176e38f03274c83","input_tokens":49,"truncated":false},+ {"id":"massive-4929-choice","group_id":"massive-alarm-74a1feb1e119db1fb400","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"bc3b693c29ce78f19435be8dd99e9451443befc07587185de8be06a0574902de","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.08660888671875,-2.994140625,1.0556640625],"temperature":1.0,"type":"choice","source_id":"4929","source_row_hash":"53f4b87a6a89da6de1f9302d72950cd98c6c9ba1bc9d35a3b33b0a2e2c4769a1","input_tokens":53,"truncated":false},+ {"id":"massive-4929-noul","group_id":"massive-alarm-74a1feb1e119db1fb400","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"59f48bdeda5d59c536e5c4364cba1a673b4afed9d9b79f6d8eab2d3e95c2e25f","option_order":["false","true"],"gold":"true","raw_logits":[-2.052734375,1.2275390625],"temperature":1.0,"type":"noul","source_id":"4929","source_row_hash":"53f4b87a6a89da6de1f9302d72950cd98c6c9ba1bc9d35a3b33b0a2e2c4769a1","input_tokens":48,"truncated":false},+ {"id":"massive-5114-choice","group_id":"massive-alarm-7fdbaa1c48c1e3afefe1","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"43567ef7d5ff87f4c3710c3424ab5f2cb19756e69c4d0e44ed470313ce41347b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.53076171875,-0.1943359375,-0.9033203125],"temperature":1.0,"type":"choice","source_id":"5114","source_row_hash":"0429eec68e26dfecd519ba81f0c92d85793aad1c8d202737bb439666a411b485","input_tokens":54,"truncated":false},+ {"id":"massive-5114-noul","group_id":"massive-alarm-7fdbaa1c48c1e3afefe1","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"7e3f35cef593675bcafc126f93ea622d79fe105f82c4e8c9ab80d884ac2201e5","option_order":["false","true"],"gold":"false","raw_logits":[-1.6279296875,1.0166015625],"temperature":1.0,"type":"noul","source_id":"5114","source_row_hash":"0429eec68e26dfecd519ba81f0c92d85793aad1c8d202737bb439666a411b485","input_tokens":49,"truncated":false},+ {"id":"massive-5172-choice","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"4f6d1a6703c39c88cebaff032394052408cba7c60cc58c80bd70f6bfd4112ba9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.008392333984375,-2.896484375,-0.76806640625],"temperature":1.0,"type":"choice","source_id":"5172","source_row_hash":"25e5a31c20bb4859195c396f45f721767e4d6c8bb6eceaaf2c63dcfc20adeb64","input_tokens":52,"truncated":false},+ {"id":"massive-5172-noul","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"69e8607aaeebf1a1350d73f7ea71224817d665a684ac4f3cdfe8ec464ace63b8","option_order":["false","true"],"gold":"true","raw_logits":[-2.07421875,1.3017578125],"temperature":1.0,"type":"noul","source_id":"5172","source_row_hash":"25e5a31c20bb4859195c396f45f721767e4d6c8bb6eceaaf2c63dcfc20adeb64","input_tokens":47,"truncated":false},+ {"id":"massive-5210-choice","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"c668b88dd8e85f1b4e78bb29f8f80248aa67aab044cfc54d576c102766632bfa","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.8671875,-3.4296875,-3.0703125],"temperature":1.0,"type":"choice","source_id":"5210","source_row_hash":"fa1ecb16b6a4385a45785dd19132a817f21fff52934ee5a626a97c780f4ab078","input_tokens":53,"truncated":false},+ {"id":"massive-5210-noul","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"067fdd02a518f2dd87f537b24dabb02f1d8133c6bab8803ba201115d10f6d82d","option_order":["false","true"],"gold":"false","raw_logits":[-2.16015625,0.9296875],"temperature":1.0,"type":"noul","source_id":"5210","source_row_hash":"fa1ecb16b6a4385a45785dd19132a817f21fff52934ee5a626a97c780f4ab078","input_tokens":48,"truncated":false},+ {"id":"massive-5319-choice","group_id":"massive-alarm-322a8df31b5ea91267e0","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"720d5cc7c24bb6954e0c41cc16a40fb2b5124e4bb102f2aed64b7acc5df81817","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.56640625,-1.7998046875,-1.109375],"temperature":1.0,"type":"choice","source_id":"5319","source_row_hash":"2d07404bc4a0e9de04cd7b8305b4d7fb4db95c545a145cc3f8fb6ab7b5934529","input_tokens":51,"truncated":false},+ {"id":"massive-5319-noul","group_id":"massive-alarm-322a8df31b5ea91267e0","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"334afe5034e053d889e7e832d8fe0b2fd80448a8f579445964b27a15650a3933","option_order":["false","true"],"gold":"true","raw_logits":[-2.2421875,1.4140625],"temperature":1.0,"type":"noul","source_id":"5319","source_row_hash":"2d07404bc4a0e9de04cd7b8305b4d7fb4db95c545a145cc3f8fb6ab7b5934529","input_tokens":46,"truncated":false},+ {"id":"massive-535-choice","group_id":"massive-alarm-669fb7803ab072ceb0b5","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"10b4de6d460909b2303c9dab7a9e033a29d50076d3e4bb27e4a936a36786ce62","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.300048828125,-1.794921875,-0.435546875],"temperature":1.0,"type":"choice","source_id":"535","source_row_hash":"0e3fcc2ba0c2c5c5f577632298fa42814922bfd1141206f7474fd190dc965c7e","input_tokens":54,"truncated":false},+ {"id":"massive-535-noul","group_id":"massive-alarm-669fb7803ab072ceb0b5","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"99646e241c5c848f95da454dcbfd7698231492b08a38270a0c1651139e49f9d3","option_order":["false","true"],"gold":"true","raw_logits":[-0.666015625,0.058319091796875],"temperature":1.0,"type":"noul","source_id":"535","source_row_hash":"0e3fcc2ba0c2c5c5f577632298fa42814922bfd1141206f7474fd190dc965c7e","input_tokens":49,"truncated":false},+ {"id":"massive-5492-choice","group_id":"massive-alarm-5791810e78edbebec8b4","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"55954676941878d2afda77659529a13bd38cd2f009e9eee4eb759261ff47ac73","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.234375,-3.1796875,1.62109375],"temperature":1.0,"type":"choice","source_id":"5492","source_row_hash":"6b09ef0b32c674bcb40f7ad0ca704a196efe0ea8ce77472e2e3088c05bd29ed3","input_tokens":53,"truncated":false},+ {"id":"massive-5492-noul","group_id":"massive-alarm-5791810e78edbebec8b4","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c1cbc2786d808f88ca9a1d631a134993d9a3357acbad234c9de4ba4cff33ae20","option_order":["false","true"],"gold":"false","raw_logits":[-1.857421875,-0.35498046875],"temperature":1.0,"type":"noul","source_id":"5492","source_row_hash":"6b09ef0b32c674bcb40f7ad0ca704a196efe0ea8ce77472e2e3088c05bd29ed3","input_tokens":48,"truncated":false},+ {"id":"massive-5673-choice","group_id":"massive-alarm-fa15e151bbd8376bf727","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"4ca5a1fef0f2d153d7bd6110a0fc74ceca4b41ad79287ee6961d656f74828ddc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.291015625,-3.931640625,2.373046875],"temperature":1.0,"type":"choice","source_id":"5673","source_row_hash":"42f49991de76cd94ab5ee5743f3e57ae2a14f2fb657869b455a94bfbbd05648f","input_tokens":52,"truncated":false},+ {"id":"massive-5673-noul","group_id":"massive-alarm-fa15e151bbd8376bf727","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"d4cb011692189f194a6894f23e0488fa6be688c3166137780caf9244766206dc","option_order":["false","true"],"gold":"false","raw_logits":[-2.53125,0.496826171875],"temperature":1.0,"type":"noul","source_id":"5673","source_row_hash":"42f49991de76cd94ab5ee5743f3e57ae2a14f2fb657869b455a94bfbbd05648f","input_tokens":47,"truncated":false},+ {"id":"massive-5675-choice","group_id":"massive-alarm-3568a7b610cf5fedeafe","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"c22de0ed7a39f40c2c13850693773a456275d5726a53d6e53dcd7355931f7c7f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.67578125,-4.43359375,1.3515625],"temperature":1.0,"type":"choice","source_id":"5675","source_row_hash":"d4455f4931eebb0ae5c429523260bb0da106c1fb1f59166ff83aa27192fd603d","input_tokens":53,"truncated":false},+ {"id":"massive-5675-noul","group_id":"massive-alarm-3568a7b610cf5fedeafe","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"2e82b6e8cdeb672a76ee85b20814106355c89d5c9c981051cff180c0105ff9d3","option_order":["false","true"],"gold":"false","raw_logits":[-2.6015625,-0.00815582275390625],"temperature":1.0,"type":"noul","source_id":"5675","source_row_hash":"d4455f4931eebb0ae5c429523260bb0da106c1fb1f59166ff83aa27192fd603d","input_tokens":48,"truncated":false},+ {"id":"massive-5736-choice","group_id":"massive-alarm-cad8645cf0d1993997ce","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d4a111329224fc89c0ce98f064feba839677a403249e06d1d2a8f49cb285191b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.5673828125,1.474609375,-0.52197265625],"temperature":1.0,"type":"choice","source_id":"5736","source_row_hash":"70f64d8c12527bf3143b24720e5e124b9c7c43de9bf0b78aca2fe33336e7f60d","input_tokens":54,"truncated":false},+ {"id":"massive-5736-noul","group_id":"massive-alarm-cad8645cf0d1993997ce","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"cd7f7f33d54024ae047cc45e45135de22fddd92c2eff1bdc2cf57668f3ef0b9d","option_order":["false","true"],"gold":"false","raw_logits":[-0.8037109375,0.301025390625],"temperature":1.0,"type":"noul","source_id":"5736","source_row_hash":"70f64d8c12527bf3143b24720e5e124b9c7c43de9bf0b78aca2fe33336e7f60d","input_tokens":49,"truncated":false},+ {"id":"massive-5891-choice","group_id":"massive-alarm-58fbb3df6930e3f46619","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"10f3ea8840e48ecbe378b6635f27574b818a01d405ff5f5435cda368088af7a3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.654296875,-1.7041015625,-3.41796875],"temperature":1.0,"type":"choice","source_id":"5891","source_row_hash":"13a21672cc8d63f41b4c4f3be3e9ee23d26cf4d483b68904d54849e76a9572ed","input_tokens":55,"truncated":false},+ {"id":"massive-5891-noul","group_id":"massive-alarm-58fbb3df6930e3f46619","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"aebc66b1557724f2c8050b8b96f8bd79808521ed13b64bc0d421e5ddee9e2ed7","option_order":["false","true"],"gold":"false","raw_logits":[-2.353515625,0.875],"temperature":1.0,"type":"noul","source_id":"5891","source_row_hash":"13a21672cc8d63f41b4c4f3be3e9ee23d26cf4d483b68904d54849e76a9572ed","input_tokens":50,"truncated":false},+ {"id":"massive-5906-choice","group_id":"massive-alarm-70d28c8cc29115db7bf6","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d429a04be92b35783c5220b12fb0d69dc0cc209556a6e30d4ccab1ab1b6957e6","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.2734375,-3.880859375,-4.16015625],"temperature":1.0,"type":"choice","source_id":"5906","source_row_hash":"c3492f95295cde551179df79a12fe2b92977fb44481de9b512aa59ed0900a40f","input_tokens":52,"truncated":false},+ {"id":"massive-5906-noul","group_id":"massive-alarm-70d28c8cc29115db7bf6","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"73d6faa4a8c642c84be73a9055997aee319398417b64d904ff2cc240cdb18649","option_order":["false","true"],"gold":"true","raw_logits":[-2.685546875,1.5634765625],"temperature":1.0,"type":"noul","source_id":"5906","source_row_hash":"c3492f95295cde551179df79a12fe2b92977fb44481de9b512aa59ed0900a40f","input_tokens":47,"truncated":false},+ {"id":"massive-5981-choice","group_id":"massive-alarm-3c4995900541ca97b2c9","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2226fcac61f5f77ed8bcf1443cada6e714f0219501e5b999600332ddff8f436b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.36328125,-1.1357421875,-2.537109375],"temperature":1.0,"type":"choice","source_id":"5981","source_row_hash":"601253fff66fad5594ffa03232ae14805096eeb3edddfabc89f23fdf86061586","input_tokens":53,"truncated":false},+ {"id":"massive-5981-noul","group_id":"massive-alarm-3c4995900541ca97b2c9","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"757dbc799dab322bc0fdd47f72321799a7049226e9dd8e045289b95fd6aa6c52","option_order":["false","true"],"gold":"false","raw_logits":[-2.056640625,1.291015625],"temperature":1.0,"type":"noul","source_id":"5981","source_row_hash":"601253fff66fad5594ffa03232ae14805096eeb3edddfabc89f23fdf86061586","input_tokens":48,"truncated":false},+ {"id":"massive-60-choice","group_id":"massive-alarm-94ab087f60890852e463","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"dfc4bfc4a569b92c621afb5cc8c39261cd7db70bd6d3ee9f1fa4f709bc840b7f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.384765625,-2.48046875,1.759765625],"temperature":1.0,"type":"choice","source_id":"60","source_row_hash":"0a0aea9a63bd5ab159c323cdd5274587f022744194c0d7b63de7f75fc36db068","input_tokens":51,"truncated":false},+ {"id":"massive-60-noul","group_id":"massive-alarm-94ab087f60890852e463","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"8d2c25f4e5a2219cdb422d36d08c2f2c51bde324bd447a0785ba5716405cc86a","option_order":["false","true"],"gold":"false","raw_logits":[-3.03515625,1.7685546875],"temperature":1.0,"type":"noul","source_id":"60","source_row_hash":"0a0aea9a63bd5ab159c323cdd5274587f022744194c0d7b63de7f75fc36db068","input_tokens":46,"truncated":false},+ {"id":"massive-603-choice","group_id":"massive-alarm-2fe60ba4951e76d11029","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a2c3805f89c00f884c27440a1dc6ca929b7eeb892ce4ceada9d89cae658a68e2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.365234375,-4.1015625,0.50048828125],"temperature":1.0,"type":"choice","source_id":"603","source_row_hash":"c421b5916d3212e4b01829e3c79d953d94f710d02093a9cae17e52e58d84f7e8","input_tokens":53,"truncated":false},+ {"id":"massive-603-noul","group_id":"massive-alarm-2fe60ba4951e76d11029","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"16ed49a37ca746de796382b34b307c96bd23e13298e2da96f4ba2c4d43c40a1e","option_order":["false","true"],"gold":"true","raw_logits":[-0.6171875,0.98193359375],"temperature":1.0,"type":"noul","source_id":"603","source_row_hash":"c421b5916d3212e4b01829e3c79d953d94f710d02093a9cae17e52e58d84f7e8","input_tokens":48,"truncated":false},+ {"id":"massive-611-choice","group_id":"massive-alarm-8563c786dc56bc354169","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"7d47fe013fa4abd1bef6768aaa6e056adc5c74153c2b553727f66e3455fc540d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[7.2578125,-4.58203125,-4.5859375],"temperature":1.0,"type":"choice","source_id":"611","source_row_hash":"bcebdb7a1783441cd8df39b63bf7fcb15b183ef32d3ed586988011b72526dd5e","input_tokens":56,"truncated":false},+ {"id":"massive-611-noul","group_id":"massive-alarm-8563c786dc56bc354169","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"2c9aee36cc50bdb10c9f6773fa1550f9c16e930556d25a6985c09c8505a49de6","option_order":["false","true"],"gold":"true","raw_logits":[-2.419921875,0.92041015625],"temperature":1.0,"type":"noul","source_id":"611","source_row_hash":"bcebdb7a1783441cd8df39b63bf7fcb15b183ef32d3ed586988011b72526dd5e","input_tokens":51,"truncated":false},+ {"id":"massive-6161-choice","group_id":"massive-alarm-ce6e77c02022e073ef83","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a8c2d9d0d4c08ca16e412474a4c2e9403ffab98cbe9aa1d0e3d144dbc1b52a1a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.0234375,-3.31640625,1.15234375],"temperature":1.0,"type":"choice","source_id":"6161","source_row_hash":"3b9476d18f0671741201b748e7aae8b4bef4da841422e3890047663b8cfc6b58","input_tokens":49,"truncated":false},+ {"id":"massive-6161-noul","group_id":"massive-alarm-ce6e77c02022e073ef83","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"de9c4de3d736330a69d1500ab613cb567c5d8a860bdd585ae9492f5e000f903b","option_order":["false","true"],"gold":"false","raw_logits":[-1.6689453125,0.68359375],"temperature":1.0,"type":"noul","source_id":"6161","source_row_hash":"3b9476d18f0671741201b748e7aae8b4bef4da841422e3890047663b8cfc6b58","input_tokens":44,"truncated":false},+ {"id":"massive-7042-choice","group_id":"massive-alarm-d393bf328d9e34b61362","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"fbeee415bfc936dd946bd415f67e0202af1611d6e2e36b7a7c5975d736165f02","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.8984375,-4.5703125,-3.0546875],"temperature":1.0,"type":"choice","source_id":"7042","source_row_hash":"c88b745c72a0ae2239775dec1abf7adfa60007cd0d466769d82d72f70bfbfc16","input_tokens":55,"truncated":false},+ {"id":"massive-7042-noul","group_id":"massive-alarm-d393bf328d9e34b61362","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"df2947e1b837c93ad24d1ba998064dae52f9b9deb1ef572a8b7b2b5f12ea8595","option_order":["false","true"],"gold":"true","raw_logits":[-1.6767578125,0.1278076171875],"temperature":1.0,"type":"noul","source_id":"7042","source_row_hash":"c88b745c72a0ae2239775dec1abf7adfa60007cd0d466769d82d72f70bfbfc16","input_tokens":50,"truncated":false},+ {"id":"massive-7277-choice","group_id":"massive-alarm-debb470de19756d91704","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"24fbe87da2055f9d886b5876c64a68454a41f99da5507c2686870465a513f63e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.0390625,-3.416015625,-1.0498046875],"temperature":1.0,"type":"choice","source_id":"7277","source_row_hash":"aeae185a00d916458ae3ab6aa4bfff1cd313a140872c45d27d15d576fe0ca9bc","input_tokens":53,"truncated":false},+ {"id":"massive-7277-noul","group_id":"massive-alarm-debb470de19756d91704","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"05b9feb244fca27da4be4e37c2cf62fdd13a90d5927b297ec2467bee1ebcaf42","option_order":["false","true"],"gold":"true","raw_logits":[-2.09375,1.1689453125],"temperature":1.0,"type":"noul","source_id":"7277","source_row_hash":"aeae185a00d916458ae3ab6aa4bfff1cd313a140872c45d27d15d576fe0ca9bc","input_tokens":48,"truncated":false},+ {"id":"massive-7865-choice","group_id":"massive-alarm-d5d345224d210f90f9f5","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"8b0593a1fdf0d6c7ced2ac1338010fe12e88ba4292f508dc4866529053d416c3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.99609375,-3.4453125,-2.880859375],"temperature":1.0,"type":"choice","source_id":"7865","source_row_hash":"1d2893dc1fc8769432ee4ae1da4f838791060f36eeb8a5465921654fea2966a2","input_tokens":53,"truncated":false},+ {"id":"massive-7865-noul","group_id":"massive-alarm-d5d345224d210f90f9f5","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"ade30874b49054351cc4d461647189dd5673e5a4a54e10ba7a5316569792c804","option_order":["false","true"],"gold":"true","raw_logits":[-2.541015625,1.515625],"temperature":1.0,"type":"noul","source_id":"7865","source_row_hash":"1d2893dc1fc8769432ee4ae1da4f838791060f36eeb8a5465921654fea2966a2","input_tokens":48,"truncated":false},+ {"id":"massive-8252-choice","group_id":"massive-alarm-b0a22b8198f4993e1291","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"ba3abaddc76eb45141e707cb4d2cae8e7bbd610553b4c9cbfcb65bc0ee921400","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.5078125,-3.552734375,-1.6689453125],"temperature":1.0,"type":"choice","source_id":"8252","source_row_hash":"47b417240a61f167aba697da9695e406562d95fbcfa8317d40ff52b052c7329b","input_tokens":50,"truncated":false},+ {"id":"massive-8252-noul","group_id":"massive-alarm-b0a22b8198f4993e1291","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"f2f944c5c1af54dd0f1ea98d42d7a2f19324eb3f9c9be5ca3bba7a6701a567b6","option_order":["false","true"],"gold":"true","raw_logits":[-2.232421875,1.2470703125],"temperature":1.0,"type":"noul","source_id":"8252","source_row_hash":"47b417240a61f167aba697da9695e406562d95fbcfa8317d40ff52b052c7329b","input_tokens":45,"truncated":false},+ {"id":"massive-8374-choice","group_id":"massive-alarm-b2c391a264a7fc0e8d05","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a54daba394b794d67ecb3fb82069387e93349d371a89e7d827f80635326ef79b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.12408447265625,-2.046875,-0.1451416015625],"temperature":1.0,"type":"choice","source_id":"8374","source_row_hash":"f9b7880829291847e8117d7cc48838e1f2289986e30778ecb70f1426263e361f","input_tokens":53,"truncated":false},+ {"id":"massive-8374-noul","group_id":"massive-alarm-b2c391a264a7fc0e8d05","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"81070c78d0e7723baaaf3f84c1722d81c18fe2e65502cfa00afa997dffefb8b0","option_order":["false","true"],"gold":"true","raw_logits":[-0.9482421875,1.0],"temperature":1.0,"type":"noul","source_id":"8374","source_row_hash":"f9b7880829291847e8117d7cc48838e1f2289986e30778ecb70f1426263e361f","input_tokens":48,"truncated":false},+ {"id":"massive-838-choice","group_id":"massive-alarm-9c1aa24d8856b1a9b594","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2e7152c29865640d059a093638cf15c424a87c28b0fbe0919bfc7f40c72e596b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.068359375,-4.296875,1.1064453125],"temperature":1.0,"type":"choice","source_id":"838","source_row_hash":"0374f1afd3ddf573375d73c97e68b801e7847739ab120bddcddb5f24ddf9301f","input_tokens":52,"truncated":false},+ {"id":"massive-838-noul","group_id":"massive-alarm-9c1aa24d8856b1a9b594","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"5553f4f5fc0c112d6e39cc75d646adbf627ea97450649ed1d5fc19aef431748b","option_order":["false","true"],"gold":"false","raw_logits":[-1.46875,-0.341796875],"temperature":1.0,"type":"noul","source_id":"838","source_row_hash":"0374f1afd3ddf573375d73c97e68b801e7847739ab120bddcddb5f24ddf9301f","input_tokens":47,"truncated":false},+ {"id":"massive-8881-choice","group_id":"massive-alarm-a3097c674dea080a2da5","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d54982cbf06da42f2e2f43ca280cb876161678636d5bb6e8ffee7a0d3bb93e7a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.8544921875,-2.931640625,-2.2109375],"temperature":1.0,"type":"choice","source_id":"8881","source_row_hash":"b4a67a7ddf8a2064162255cea468172e3b6117bc5912380b2561538d15fc942f","input_tokens":55,"truncated":false},+ {"id":"massive-8881-noul","group_id":"massive-alarm-a3097c674dea080a2da5","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"158e662345ac34c7894aaae13fa572924b0d5f2caacf6e3b84764fb118746db5","option_order":["false","true"],"gold":"true","raw_logits":[-2.541015625,1.11328125],"temperature":1.0,"type":"noul","source_id":"8881","source_row_hash":"b4a67a7ddf8a2064162255cea468172e3b6117bc5912380b2561538d15fc942f","input_tokens":50,"truncated":false},+ {"id":"massive-8931-choice","group_id":"massive-alarm-db8555fe3197239b3206","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d980fa7dedacf5e72ae37665538e2793b4f022673a8d82eb2e41284118fa51d0","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.3046875,-2.083984375,-1.1513671875],"temperature":1.0,"type":"choice","source_id":"8931","source_row_hash":"8f70a2fc9af29a53f173113c232962f4df7836f331b2282b37caf1ccab49a781","input_tokens":52,"truncated":false},+ {"id":"massive-8931-noul","group_id":"massive-alarm-db8555fe3197239b3206","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"7ec3b962a5c5a78cb83727987c51167dc8c8d7c76e5cbc8931b315728f12ca47","option_order":["false","true"],"gold":"true","raw_logits":[-1.7548828125,1.1884765625],"temperature":1.0,"type":"noul","source_id":"8931","source_row_hash":"8f70a2fc9af29a53f173113c232962f4df7836f331b2282b37caf1ccab49a781","input_tokens":47,"truncated":false},+ {"id":"massive-9113-choice","group_id":"massive-alarm-397aba5493c3b2aa0106","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"cbf50a8bce382e1c8069c25f24d9187c0581bbf3a4557c96ba8e8c0bd4d094e4","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.126953125,-4.140625,0.7939453125],"temperature":1.0,"type":"choice","source_id":"9113","source_row_hash":"fb4976f4148eaf6a56d495a37fe34a0ea3cbc0cde2a2b924dfb2bb9d43e3fcc9","input_tokens":55,"truncated":false},+ {"id":"massive-9113-noul","group_id":"massive-alarm-397aba5493c3b2aa0106","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"1eb2651db8e5fb0ffe9c87a86a1eea9a93c803613b6109ed7b243775301390bc","option_order":["false","true"],"gold":"true","raw_logits":[-0.2366943359375,0.02410888671875],"temperature":1.0,"type":"noul","source_id":"9113","source_row_hash":"fb4976f4148eaf6a56d495a37fe34a0ea3cbc0cde2a2b924dfb2bb9d43e3fcc9","input_tokens":50,"truncated":false},+ {"id":"massive-919-choice","group_id":"massive-alarm-4c0a77d504ba07897c11","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1df3beb000463e7bc08b13bb810b877d176680c61686b53599cffc5b0d579beb","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.984375,-3.20703125,4.3359375],"temperature":1.0,"type":"choice","source_id":"919","source_row_hash":"b899f7728a260ed0af75e47ce729fb61bac2aa810bbdcc2b39e068dd142e1140","input_tokens":52,"truncated":false},+ {"id":"massive-919-noul","group_id":"massive-alarm-4c0a77d504ba07897c11","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"65b4e822aafcca31aac32995a5110244feba92743a197f2bb18d469b77384537","option_order":["false","true"],"gold":"false","raw_logits":[-1.474609375,0.272705078125],"temperature":1.0,"type":"noul","source_id":"919","source_row_hash":"b899f7728a260ed0af75e47ce729fb61bac2aa810bbdcc2b39e068dd142e1140","input_tokens":47,"truncated":false},+ {"id":"massive-0-choice","group_id":"massive-alarm-a37bbbb0764cc5bf8d8c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"77b0b32f31535aa2bdd9f6f747335478e4a43d19f2018d740cec1c1fa264a584","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.77197265625,-3.3671875,-0.1776123046875],"temperature":1.0,"type":"choice","source_id":"0","source_row_hash":"c7526effce043a65aeb540d2fd4ed2c638ea92efb82184525bfa946c59419162","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.77197265625,-3.3671875,-0.1776123046875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-0-noul","group_id":"massive-alarm-a37bbbb0764cc5bf8d8c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"35ff02590dfdcdf3d2f174132d6178af71b02166065f1ef047c1e306f4cae5f9","option_order":["false","true"],"gold":"true","raw_logits":[-1.6298828125,0.81640625],"temperature":1.0,"type":"noul","source_id":"0","source_row_hash":"c7526effce043a65aeb540d2fd4ed2c638ea92efb82184525bfa946c59419162","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.6298828125,0.81640625],"calibrated_temperature":5.0},+ {"id":"massive-1205-choice","group_id":"massive-alarm-46ec519726ad99259312","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c5336a0dbeaab10f6d75e474cd8ca690a4a17b614be1c36aa9bc422f3420b0e3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.78125,-4.015625,2.580078125],"temperature":1.0,"type":"choice","source_id":"1205","source_row_hash":"c6f272625d9dddb2157b268ff169f20764ccb2c15d8194a208dd4572661cf287","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-1.78125,-4.015625,2.580078125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1205-noul","group_id":"massive-alarm-46ec519726ad99259312","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ad37d94fa29dae7c5a19037407667cf3ec16629415f0f66c6d35b1bf2d99bd10","option_order":["false","true"],"gold":"false","raw_logits":[-1.56640625,0.179443359375],"temperature":1.0,"type":"noul","source_id":"1205","source_row_hash":"c6f272625d9dddb2157b268ff169f20764ccb2c15d8194a208dd4572661cf287","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.56640625,0.179443359375],"calibrated_temperature":5.0},+ {"id":"massive-1264-choice","group_id":"massive-alarm-81721bc34387bf04c532","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"5d1b1f024710e0ccc55d6f8617daa19c339f01457cfb0e6ce8f41ad766f25b55","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[1.2314453125,-1.3837890625,-0.91796875],"temperature":1.0,"type":"choice","source_id":"1264","source_row_hash":"b0e2b6e0b5645a28c5ef36a7b84a2f2c1967d0af1db72136cb632466ac4a2bde","input_tokens":53,"truncated":false,"calibrated_raw_logits":[1.2314453125,-1.3837890625,-0.91796875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1264-noul","group_id":"massive-alarm-81721bc34387bf04c532","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"132b85b5df1b397bcfc73c5408e767840171c95421a29b96e6f3ead9162b793b","option_order":["false","true"],"gold":"false","raw_logits":[-2.1015625,0.94873046875],"temperature":1.0,"type":"noul","source_id":"1264","source_row_hash":"b0e2b6e0b5645a28c5ef36a7b84a2f2c1967d0af1db72136cb632466ac4a2bde","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.1015625,0.94873046875],"calibrated_temperature":5.0},+ {"id":"massive-1452-choice","group_id":"massive-alarm-426bdf23716beff075f4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ba17e7a041b87e59af379c45db0671a59b60e5fa051c7555195d8a9da3aa2337","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.24609375,-3.59375,1.53515625],"temperature":1.0,"type":"choice","source_id":"1452","source_row_hash":"1bfe246bb5276edcdcb98bb0446151ac77373bd9a92f38c219359eb38e6f7682","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-3.24609375,-3.59375,1.53515625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1452-noul","group_id":"massive-alarm-426bdf23716beff075f4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"12849a0b88a8ff2a993968e6b0182681c95a4694070c5ccacee0ed5f2b6524a0","option_order":["false","true"],"gold":"false","raw_logits":[-1.767578125,0.370849609375],"temperature":1.0,"type":"noul","source_id":"1452","source_row_hash":"1bfe246bb5276edcdcb98bb0446151ac77373bd9a92f38c219359eb38e6f7682","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.767578125,0.370849609375],"calibrated_temperature":5.0},+ {"id":"massive-1454-choice","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ce41769466fb7357d3ccff960d17d722b3d6847a9cb11833b48f3c85dca068d5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.6337890625,-3.45703125,1.048828125],"temperature":1.0,"type":"choice","source_id":"1454","source_row_hash":"145608279739b62c2458975fa2f71167df19ca5eba37351354515f842abe0aee","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-1.6337890625,-3.45703125,1.048828125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1454-noul","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"792f47e0d63956a5accbee0429b61de43a74bda6d3f490cf082803c5b8f38138","option_order":["false","true"],"gold":"false","raw_logits":[-2.0078125,0.259521484375],"temperature":1.0,"type":"noul","source_id":"1454","source_row_hash":"145608279739b62c2458975fa2f71167df19ca5eba37351354515f842abe0aee","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.0078125,0.259521484375],"calibrated_temperature":5.0},+ {"id":"massive-1483-choice","group_id":"massive-alarm-2c21587cc19c72b110b3","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"f30db95f41dc4f0696ec72519901b57e25e672544e091d5cb2dc86ea24665f7e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[6.015625,-2.25390625,-2.994140625],"temperature":1.0,"type":"choice","source_id":"1483","source_row_hash":"890f145f8f9f0b5fd1d7091f1f3c2e12575f505e096fb255d1ead352d3634b9a","input_tokens":51,"truncated":false,"calibrated_raw_logits":[6.015625,-2.25390625,-2.994140625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1483-noul","group_id":"massive-alarm-2c21587cc19c72b110b3","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7ddce951250db5a4434c1ab2a0c87281815f837602586c62fb1730e47234864b","option_order":["false","true"],"gold":"false","raw_logits":[-1.54296875,0.85107421875],"temperature":1.0,"type":"noul","source_id":"1483","source_row_hash":"890f145f8f9f0b5fd1d7091f1f3c2e12575f505e096fb255d1ead352d3634b9a","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.54296875,0.85107421875],"calibrated_temperature":5.0},+ {"id":"massive-1635-choice","group_id":"massive-alarm-f9e0722a15d811b9f803","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"bb88865e7bc83986e78d070eaa64317d656ec97f98105b2ad26d92a38d668e08","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.658203125,0.01424407958984375,-2.0234375],"temperature":1.0,"type":"choice","source_id":"1635","source_row_hash":"d1953237537debb810d6f0e9c57dfd7c06257fcb558e7b2a16b137f96c4304d0","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.658203125,0.01424407958984375,-2.0234375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1635-noul","group_id":"massive-alarm-f9e0722a15d811b9f803","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ea73bed5f72a1db4db41433a4c84c6d710acf14be98e446f68d74d9b1dcdfd05","option_order":["false","true"],"gold":"false","raw_logits":[-1.5322265625,0.79248046875],"temperature":1.0,"type":"noul","source_id":"1635","source_row_hash":"d1953237537debb810d6f0e9c57dfd7c06257fcb558e7b2a16b137f96c4304d0","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.5322265625,0.79248046875],"calibrated_temperature":5.0},+ {"id":"massive-1646-choice","group_id":"massive-alarm-62749a68e3e345ba0baf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"64d7678b68d45c87ca45fc5bf39f26b3d45490f848ea4a4b5a760af9d994e293","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.42578125,-0.8388671875,-0.951171875],"temperature":1.0,"type":"choice","source_id":"1646","source_row_hash":"45be821060a8270e88285729f71d4851684eccb3f8e08dc3af41d53e22bd4a93","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.42578125,-0.8388671875,-0.951171875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1646-noul","group_id":"massive-alarm-62749a68e3e345ba0baf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fea796b609111eb3c000991a208e87211ae8c359a7cdc07e754bc56caa22d373","option_order":["false","true"],"gold":"false","raw_logits":[-2.109375,1.1884765625],"temperature":1.0,"type":"noul","source_id":"1646","source_row_hash":"45be821060a8270e88285729f71d4851684eccb3f8e08dc3af41d53e22bd4a93","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.109375,1.1884765625],"calibrated_temperature":5.0},+ {"id":"massive-1658-choice","group_id":"massive-alarm-364266652280dd7c910e","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0230e908a3762610b091476be9f4a05d8480c6e8cbaa6f719a31d29d363d5a58","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.298828125,-0.3212890625,-0.67919921875],"temperature":1.0,"type":"choice","source_id":"1658","source_row_hash":"93fa8ba611a374efaa1f7aaead3325da80c2330ae0ae622640f247c61e01d54b","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.298828125,-0.3212890625,-0.67919921875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1658-noul","group_id":"massive-alarm-364266652280dd7c910e","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"84713593dc3c74a749551857971c03d1a91d64f10e9b64a4c0f9dcde5e4a3ea7","option_order":["false","true"],"gold":"false","raw_logits":[-1.306640625,0.36962890625],"temperature":1.0,"type":"noul","source_id":"1658","source_row_hash":"93fa8ba611a374efaa1f7aaead3325da80c2330ae0ae622640f247c61e01d54b","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.306640625,0.36962890625],"calibrated_temperature":5.0},+ {"id":"massive-1675-choice","group_id":"massive-alarm-bf4ac6ee8fec3f0b558c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1466e3c3f90ac2ac30f3f4f1b0d94ead57c08f3d7497b6ef1f9ea9c4eac8711f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.48046875,-4.32421875,2.890625],"temperature":1.0,"type":"choice","source_id":"1675","source_row_hash":"4d208d74d0efe7e2c8ca898d3b22951bcde4612a05aa052742a78e2af0f54efa","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-3.48046875,-4.32421875,2.890625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1675-noul","group_id":"massive-alarm-bf4ac6ee8fec3f0b558c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"275b4e3cc0377df561399ef1fad6be14b832e24bd56aae9ad0a547a7d358c00e","option_order":["false","true"],"gold":"false","raw_logits":[-2.11328125,0.400146484375],"temperature":1.0,"type":"noul","source_id":"1675","source_row_hash":"4d208d74d0efe7e2c8ca898d3b22951bcde4612a05aa052742a78e2af0f54efa","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.11328125,0.400146484375],"calibrated_temperature":5.0},+ {"id":"massive-1719-choice","group_id":"massive-alarm-3aeb032bc1ab13756f0a","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8c5933a9aeb20b97adbd1e5b98c5cbd42e62e879e8e780ffa59c46e4d607a736","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.390625,-3.01171875,-1.6962890625],"temperature":1.0,"type":"choice","source_id":"1719","source_row_hash":"c920466329cad750fe4b3605a5197b2eb9ea1b411cb18b89cd499d0a209dd0da","input_tokens":56,"truncated":false,"calibrated_raw_logits":[4.390625,-3.01171875,-1.6962890625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1719-noul","group_id":"massive-alarm-3aeb032bc1ab13756f0a","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fd8b7628ff216af2d1a0577794f912897a4b1852599ad9d32b74f55fe23bfdc1","option_order":["false","true"],"gold":"true","raw_logits":[-1.40625,0.93798828125],"temperature":1.0,"type":"noul","source_id":"1719","source_row_hash":"c920466329cad750fe4b3605a5197b2eb9ea1b411cb18b89cd499d0a209dd0da","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.40625,0.93798828125],"calibrated_temperature":5.0},+ {"id":"massive-174-choice","group_id":"massive-alarm-ce0229b7c6149d882cf3","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7839990611efc7091e313b70ab12446f38e919bf3775e40e3288cb228c8e2b77","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.22265625,-2.859375,-1.3046875],"temperature":1.0,"type":"choice","source_id":"174","source_row_hash":"e8f8762e0d5998c6602731393d2fab6fdfb7662662c9a73a02e95041799cc0a9","input_tokens":54,"truncated":false,"calibrated_raw_logits":[4.22265625,-2.859375,-1.3046875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-174-noul","group_id":"massive-alarm-ce0229b7c6149d882cf3","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"46d0f88a1bb77721bef837b24d61db6244bac9678ca6744ac32fa010615a69c6","option_order":["false","true"],"gold":"true","raw_logits":[-2.4375,1.5888671875],"temperature":1.0,"type":"noul","source_id":"174","source_row_hash":"e8f8762e0d5998c6602731393d2fab6fdfb7662662c9a73a02e95041799cc0a9","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.4375,1.5888671875],"calibrated_temperature":5.0},+ {"id":"massive-1774-choice","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6b00cec1b29c2fa5a974ad219e812cc757b1e41937665cfacf265bb6c5a0a327","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.6337890625,-3.45703125,1.048828125],"temperature":1.0,"type":"choice","source_id":"1774","source_row_hash":"e2d9115544309a1c262318bb19f1d82615d1bb4704ad53d26c3bb5265d1cd2f0","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-1.6337890625,-3.45703125,1.048828125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1774-noul","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"10fce9ced8e6eec5b8b870dbf1b532e302afac2b8bc85d93119e264ea2ffebe3","option_order":["false","true"],"gold":"false","raw_logits":[-2.0078125,0.259521484375],"temperature":1.0,"type":"noul","source_id":"1774","source_row_hash":"e2d9115544309a1c262318bb19f1d82615d1bb4704ad53d26c3bb5265d1cd2f0","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.0078125,0.259521484375],"calibrated_temperature":5.0},+ {"id":"massive-1862-choice","group_id":"massive-alarm-869c1474b1c82e78b00c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"81e679b9cc2bda219fc616a52ccb9d7eecdad4f0c7350f44c21a5b12e38f9f44","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.253173828125,-0.1468505859375,-0.475341796875],"temperature":1.0,"type":"choice","source_id":"1862","source_row_hash":"d7dcb916b78b9417322277ed5947fe9386a9b6a258ec5d7237bf0a8db3eab495","input_tokens":51,"truncated":false,"calibrated_raw_logits":[0.253173828125,-0.1468505859375,-0.475341796875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1862-noul","group_id":"massive-alarm-869c1474b1c82e78b00c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"03ca4bdaf3b5b365cb529d8d6fcd255ea2c26a37f058442867ac3939e66adcf1","option_order":["false","true"],"gold":"false","raw_logits":[-1.8515625,1.146484375],"temperature":1.0,"type":"noul","source_id":"1862","source_row_hash":"d7dcb916b78b9417322277ed5947fe9386a9b6a258ec5d7237bf0a8db3eab495","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.8515625,1.146484375],"calibrated_temperature":5.0},+ {"id":"massive-1913-choice","group_id":"massive-alarm-04a340be86134fb825f2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0c21d545d8a862238e988f7cb58faa8ea1930a3d9adddb2d93adc69289d959ef","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.48974609375,-2.974609375,-0.37744140625],"temperature":1.0,"type":"choice","source_id":"1913","source_row_hash":"fd24179d6c9dc537d667cb6ff6fe25ab9e00652ce36f045a39da3a2eb3efeeae","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-0.48974609375,-2.974609375,-0.37744140625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1913-noul","group_id":"massive-alarm-04a340be86134fb825f2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f821536cce50ca455cf0e83aa0589f117ccdc48ac96ee7b0de3c0e3f00fe5885","option_order":["false","true"],"gold":"true","raw_logits":[-2.65234375,1.6552734375],"temperature":1.0,"type":"noul","source_id":"1913","source_row_hash":"fd24179d6c9dc537d667cb6ff6fe25ab9e00652ce36f045a39da3a2eb3efeeae","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.65234375,1.6552734375],"calibrated_temperature":5.0},+ {"id":"massive-1982-choice","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"046ef58708776eb61d36aba5282ff98022582353215e6eae4cb389226c6c9ff2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.10546875,-3.75,-2.833984375],"temperature":1.0,"type":"choice","source_id":"1982","source_row_hash":"68a92316a11c74af3d569a87cc55e8d8e9f116b771db9601d3c65b44d659be25","input_tokens":55,"truncated":false,"calibrated_raw_logits":[5.10546875,-3.75,-2.833984375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-1982-noul","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"6f13803fd13041a079ba5f448a57ab23e1d609a8fd17f4c9b68635b312890988","option_order":["false","true"],"gold":"true","raw_logits":[-2.30859375,2.40625],"temperature":1.0,"type":"noul","source_id":"1982","source_row_hash":"68a92316a11c74af3d569a87cc55e8d8e9f116b771db9601d3c65b44d659be25","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-2.30859375,2.40625],"calibrated_temperature":5.0},+ {"id":"massive-201-choice","group_id":"massive-alarm-34510ceb5116eee89abf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1919418590df3569bb082e98b9368556569025bf81fd8c5c80d0f62fb8012279","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.8603515625,-2.9609375,3.517578125],"temperature":1.0,"type":"choice","source_id":"201","source_row_hash":"fec125430204fc240f59fa0b8afc5ab33fb231589e377ca142563556d36313c1","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.8603515625,-2.9609375,3.517578125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-201-noul","group_id":"massive-alarm-34510ceb5116eee89abf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e682c7af0260299ab1b72d73ba69134a323fd6c0a4488f9e8ebb899d12e2f95c","option_order":["false","true"],"gold":"false","raw_logits":[-2.462890625,0.419921875],"temperature":1.0,"type":"noul","source_id":"201","source_row_hash":"fec125430204fc240f59fa0b8afc5ab33fb231589e377ca142563556d36313c1","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.462890625,0.419921875],"calibrated_temperature":5.0},+ {"id":"massive-2134-choice","group_id":"massive-alarm-f8e7f38a4927528ba9b2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8cb6e812da09a75b6e72019b09c11e0a85c207aa5c8dbfddc570fd92d30458de","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.0035247802734375,-3.1875,-0.9208984375],"temperature":1.0,"type":"choice","source_id":"2134","source_row_hash":"7d7e16a4e1ed61a8de9442d3e01f17fafdc055b6548e09778de80844d7b5df78","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-0.0035247802734375,-3.1875,-0.9208984375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2134-noul","group_id":"massive-alarm-f8e7f38a4927528ba9b2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"9689620330e6aaa79089562fe3f74c74152b155ba69f48bbb97341c234944c63","option_order":["false","true"],"gold":"true","raw_logits":[-2.033203125,1.36328125],"temperature":1.0,"type":"noul","source_id":"2134","source_row_hash":"7d7e16a4e1ed61a8de9442d3e01f17fafdc055b6548e09778de80844d7b5df78","input_tokens":43,"truncated":false,"calibrated_raw_logits":[-2.033203125,1.36328125],"calibrated_temperature":5.0},+ {"id":"massive-2166-choice","group_id":"massive-alarm-9519286825bfef55a4e0","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"69ab8ef1a844ffe7974da7b965063744250ea864ceb8d5a1efbdbdaa3022c36c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.90625,-4.01171875,-3.65234375],"temperature":1.0,"type":"choice","source_id":"2166","source_row_hash":"06505fac3fedd536d679acbcbc516ba66217322dc627d35a70fed7066fc76658","input_tokens":54,"truncated":false,"calibrated_raw_logits":[5.90625,-4.01171875,-3.65234375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2166-noul","group_id":"massive-alarm-9519286825bfef55a4e0","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"473a9f5612897736e32cafd3b6a9730e7740e1758fb072b71087bd827cdaf1b6","option_order":["false","true"],"gold":"true","raw_logits":[-2.603515625,1.41796875],"temperature":1.0,"type":"noul","source_id":"2166","source_row_hash":"06505fac3fedd536d679acbcbc516ba66217322dc627d35a70fed7066fc76658","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.603515625,1.41796875],"calibrated_temperature":5.0},+ {"id":"massive-2167-choice","group_id":"massive-alarm-d02ea237a24fb1becb66","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cde8b8d2e32f6683425e46e437ca44af31aa51bb38d8e675498c818dde0ec4d8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-1.3681640625,-3.48046875,0.047393798828125],"temperature":1.0,"type":"choice","source_id":"2167","source_row_hash":"bc3911b9a1c3926a1c45f4eb47de667f89e300ef4b21a13428a67c529a0efa24","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.3681640625,-3.48046875,0.047393798828125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2167-noul","group_id":"massive-alarm-d02ea237a24fb1becb66","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"83343991208595707f936ac6728585bd5637a32b678c39ae9802bf00e123d2f7","option_order":["false","true"],"gold":"true","raw_logits":[-1.54296875,1.0595703125],"temperature":1.0,"type":"noul","source_id":"2167","source_row_hash":"bc3911b9a1c3926a1c45f4eb47de667f89e300ef4b21a13428a67c529a0efa24","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.54296875,1.0595703125],"calibrated_temperature":5.0},+ {"id":"massive-2168-choice","group_id":"massive-alarm-cff75109d691bab986bf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7d2f0a6e8d4f34fa7bbf37a4a5aed96e8976d9c64cfe47bbc2c9660bdaaf199c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-0.76611328125,-3.515625,2.119140625],"temperature":1.0,"type":"choice","source_id":"2168","source_row_hash":"43fe92fb79734cdcfc39bb3d6f1df8e11c61fff1e834e468a137c272fb88ddb4","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-0.76611328125,-3.515625,2.119140625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2168-noul","group_id":"massive-alarm-cff75109d691bab986bf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"b6eb787be2412c61dd4e39b1920656d4a1bd9f081a93f2fea79db2f421bd3f37","option_order":["false","true"],"gold":"false","raw_logits":[-1.7783203125,0.79736328125],"temperature":1.0,"type":"noul","source_id":"2168","source_row_hash":"43fe92fb79734cdcfc39bb3d6f1df8e11c61fff1e834e468a137c272fb88ddb4","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.7783203125,0.79736328125],"calibrated_temperature":5.0},+ {"id":"massive-2334-choice","group_id":"massive-alarm-68e901c570a8bc247ea5","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"dad5cc765a8b621e7d161351f5c7d96e7990e74890f935c8da2bcc8f4465dcaf","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.072265625,-4.1484375,-0.0853271484375],"temperature":1.0,"type":"choice","source_id":"2334","source_row_hash":"37ed15aaa5179f417ed1db5f5e5fbf3fd407b5f570abad133bf0c99a1f824b8a","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.072265625,-4.1484375,-0.0853271484375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2334-noul","group_id":"massive-alarm-68e901c570a8bc247ea5","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"979a7e2f21c7c3cc727090ad2a84fe0ff99b663a18c7fa9599ef652f08bb46d2","option_order":["false","true"],"gold":"true","raw_logits":[-0.97021484375,1.2099609375],"temperature":1.0,"type":"noul","source_id":"2334","source_row_hash":"37ed15aaa5179f417ed1db5f5e5fbf3fd407b5f570abad133bf0c99a1f824b8a","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-0.97021484375,1.2099609375],"calibrated_temperature":5.0},+ {"id":"massive-2564-choice","group_id":"massive-alarm-606907415c29daf2edfb","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c44f45e943b2a653cda222a15ba4d17ed7ac9b9d5bae688e436eca315eaa7e16","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.66796875,-3.73046875,-2.88671875],"temperature":1.0,"type":"choice","source_id":"2564","source_row_hash":"95d1ba4090b52ba0ce5fb3444edef0250d596ad13f4f69a621c4a90930827e9f","input_tokens":56,"truncated":false,"calibrated_raw_logits":[4.66796875,-3.73046875,-2.88671875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2564-noul","group_id":"massive-alarm-606907415c29daf2edfb","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"87f3caebd55ff64483153128b3dc7a98c34a40ae17290470a5b5123949a92735","option_order":["false","true"],"gold":"true","raw_logits":[-2.638671875,1.4833984375],"temperature":1.0,"type":"noul","source_id":"2564","source_row_hash":"95d1ba4090b52ba0ce5fb3444edef0250d596ad13f4f69a621c4a90930827e9f","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.638671875,1.4833984375],"calibrated_temperature":5.0},+ {"id":"massive-2673-choice","group_id":"massive-alarm-8691ee9b0cf94dc2728b","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2e1c9599701f771117cd159e0ce73abb95b349419aacf2f200688397632c2c17","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.77294921875,-4.08203125,-2.396484375],"temperature":1.0,"type":"choice","source_id":"2673","source_row_hash":"1afaa7047e42a574d8e3230ec7c390cd32494a5058aadd8ceba92aa8d8381c72","input_tokens":58,"truncated":false,"calibrated_raw_logits":[-0.77294921875,-4.08203125,-2.396484375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2673-noul","group_id":"massive-alarm-8691ee9b0cf94dc2728b","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7a310326662b1ceb900e0ce2c672d76155807ff8771cd5e70b6705cdbf11793a","option_order":["false","true"],"gold":"false","raw_logits":[-1.7099609375,0.22412109375],"temperature":1.0,"type":"noul","source_id":"2673","source_row_hash":"1afaa7047e42a574d8e3230ec7c390cd32494a5058aadd8ceba92aa8d8381c72","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-1.7099609375,0.22412109375],"calibrated_temperature":5.0},+ {"id":"massive-2870-choice","group_id":"massive-alarm-8a430eb6516ce866e134","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ad6afdc347ef21ce011640d137edaa887c55f4963941c347b23f6b7b21a67dd2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.9970703125,4.4453125,-2.201171875],"temperature":1.0,"type":"choice","source_id":"2870","source_row_hash":"30d1945137ce312fac5d58fb2da24a2c2971e65f2403ad3ac1204ea1451af255","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-0.9970703125,4.4453125,-2.201171875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2870-noul","group_id":"massive-alarm-8a430eb6516ce866e134","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"9d2b0868a4c40cd0a685f0a6b8d495f5d7d98653eab390d7b4b1421dac4b84f1","option_order":["false","true"],"gold":"false","raw_logits":[-1.0615234375,0.373046875],"temperature":1.0,"type":"noul","source_id":"2870","source_row_hash":"30d1945137ce312fac5d58fb2da24a2c2971e65f2403ad3ac1204ea1451af255","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.0615234375,0.373046875],"calibrated_temperature":5.0},+ {"id":"massive-2957-choice","group_id":"massive-alarm-ee34ced4068dccdafbd4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2fc5caa4ae2985c5045322f402b6fbaec115c14788046392c7a1b7eb9e7278ee","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.16796875,-0.132080078125,-0.9990234375],"temperature":1.0,"type":"choice","source_id":"2957","source_row_hash":"cad2ff93096738a0cc95d453cd87af036fed8518b9c6804a76474027052392b2","input_tokens":52,"truncated":false,"calibrated_raw_logits":[4.16796875,-0.132080078125,-0.9990234375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2957-noul","group_id":"massive-alarm-ee34ced4068dccdafbd4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"c01ab9e555be8266d496a98398689b99aa6cb691f9d6a87897ebcd8dd5ee0a67","option_order":["false","true"],"gold":"false","raw_logits":[-1.783203125,1.5283203125],"temperature":1.0,"type":"noul","source_id":"2957","source_row_hash":"cad2ff93096738a0cc95d453cd87af036fed8518b9c6804a76474027052392b2","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.783203125,1.5283203125],"calibrated_temperature":5.0},+ {"id":"massive-2983-choice","group_id":"massive-alarm-80f7027d6c0b934a23f1","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"92c9f6b64792017733c2fa0d9e61f8d9579da249ea18cb26cc1de02a7df19f04","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.037109375,-3.71875,0.493408203125],"temperature":1.0,"type":"choice","source_id":"2983","source_row_hash":"51967e6f2526f9032885c83079a3b964f6d61de7941722049a83749bad8643b9","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-3.037109375,-3.71875,0.493408203125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2983-noul","group_id":"massive-alarm-80f7027d6c0b934a23f1","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a91925e0946afa49892ffde2b9cb42fe9240c5b085c14bff1f9663280a3b296f","option_order":["false","true"],"gold":"false","raw_logits":[-2.271484375,0.56787109375],"temperature":1.0,"type":"noul","source_id":"2983","source_row_hash":"51967e6f2526f9032885c83079a3b964f6d61de7941722049a83749bad8643b9","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.271484375,0.56787109375],"calibrated_temperature":5.0},+ {"id":"massive-2985-choice","group_id":"massive-alarm-cadeebcba526c5ce11e1","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c77bbecf7a4f64203fd3018a54e31f1f7e265dd20b4402e784ae564bf479b282","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.66796875,-4.046875,2.580078125],"temperature":1.0,"type":"choice","source_id":"2985","source_row_hash":"45de5ba6489c1472d66739caf602f19f3dddeb4654edb0633098362cb7245dc5","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-3.66796875,-4.046875,2.580078125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-2985-noul","group_id":"massive-alarm-cadeebcba526c5ce11e1","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ce7d0fa964fdf4b79e9e0090ed6dab1bf8d2a0ba07586c8de0979042fd97fabe","option_order":["false","true"],"gold":"false","raw_logits":[-2.65625,0.5087890625],"temperature":1.0,"type":"noul","source_id":"2985","source_row_hash":"45de5ba6489c1472d66739caf602f19f3dddeb4654edb0633098362cb7245dc5","input_tokens":43,"truncated":false,"calibrated_raw_logits":[-2.65625,0.5087890625],"calibrated_temperature":5.0},+ {"id":"massive-3161-choice","group_id":"massive-alarm-76d4b32f23cb9a972b27","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"70f06e64a39a6ef6fe0fc89ca848b8a2d66e40dbe52d56c3b6b62aa198f16381","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.1484375,2.0390625,-0.23193359375],"temperature":1.0,"type":"choice","source_id":"3161","source_row_hash":"21b811432a21e010c74460ef7e1b4573c17346e6fd3f6b869abdd9ef6ae0c1ef","input_tokens":56,"truncated":false,"calibrated_raw_logits":[2.1484375,2.0390625,-0.23193359375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3161-noul","group_id":"massive-alarm-76d4b32f23cb9a972b27","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"2bbbb3a8362746db393fa624e9043f8dad23e400182090154aade1921dc28ffb","option_order":["false","true"],"gold":"false","raw_logits":[-1.357421875,0.58740234375],"temperature":1.0,"type":"noul","source_id":"3161","source_row_hash":"21b811432a21e010c74460ef7e1b4573c17346e6fd3f6b869abdd9ef6ae0c1ef","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.357421875,0.58740234375],"calibrated_temperature":5.0},+ {"id":"massive-3162-choice","group_id":"massive-alarm-6e7b23cce190bc2d85c5","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ea7591a2f602f59dcb876b7ae44597587400890381314433fe0308a51e1f0d6d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.9912109375,-0.8330078125,-0.1859130859375],"temperature":1.0,"type":"choice","source_id":"3162","source_row_hash":"d15cb4a55b87aa0a3910749116e4161f2684c7546aa4c19dc3c4f99f0f375afe","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.9912109375,-0.8330078125,-0.1859130859375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3162-noul","group_id":"massive-alarm-6e7b23cce190bc2d85c5","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"de8da33771b147c3186d74aa65038831e1cb9ca49cd32cbe4c043568bcde65ae","option_order":["false","true"],"gold":"false","raw_logits":[-1.0283203125,-0.179443359375],"temperature":1.0,"type":"noul","source_id":"3162","source_row_hash":"d15cb4a55b87aa0a3910749116e4161f2684c7546aa4c19dc3c4f99f0f375afe","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.0283203125,-0.179443359375],"calibrated_temperature":5.0},+ {"id":"massive-3213-choice","group_id":"massive-alarm-130f5b17e0470747d338","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0799db4758f49463128dbabd40c943aed44fcc559c87572c05e7ab6d128b4789","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.97900390625,-3.73828125,-0.10302734375],"temperature":1.0,"type":"choice","source_id":"3213","source_row_hash":"9fd345a0d9dc2aed398ed21031246ff893fe651a70a0d745ba90398f720b9acb","input_tokens":53,"truncated":false,"calibrated_raw_logits":[0.97900390625,-3.73828125,-0.10302734375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3213-noul","group_id":"massive-alarm-130f5b17e0470747d338","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f43c35f39a7dc113b60b52e5f60073b04f43737051ee83161483c56e9fe2cc57","option_order":["false","true"],"gold":"true","raw_logits":[-2.564453125,1.2177734375],"temperature":1.0,"type":"noul","source_id":"3213","source_row_hash":"9fd345a0d9dc2aed398ed21031246ff893fe651a70a0d745ba90398f720b9acb","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.564453125,1.2177734375],"calibrated_temperature":5.0},+ {"id":"massive-3398-choice","group_id":"massive-alarm-da466cfc858133a8c89c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"dcb2df6d7f361c5165fff930443cdd8d616f8bfac79a1cfc8eef312ee5956872","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.0,-4.55859375,-3.73046875],"temperature":1.0,"type":"choice","source_id":"3398","source_row_hash":"bf8610ee8a05e9800dd90fb7461d52940fcfc1bf777b1aef8138784628240668","input_tokens":54,"truncated":false,"calibrated_raw_logits":[5.0,-4.55859375,-3.73046875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3398-noul","group_id":"massive-alarm-da466cfc858133a8c89c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f268081c1d5a7aa3525ee6e1106a49e8f95cde1b4b9f2cee63a2f1b497c0ca98","option_order":["false","true"],"gold":"true","raw_logits":[-1.8974609375,1.115234375],"temperature":1.0,"type":"noul","source_id":"3398","source_row_hash":"bf8610ee8a05e9800dd90fb7461d52940fcfc1bf777b1aef8138784628240668","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-1.8974609375,1.115234375],"calibrated_temperature":5.0},+ {"id":"massive-3435-choice","group_id":"massive-alarm-462210a662d596b5be80","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"fd24fd1ddefae756012ea9809480034da52e1590ecb9f855fea742309bb004ad","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.26171875,-3.630859375,1.640625],"temperature":1.0,"type":"choice","source_id":"3435","source_row_hash":"e0d15beb0c523aab28095c68f8a36e66e1f0ce73e302239dc482aad749d2fc21","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-3.26171875,-3.630859375,1.640625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3435-noul","group_id":"massive-alarm-462210a662d596b5be80","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"da919c2441ffa633bc40be48a80679231ee8f87a5a09323f8d1b0de2feb94ef9","option_order":["false","true"],"gold":"false","raw_logits":[-1.9365234375,0.5263671875],"temperature":1.0,"type":"noul","source_id":"3435","source_row_hash":"e0d15beb0c523aab28095c68f8a36e66e1f0ce73e302239dc482aad749d2fc21","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.9365234375,0.5263671875],"calibrated_temperature":5.0},+ {"id":"massive-3694-choice","group_id":"massive-alarm-ba350675a0496c49938e","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"f1f8330521612e78a2cde0c6656f2fba239789708588e8a80b2fc20b737e7845","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.71875,-4.5234375,-4.33984375],"temperature":1.0,"type":"choice","source_id":"3694","source_row_hash":"d1eb20fde014fb2b17be65874b712b3c63c6f59e4a2a59b4dcf2d38b52cce3b5","input_tokens":56,"truncated":false,"calibrated_raw_logits":[6.71875,-4.5234375,-4.33984375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3694-noul","group_id":"massive-alarm-ba350675a0496c49938e","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a62d58a5600dea097aa91ce2c7c14ed15568eee6c9bfd4c6389f8e279899de18","option_order":["false","true"],"gold":"true","raw_logits":[-2.154296875,1.7275390625],"temperature":1.0,"type":"noul","source_id":"3694","source_row_hash":"d1eb20fde014fb2b17be65874b712b3c63c6f59e4a2a59b4dcf2d38b52cce3b5","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.154296875,1.7275390625],"calibrated_temperature":5.0},+ {"id":"massive-3695-choice","group_id":"massive-alarm-b52edfb5a71ea1dcee30","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"202cb568bede45bcb290436753573690741c6ce9aa2bd2488fea63edefc3c144","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.623046875,1.001953125,1.2080078125],"temperature":1.0,"type":"choice","source_id":"3695","source_row_hash":"5f84198d661ad46d3f9fd8c6e1da9ee2f4afe6fa1c77ec4b8ecbb1a18e582de6","input_tokens":52,"truncated":false,"calibrated_raw_logits":[1.623046875,1.001953125,1.2080078125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3695-noul","group_id":"massive-alarm-b52edfb5a71ea1dcee30","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"96938fc7026fb8a5848686387930d87db6f09856773963de5c13c5adc8ceaf17","option_order":["false","true"],"gold":"true","raw_logits":[-1.865234375,1.1982421875],"temperature":1.0,"type":"noul","source_id":"3695","source_row_hash":"5f84198d661ad46d3f9fd8c6e1da9ee2f4afe6fa1c77ec4b8ecbb1a18e582de6","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.865234375,1.1982421875],"calibrated_temperature":5.0},+ {"id":"massive-3702-choice","group_id":"massive-alarm-d974ab158bb24355505f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7e3fea92fc16c779c9bda789dafacaa6be5cbe622e4f5560bfa3fdc82a74801e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[6.21484375,-4.44140625,-4.55859375],"temperature":1.0,"type":"choice","source_id":"3702","source_row_hash":"e618c546f5bea2ac0fe003c4c1dc9c54f82f1a3ab0ae22346f48031406b951c4","input_tokens":55,"truncated":false,"calibrated_raw_logits":[6.21484375,-4.44140625,-4.55859375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3702-noul","group_id":"massive-alarm-d974ab158bb24355505f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f74bf9955f1d89ea5586cb69e06a3ef816cc3cb2a5b7151d8273a1723839c8e2","option_order":["false","true"],"gold":"false","raw_logits":[-2.099609375,1.6357421875],"temperature":1.0,"type":"noul","source_id":"3702","source_row_hash":"e618c546f5bea2ac0fe003c4c1dc9c54f82f1a3ab0ae22346f48031406b951c4","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-2.099609375,1.6357421875],"calibrated_temperature":5.0},+ {"id":"massive-3761-choice","group_id":"massive-alarm-906532f048bd9c89f168","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0599547b574a8679aa3e878e382da40df6e16c9d94f84a30b2d769fbe4b85de0","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-1.2392578125,-3.041015625,-0.41552734375],"temperature":1.0,"type":"choice","source_id":"3761","source_row_hash":"ca9d72e81a4b3f3d7e2235562d7d91f1f303f58748ed6e425cb54f84d73dd906","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.2392578125,-3.041015625,-0.41552734375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3761-noul","group_id":"massive-alarm-906532f048bd9c89f168","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4c43d963e2942b0e4b14bba53a12af4bfc6c6e9765a4a95b0b98a038cb326f38","option_order":["false","true"],"gold":"true","raw_logits":[-1.900390625,1.345703125],"temperature":1.0,"type":"noul","source_id":"3761","source_row_hash":"ca9d72e81a4b3f3d7e2235562d7d91f1f303f58748ed6e425cb54f84d73dd906","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.900390625,1.345703125],"calibrated_temperature":5.0},+ {"id":"massive-3771-choice","group_id":"massive-alarm-c45435c75f999b77b736","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"25a21359b2b1ed7367720091eb1992eb1dad9c87afdf98e01b88ad12e7c8b6bd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.8818359375,3.08203125,-1.3564453125],"temperature":1.0,"type":"choice","source_id":"3771","source_row_hash":"2e4a7daa393af56cd536025155d462dbecd3c006ec8952136555253adea64973","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-0.8818359375,3.08203125,-1.3564453125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3771-noul","group_id":"massive-alarm-c45435c75f999b77b736","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"65677218b234245fd090491dad4d645564fa868f49e1c5ff60d2157d2d64e3d0","option_order":["false","true"],"gold":"false","raw_logits":[-1.66796875,0.7587890625],"temperature":1.0,"type":"noul","source_id":"3771","source_row_hash":"2e4a7daa393af56cd536025155d462dbecd3c006ec8952136555253adea64973","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.66796875,0.7587890625],"calibrated_temperature":5.0},+ {"id":"massive-3829-choice","group_id":"massive-alarm-f60b61ceed3880a795df","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7b86cc5b86e8345e60c6261aabec21980fd9dfa33baff9d168197fd184a36dec","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-0.89892578125,-4.5,3.939453125],"temperature":1.0,"type":"choice","source_id":"3829","source_row_hash":"cbdd61b9e23a630e30d582423ee8353b28ecafc5c9b93cb0b7c7d327d1e9ac5c","input_tokens":54,"truncated":false,"calibrated_raw_logits":[-0.89892578125,-4.5,3.939453125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3829-noul","group_id":"massive-alarm-f60b61ceed3880a795df","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"591cecb6a8d39597824aa757e6a24856a9a87ab456991e3c95b4abf9246b2b5c","option_order":["false","true"],"gold":"false","raw_logits":[-2.30078125,0.60546875],"temperature":1.0,"type":"noul","source_id":"3829","source_row_hash":"cbdd61b9e23a630e30d582423ee8353b28ecafc5c9b93cb0b7c7d327d1e9ac5c","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.30078125,0.60546875],"calibrated_temperature":5.0},+ {"id":"massive-3870-choice","group_id":"massive-alarm-cc3a484250d2efcc2043","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1fb7c02d1b63c62c515f630c6f6b1a190f806d8933fa807c965f98c542ea3734","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.9140625,0.95361328125,-0.422119140625],"temperature":1.0,"type":"choice","source_id":"3870","source_row_hash":"1eb812098d9c31e9b8c54e259a28eb60b0a677a306bbd79412ca17e0fd267621","input_tokens":56,"truncated":false,"calibrated_raw_logits":[2.9140625,0.95361328125,-0.422119140625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3870-noul","group_id":"massive-alarm-cc3a484250d2efcc2043","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4b85b5fb27bfdd2ab902c1aeebbb8681e8bba5cd4ab7ecb99502da2692776530","option_order":["false","true"],"gold":"false","raw_logits":[-2.1953125,1.1767578125],"temperature":1.0,"type":"noul","source_id":"3870","source_row_hash":"1eb812098d9c31e9b8c54e259a28eb60b0a677a306bbd79412ca17e0fd267621","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.1953125,1.1767578125],"calibrated_temperature":5.0},+ {"id":"massive-3884-choice","group_id":"massive-alarm-021f9f46cfb6e0326d91","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"607244bfee195714e4c96cab20b3b0e4c30e225e8835abe4fc93cbf89fd30998","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.5439453125,-0.671875,-2.466796875],"temperature":1.0,"type":"choice","source_id":"3884","source_row_hash":"51b89a83ff9ab846f795d6b36a0f992d19c4db4c0a9f3ed29cdd233b99bfa59f","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.5439453125,-0.671875,-2.466796875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3884-noul","group_id":"massive-alarm-021f9f46cfb6e0326d91","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"0481094ed9ba36cd8746ca74f5c7144fd855684f2e353ce3d0763a903f4d024d","option_order":["false","true"],"gold":"false","raw_logits":[-1.958984375,0.455322265625],"temperature":1.0,"type":"noul","source_id":"3884","source_row_hash":"51b89a83ff9ab846f795d6b36a0f992d19c4db4c0a9f3ed29cdd233b99bfa59f","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.958984375,0.455322265625],"calibrated_temperature":5.0},+ {"id":"massive-3886-choice","group_id":"massive-alarm-661b37ed9fdd982c94bf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"27c0ff2ef671d83abaebeff897b97cd739dc1770c284ea8bd367da824127f203","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.759765625,-1.4873046875,0.60546875],"temperature":1.0,"type":"choice","source_id":"3886","source_row_hash":"d6664d601ddccd6310963193b24469d0e9d111c0a7f3f23bf551b85d96113ccf","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-1.759765625,-1.4873046875,0.60546875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3886-noul","group_id":"massive-alarm-661b37ed9fdd982c94bf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"1fc4b67408e23e34752d030fee7275c22033746a1eaecef7e6277bffd34047f8","option_order":["false","true"],"gold":"false","raw_logits":[-1.8095703125,1.1162109375],"temperature":1.0,"type":"noul","source_id":"3886","source_row_hash":"d6664d601ddccd6310963193b24469d0e9d111c0a7f3f23bf551b85d96113ccf","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.8095703125,1.1162109375],"calibrated_temperature":5.0},+ {"id":"massive-3986-choice","group_id":"massive-alarm-f2693fd957a1a13798ca","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"d542a5fa84c401daab24a69a30c193f6a2a09bf9263e9bc7a51bf617211fd54c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.13671875,-3.943359375,-2.3984375],"temperature":1.0,"type":"choice","source_id":"3986","source_row_hash":"a52237c94225b1553a4b9b64b99a6de6c5747841054a315e869690c5c40c0042","input_tokens":51,"truncated":false,"calibrated_raw_logits":[6.13671875,-3.943359375,-2.3984375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-3986-noul","group_id":"massive-alarm-f2693fd957a1a13798ca","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"b8b2c739fdb3846f88484b30634ef774a8bf95921c0db9777d23d18c672698ad","option_order":["false","true"],"gold":"true","raw_logits":[-1.990234375,1.25390625],"temperature":1.0,"type":"noul","source_id":"3986","source_row_hash":"a52237c94225b1553a4b9b64b99a6de6c5747841054a315e869690c5c40c0042","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.990234375,1.25390625],"calibrated_temperature":5.0},+ {"id":"massive-399-choice","group_id":"massive-alarm-5064cd0ab19f1ced3b63","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7f7f324de0a5abcdfb147204c078ff9f5e6627709efd66a92c90f2f11eda84c8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.845703125,-2.98046875,2.666015625],"temperature":1.0,"type":"choice","source_id":"399","source_row_hash":"1c0573b1dc35d2d56f63d762ce3123e67d36e4fe7a90b405693688e9f3c0fb74","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-2.845703125,-2.98046875,2.666015625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-399-noul","group_id":"massive-alarm-5064cd0ab19f1ced3b63","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"3f557ad186f1a17d7172adf20667e35f775f5335dbae9db5de749c0a85692384","option_order":["false","true"],"gold":"false","raw_logits":[-1.7451171875,0.732421875],"temperature":1.0,"type":"noul","source_id":"399","source_row_hash":"1c0573b1dc35d2d56f63d762ce3123e67d36e4fe7a90b405693688e9f3c0fb74","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.7451171875,0.732421875],"calibrated_temperature":5.0},+ {"id":"massive-4017-choice","group_id":"massive-alarm-7bab44deabd44eeba1d4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c6aa025ee3f6243b63d4112e57528c1a0d09804b794a47a52f827cd845c22365","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.89453125,-3.54296875,1.318359375],"temperature":1.0,"type":"choice","source_id":"4017","source_row_hash":"b950ef4e3cfec6148b5f70b8cc2699e018e3443f68871089bb3b3bac77a915b3","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-2.89453125,-3.54296875,1.318359375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4017-noul","group_id":"massive-alarm-7bab44deabd44eeba1d4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"91e79a9efc33926738ab1aca86bf6652646cab200330672ca9fdd14ff4a13f95","option_order":["false","true"],"gold":"false","raw_logits":[-2.76953125,0.5654296875],"temperature":1.0,"type":"noul","source_id":"4017","source_row_hash":"b950ef4e3cfec6148b5f70b8cc2699e018e3443f68871089bb3b3bac77a915b3","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.76953125,0.5654296875],"calibrated_temperature":5.0},+ {"id":"massive-4081-choice","group_id":"massive-alarm-0fbfcc011e34434db270","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"96a5e44b26bf23a409304bc33c251790f9f8bebf6fc652e9b1b60e8941003ce4","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-1.23828125,-2.173828125,-0.37451171875],"temperature":1.0,"type":"choice","source_id":"4081","source_row_hash":"ed08495bac8110f45e50260d015d1d52615f67a00d5c47ffcd960745bd32f98f","input_tokens":60,"truncated":false,"calibrated_raw_logits":[-1.23828125,-2.173828125,-0.37451171875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4081-noul","group_id":"massive-alarm-0fbfcc011e34434db270","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"db72d9a98ad646aab2a2727af0b22a3dd95d9afb6b7053284680c34d1e22848b","option_order":["false","true"],"gold":"true","raw_logits":[-1.77734375,0.5439453125],"temperature":1.0,"type":"noul","source_id":"4081","source_row_hash":"ed08495bac8110f45e50260d015d1d52615f67a00d5c47ffcd960745bd32f98f","input_tokens":55,"truncated":false,"calibrated_raw_logits":[-1.77734375,0.5439453125],"calibrated_temperature":5.0},+ {"id":"massive-4121-choice","group_id":"massive-alarm-06d7dae0b6a5251a04b7","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"856f98d8094e873994e21182bb63f23d0513e3176c63e27cd351ac91cb60e697","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.19140625,-1.458984375,-1.919921875],"temperature":1.0,"type":"choice","source_id":"4121","source_row_hash":"905a75e6d0482805183ad0e81410418cc470ec3cb4cfabe1d76f9f30a337c809","input_tokens":52,"truncated":false,"calibrated_raw_logits":[4.19140625,-1.458984375,-1.919921875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4121-noul","group_id":"massive-alarm-06d7dae0b6a5251a04b7","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a09915d1ead2a4e4ef77bcfb9cfdc28ff092aab9317064470bf2d91fa130fcf9","option_order":["false","true"],"gold":"false","raw_logits":[-1.306640625,1.3212890625],"temperature":1.0,"type":"noul","source_id":"4121","source_row_hash":"905a75e6d0482805183ad0e81410418cc470ec3cb4cfabe1d76f9f30a337c809","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.306640625,1.3212890625],"calibrated_temperature":5.0},+ {"id":"massive-4130-choice","group_id":"massive-alarm-e36c4391c2490fb6ea69","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"9e20290617b2c369552819c32b78e1c50dadbb853aaa45e3fbaa353b5ac9a1e1","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.81005859375,-0.61181640625,-0.282958984375],"temperature":1.0,"type":"choice","source_id":"4130","source_row_hash":"45b0bb6e019be688e56cc7d5540bc815c6cc6845e6f17adfa152bee81fedec9a","input_tokens":55,"truncated":false,"calibrated_raw_logits":[0.81005859375,-0.61181640625,-0.282958984375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4130-noul","group_id":"massive-alarm-e36c4391c2490fb6ea69","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ca8193f9310a7214cfdea09f2f789252a72d64e3aa770830d4d71eb920257fc9","option_order":["false","true"],"gold":"false","raw_logits":[-0.470458984375,0.58203125],"temperature":1.0,"type":"noul","source_id":"4130","source_row_hash":"45b0bb6e019be688e56cc7d5540bc815c6cc6845e6f17adfa152bee81fedec9a","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-0.470458984375,0.58203125],"calibrated_temperature":5.0},+ {"id":"massive-4145-choice","group_id":"massive-alarm-193674ca7cd9463d8724","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"d3989f582d7f1cb6a535ec95163ad30df3e578d1b8610a327b515b423f69c862","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.52734375,-3.193359375,-2.767578125],"temperature":1.0,"type":"choice","source_id":"4145","source_row_hash":"4e06467ae4327b6843c2fd71dea717e128c9c5e776c173ba67f85c47f7caadb4","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.52734375,-3.193359375,-2.767578125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4145-noul","group_id":"massive-alarm-193674ca7cd9463d8724","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"eeda34b2c3c8364306cca7d520527d07f47fb1da66cbbb5627cff7d9a0127a78","option_order":["false","true"],"gold":"false","raw_logits":[-2.39453125,1.0224609375],"temperature":1.0,"type":"noul","source_id":"4145","source_row_hash":"4e06467ae4327b6843c2fd71dea717e128c9c5e776c173ba67f85c47f7caadb4","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.39453125,1.0224609375],"calibrated_temperature":5.0},+ {"id":"massive-4319-choice","group_id":"massive-alarm-3a6e1e04dfdb2826f3e6","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"59054d0457ed9b0e150ca8e4946d327901f4ca6957ecfd3ece87322ba3eac9a5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.406494140625,-2.6796875,-0.0236053466796875],"temperature":1.0,"type":"choice","source_id":"4319","source_row_hash":"f81d298a66e00e967f0e503432b25d08e7c32fae30fc39b9d68651d1c7e22915","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.406494140625,-2.6796875,-0.0236053466796875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4319-noul","group_id":"massive-alarm-3a6e1e04dfdb2826f3e6","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"aeeb6f93631dbebaa9ddf60f34c6b43d9dc7a176e9667f7b3efb94eb3d775fa2","option_order":["false","true"],"gold":"true","raw_logits":[-1.3876953125,0.8994140625],"temperature":1.0,"type":"noul","source_id":"4319","source_row_hash":"f81d298a66e00e967f0e503432b25d08e7c32fae30fc39b9d68651d1c7e22915","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.3876953125,0.8994140625],"calibrated_temperature":5.0},+ {"id":"massive-4375-choice","group_id":"massive-alarm-d0710f72b80d46f21bd2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"91da7137e905fecc5e2cdbd252323de33015b18aefbddd160effc10409a0a0ec","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.3505859375,0.5517578125,1.2265625],"temperature":1.0,"type":"choice","source_id":"4375","source_row_hash":"2d5824d9d29c077e250c5812fe3e84908d5bf837ce7bc903743164f8d11624dd","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.3505859375,0.5517578125,1.2265625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4375-noul","group_id":"massive-alarm-d0710f72b80d46f21bd2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fbbeae282b0e1b151a5b50a77f85d81ef6c08f60ec2811a7a2a773c2e789e581","option_order":["false","true"],"gold":"false","raw_logits":[-1.2119140625,0.95458984375],"temperature":1.0,"type":"noul","source_id":"4375","source_row_hash":"2d5824d9d29c077e250c5812fe3e84908d5bf837ce7bc903743164f8d11624dd","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.2119140625,0.95458984375],"calibrated_temperature":5.0},+ {"id":"massive-4654-choice","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"77cebc8da0a8189f2262bdd3a39f31d61fc360a7956bcd0f65463b08dc21bf4f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.2578125,-2.5625,-2.025390625],"temperature":1.0,"type":"choice","source_id":"4654","source_row_hash":"b7e67a9331d7f20bc014914c88b8be8ba51ce9bf981915ffe1d2138b91b6a01e","input_tokens":53,"truncated":false,"calibrated_raw_logits":[4.2578125,-2.5625,-2.025390625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4654-noul","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"456e9b4d2821b1b3662f84db099f4a9923f03141ad9b289aa4cb01e7b434fac1","option_order":["false","true"],"gold":"true","raw_logits":[-1.9423828125,1.6611328125],"temperature":1.0,"type":"noul","source_id":"4654","source_row_hash":"b7e67a9331d7f20bc014914c88b8be8ba51ce9bf981915ffe1d2138b91b6a01e","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.9423828125,1.6611328125],"calibrated_temperature":5.0},+ {"id":"massive-4673-choice","group_id":"massive-alarm-84fb06bfa21c395f5aaf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"bbc99315926deca3de03ece3f5e4184df6aa7262d5442d04979bf9dbf2a8cbd9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-2.296875,3.982421875,-0.8330078125],"temperature":1.0,"type":"choice","source_id":"4673","source_row_hash":"b9cefc1b3d178f9664ac742c0d5fd6c3541b41453ce33986bd45ded2e5baa606","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.296875,3.982421875,-0.8330078125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4673-noul","group_id":"massive-alarm-84fb06bfa21c395f5aaf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"29dce85d99aa74e32a858b60eb0de1ff518b8819f65f192a048517fb9a9d798b","option_order":["false","true"],"gold":"false","raw_logits":[-1.3115234375,0.343994140625],"temperature":1.0,"type":"noul","source_id":"4673","source_row_hash":"b9cefc1b3d178f9664ac742c0d5fd6c3541b41453ce33986bd45ded2e5baa606","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.3115234375,0.343994140625],"calibrated_temperature":5.0},+ {"id":"massive-4712-choice","group_id":"massive-alarm-2c6ac9f498036d649d66","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"934f5e684a6cbf814f1536b5b6b017245af8638cf21ae0290e64f1a32bd58c4c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.005859375,-3.384765625,-0.11474609375],"temperature":1.0,"type":"choice","source_id":"4712","source_row_hash":"4386c04043d8c28ace7ee145b5b749efc7ff6ad5efb0664e6ffb4a359864a2ba","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.005859375,-3.384765625,-0.11474609375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4712-noul","group_id":"massive-alarm-2c6ac9f498036d649d66","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"17794909495f150f505eb1cd6a46d9bcaa7531479f2f9d0165ee8b8193811d67","option_order":["false","true"],"gold":"false","raw_logits":[-1.8779296875,0.9365234375],"temperature":1.0,"type":"noul","source_id":"4712","source_row_hash":"4386c04043d8c28ace7ee145b5b749efc7ff6ad5efb0664e6ffb4a359864a2ba","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.8779296875,0.9365234375],"calibrated_temperature":5.0},+ {"id":"massive-4776-choice","group_id":"massive-alarm-44615da0c0274196323c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"5fec435b1670ec5cf5fb855e00e31fce4474fe0e0fc3105fcb24aeceaeaeb6b5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.238037109375,-1.462890625,-1.5185546875],"temperature":1.0,"type":"choice","source_id":"4776","source_row_hash":"81079ab4bf8ea6953a66c7c91e9a60e6fe22b3e698c003ffa5c411c0da2afe20","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.238037109375,-1.462890625,-1.5185546875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4776-noul","group_id":"massive-alarm-44615da0c0274196323c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"0fc3aba2af6c95f1c6f27d8149e96b826c80dd710f2427776cd09b5660c919cf","option_order":["false","true"],"gold":"false","raw_logits":[-1.685546875,1.1787109375],"temperature":1.0,"type":"noul","source_id":"4776","source_row_hash":"81079ab4bf8ea6953a66c7c91e9a60e6fe22b3e698c003ffa5c411c0da2afe20","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.685546875,1.1787109375],"calibrated_temperature":5.0},+ {"id":"massive-4848-choice","group_id":"massive-alarm-a708c3be70feb0cd7499","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7522ec4a0488ba7e56fa08b528c3df20f40223c6894ac0dc46d8bd184912bcb3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.416015625,-2.8828125,-0.599609375],"temperature":1.0,"type":"choice","source_id":"4848","source_row_hash":"b0b8d755300f645d4383b66847a7614fea92b600c73c2cf5542dff3eca2e27bd","input_tokens":50,"truncated":false,"calibrated_raw_logits":[0.416015625,-2.8828125,-0.599609375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4848-noul","group_id":"massive-alarm-a708c3be70feb0cd7499","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"956ba45262eca715bf5c90a06427ede7e3364a644ee12016e6da052866a0845b","option_order":["false","true"],"gold":"true","raw_logits":[-1.8486328125,0.95556640625],"temperature":1.0,"type":"noul","source_id":"4848","source_row_hash":"b0b8d755300f645d4383b66847a7614fea92b600c73c2cf5542dff3eca2e27bd","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.8486328125,0.95556640625],"calibrated_temperature":5.0},+ {"id":"massive-4851-choice","group_id":"massive-alarm-0fe1a1db9c374422445f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"fdeb0308d07e167e345d0809a6395c4969a801c33d5f80ca07ffed816020bbcb","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.86328125,-3.0859375,-2.537109375],"temperature":1.0,"type":"choice","source_id":"4851","source_row_hash":"9fe5af17055439dfa1b1610ca5fb2973b66459a1e0614af0e2452d2ce7a1e77c","input_tokens":50,"truncated":false,"calibrated_raw_logits":[4.86328125,-3.0859375,-2.537109375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4851-noul","group_id":"massive-alarm-0fe1a1db9c374422445f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"133b6950426a63d754fc346a4d151cbafe939b8e37e37fc290c101f6dcc77a0f","option_order":["false","true"],"gold":"false","raw_logits":[-2.02734375,1.529296875],"temperature":1.0,"type":"noul","source_id":"4851","source_row_hash":"9fe5af17055439dfa1b1610ca5fb2973b66459a1e0614af0e2452d2ce7a1e77c","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.02734375,1.529296875],"calibrated_temperature":5.0},+ {"id":"massive-4921-choice","group_id":"massive-alarm-5c308695c6a8b9f486cd","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"72d43a129cdaffcad2fbed561838f632af3ac96bca5743580ba0401a3665095e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.8505859375,-3.353515625,1.2998046875],"temperature":1.0,"type":"choice","source_id":"4921","source_row_hash":"2c2b240390fb93b6986489979cea8c3377a2c30ff81aab5abde369bc1a921d66","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.8505859375,-3.353515625,1.2998046875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4921-noul","group_id":"massive-alarm-5c308695c6a8b9f486cd","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4437a78e7f0d9e73db14372aa4f2b1cb1ee2f75571651b05dcb50bef8dac6af6","option_order":["false","true"],"gold":"false","raw_logits":[-2.416015625,0.92822265625],"temperature":1.0,"type":"noul","source_id":"4921","source_row_hash":"2c2b240390fb93b6986489979cea8c3377a2c30ff81aab5abde369bc1a921d66","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.416015625,0.92822265625],"calibrated_temperature":5.0},+ {"id":"massive-4930-choice","group_id":"massive-alarm-54d0fa25a96d8072f5b5","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cb8fffe4d7f15dd6507c6a0d7fa59fd180ba95cd1fb191f63aa8c1fe6aaf1b3b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.9716796875,-3.0546875,-0.66943359375],"temperature":1.0,"type":"choice","source_id":"4930","source_row_hash":"fd5035b32bf0e5c2a53512a0663af50295468117efba589c8277e039fbaace04","input_tokens":54,"truncated":false,"calibrated_raw_logits":[1.9716796875,-3.0546875,-0.66943359375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-4930-noul","group_id":"massive-alarm-54d0fa25a96d8072f5b5","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"5f559d4cf48d8ed51abf809f544bc6ed69e826262d2c3aa6a1980713386add37","option_order":["false","true"],"gold":"true","raw_logits":[-2.16796875,1.2373046875],"temperature":1.0,"type":"noul","source_id":"4930","source_row_hash":"fd5035b32bf0e5c2a53512a0663af50295468117efba589c8277e039fbaace04","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.16796875,1.2373046875],"calibrated_temperature":5.0},+ {"id":"massive-5006-choice","group_id":"massive-alarm-36c1ee0067291e81dfe0","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c0cf2686b16917d7ab80250b7f585134b8169f9fdc84e27e9c480875e60f94a2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.9462890625,1.2626953125,-2.009765625],"temperature":1.0,"type":"choice","source_id":"5006","source_row_hash":"c23192b2816c3f5f07c27870d69fdf383d4bc79de8202365cce80d52d65ef321","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.9462890625,1.2626953125,-2.009765625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5006-noul","group_id":"massive-alarm-36c1ee0067291e81dfe0","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4a7e732be2948711919ca7d7eafa8af5c50381df15405ffb995261ae054c1fd5","option_order":["false","true"],"gold":"false","raw_logits":[-1.046875,0.1993408203125],"temperature":1.0,"type":"noul","source_id":"5006","source_row_hash":"c23192b2816c3f5f07c27870d69fdf383d4bc79de8202365cce80d52d65ef321","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.046875,0.1993408203125],"calibrated_temperature":5.0},+ {"id":"massive-5043-choice","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"9209f20ab1cba0e22314b3adfb3a5459bc888eb018e2160a81d37579238baa7a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.076171875,-1.1318359375,-0.78173828125],"temperature":1.0,"type":"choice","source_id":"5043","source_row_hash":"837ce3dfa54a2caaa3feddb1add4699979a13624fa240aedd9d48eb841d7c3e3","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.076171875,-1.1318359375,-0.78173828125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5043-noul","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"24c1573dfcf63931fc5ed541e2ad4eb9e2eac99472bee9d3f85e0b3c80b7ee59","option_order":["false","true"],"gold":"true","raw_logits":[-2.505859375,1.9638671875],"temperature":1.0,"type":"noul","source_id":"5043","source_row_hash":"837ce3dfa54a2caaa3feddb1add4699979a13624fa240aedd9d48eb841d7c3e3","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.505859375,1.9638671875],"calibrated_temperature":5.0},+ {"id":"massive-5051-choice","group_id":"massive-alarm-16e090610da060f7511f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"e5fbc20b0a5aad3f0006a418d26781efe4e3d935dd3ae44b4fcc7219d7760b64","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.53125,-3.443359375,-0.4462890625],"temperature":1.0,"type":"choice","source_id":"5051","source_row_hash":"f3bbc1f439396e98e81c6c680dcba8b9fbe57a12e24a67d449622e4042bb9671","input_tokens":50,"truncated":false,"calibrated_raw_logits":[0.53125,-3.443359375,-0.4462890625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5051-noul","group_id":"massive-alarm-16e090610da060f7511f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"426104e055a8c3026abeb0f10d97c3466a4cfacf4037b56e3596670dc21c7368","option_order":["false","true"],"gold":"true","raw_logits":[-2.826171875,1.2958984375],"temperature":1.0,"type":"noul","source_id":"5051","source_row_hash":"f3bbc1f439396e98e81c6c680dcba8b9fbe57a12e24a67d449622e4042bb9671","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.826171875,1.2958984375],"calibrated_temperature":5.0},+ {"id":"massive-508-choice","group_id":"massive-alarm-41a421223f2457d4ae35","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2fc92d0e251aa21634ffc2b072bab5a2ca2442251f5e82ccf62197d9b4d9194d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.478515625,-3.337890625,-3.78515625],"temperature":1.0,"type":"choice","source_id":"508","source_row_hash":"71ecc0e574fd987cffcdeee6026d2de621df0225d0e6be68cf09db826a9bcd9b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[3.478515625,-3.337890625,-3.78515625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-508-noul","group_id":"massive-alarm-41a421223f2457d4ae35","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"61946bd8d19af979160d933e5b004a05b212d1f720b0866c522637f99f21ea7b","option_order":["false","true"],"gold":"false","raw_logits":[-2.60546875,0.544921875],"temperature":1.0,"type":"noul","source_id":"508","source_row_hash":"71ecc0e574fd987cffcdeee6026d2de621df0225d0e6be68cf09db826a9bcd9b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.60546875,0.544921875],"calibrated_temperature":5.0},+ {"id":"massive-5082-choice","group_id":"massive-alarm-dff7933fbae6d32c0fc9","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"65e59f1014104082d5ba399967ee15984ae0fa87f363cd154ec0f46b5075265f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.66650390625,-3.208984375,0.7880859375],"temperature":1.0,"type":"choice","source_id":"5082","source_row_hash":"eb3b740e165f04e6c1419581c82a67b0e075496a1bb875d944161266c0f98fd6","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.66650390625,-3.208984375,0.7880859375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5082-noul","group_id":"massive-alarm-dff7933fbae6d32c0fc9","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4d65d5fa341600065d5f523f2936bd98d886c1b3ddd05ef2f97bca7116d2413b","option_order":["false","true"],"gold":"true","raw_logits":[-2.341796875,1.470703125],"temperature":1.0,"type":"noul","source_id":"5082","source_row_hash":"eb3b740e165f04e6c1419581c82a67b0e075496a1bb875d944161266c0f98fd6","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.341796875,1.470703125],"calibrated_temperature":5.0},+ {"id":"massive-5103-choice","group_id":"massive-alarm-a0f523ed66d86dd66928","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1474862757e271b75a651b0a5c023ae431e3471fbd2b83e930cc010904b7f605","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.208984375,-4.21484375,2.35546875],"temperature":1.0,"type":"choice","source_id":"5103","source_row_hash":"5e4324ac53ee00213541d5386e0f8f520d4b7930fe26bd416545ed9b3681e81c","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-3.208984375,-4.21484375,2.35546875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5103-noul","group_id":"massive-alarm-a0f523ed66d86dd66928","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7cc1683f408cddf9993f59351a691ac692dcd2f2be18820e5d088f5d607d95e5","option_order":["false","true"],"gold":"false","raw_logits":[-1.99609375,-0.01267242431640625],"temperature":1.0,"type":"noul","source_id":"5103","source_row_hash":"5e4324ac53ee00213541d5386e0f8f520d4b7930fe26bd416545ed9b3681e81c","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.99609375,-0.01267242431640625],"calibrated_temperature":5.0},+ {"id":"massive-5116-choice","group_id":"massive-alarm-eef8eb3baf2d9be9f032","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ff5afaf8010c0ec76afa02463532a835bf618957bb435b423b572a34d1f72b9b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.78125,-1.4833984375,2.59765625],"temperature":1.0,"type":"choice","source_id":"5116","source_row_hash":"81e95101c290ce21984ebbc4b02d5abbd6cc8ca4e784b97182671444871de55b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.78125,-1.4833984375,2.59765625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5116-noul","group_id":"massive-alarm-eef8eb3baf2d9be9f032","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e55857f2a894f6be93794ed4131d1b24b3d1e25bd0fb009c6bed1068297afa36","option_order":["false","true"],"gold":"false","raw_logits":[-1.5234375,-0.00075531005859375],"temperature":1.0,"type":"noul","source_id":"5116","source_row_hash":"81e95101c290ce21984ebbc4b02d5abbd6cc8ca4e784b97182671444871de55b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.5234375,-0.00075531005859375],"calibrated_temperature":5.0},+ {"id":"massive-5122-choice","group_id":"massive-alarm-a7c3622f745708add50b","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"b8cecf6e8d365c8cca3910552fdf85dce9653479d4d50a00d9437b329657fca3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.7109375,-2.01953125,-0.6689453125],"temperature":1.0,"type":"choice","source_id":"5122","source_row_hash":"60429c8fba1d01a9b236a868780f5aab91fdfdfd7d7193d219e29dda4ff40f7b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.7109375,-2.01953125,-0.6689453125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5122-noul","group_id":"massive-alarm-a7c3622f745708add50b","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"3c3cccaa225b3031122a45bab64f407013dfce7331773fbb68584397b853f187","option_order":["false","true"],"gold":"false","raw_logits":[-1.8408203125,0.66845703125],"temperature":1.0,"type":"noul","source_id":"5122","source_row_hash":"60429c8fba1d01a9b236a868780f5aab91fdfdfd7d7193d219e29dda4ff40f7b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.8408203125,0.66845703125],"calibrated_temperature":5.0},+ {"id":"massive-5292-choice","group_id":"massive-alarm-7479d40399b64ecb695e","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6b263f0198363465a79d92000b9f4549cac9e8fdec4c19381966d4af03853c2e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.376953125,-4.47265625,-4.5078125],"temperature":1.0,"type":"choice","source_id":"5292","source_row_hash":"813c05e407bea752a81f88e33c115e1e14729209d2ef4bb63bb371d3d394049c","input_tokens":50,"truncated":false,"calibrated_raw_logits":[3.376953125,-4.47265625,-4.5078125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5292-noul","group_id":"massive-alarm-7479d40399b64ecb695e","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"2bae5052bc8cf5c358997f4b0612f1f6d00852a3884983f2e3b12c9cacfa0e9d","option_order":["false","true"],"gold":"true","raw_logits":[-3.390625,1.625],"temperature":1.0,"type":"noul","source_id":"5292","source_row_hash":"813c05e407bea752a81f88e33c115e1e14729209d2ef4bb63bb371d3d394049c","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-3.390625,1.625],"calibrated_temperature":5.0},+ {"id":"massive-533-choice","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"99a85134b83e2e949a3e0c54f4e95c4e40f8bf0a6b5f395df71cd8a3e029bdec","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.0703125,-4.25,-3.58203125],"temperature":1.0,"type":"choice","source_id":"533","source_row_hash":"eca3b5ac906fb8a35343d18eb18cad818f1db7770f69dcac2648292b16b0ffbc","input_tokens":52,"truncated":false,"calibrated_raw_logits":[6.0703125,-4.25,-3.58203125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-533-noul","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e0c12ffe2cd33b8d0a94ddaa9dab71028665b02ed35d0295366709686a915d67","option_order":["false","true"],"gold":"true","raw_logits":[-2.1328125,1.4267578125],"temperature":1.0,"type":"noul","source_id":"533","source_row_hash":"eca3b5ac906fb8a35343d18eb18cad818f1db7770f69dcac2648292b16b0ffbc","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.1328125,1.4267578125],"calibrated_temperature":5.0},+ {"id":"massive-5365-choice","group_id":"massive-alarm-d0dbaab04348ee02b2e4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7dae85cf680af0945a163fff2386a84f8d4b0fe67d9c437a5527258473b91288","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.703125,-3.419921875,-1.2119140625],"temperature":1.0,"type":"choice","source_id":"5365","source_row_hash":"3e646d241a20c6d18260b38f64e3d5b4045894baca11ab213b09d65fc04fd4d5","input_tokens":53,"truncated":false,"calibrated_raw_logits":[4.703125,-3.419921875,-1.2119140625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5365-noul","group_id":"massive-alarm-d0dbaab04348ee02b2e4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a3d43d9d9cfd2a79d27de7f606676fe8fd10e9ede4c75b655407327e83fe27ab","option_order":["false","true"],"gold":"true","raw_logits":[-1.537109375,1.2392578125],"temperature":1.0,"type":"noul","source_id":"5365","source_row_hash":"3e646d241a20c6d18260b38f64e3d5b4045894baca11ab213b09d65fc04fd4d5","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.537109375,1.2392578125],"calibrated_temperature":5.0},+ {"id":"massive-5475-choice","group_id":"massive-alarm-e7a25c599f07a27b07a8","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6ce27768ae10f0206a17038e7f7e3bd69bc0584f9ed44947f46347b4423df338","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.0107421875,-2.423828125,0.060760498046875],"temperature":1.0,"type":"choice","source_id":"5475","source_row_hash":"dc6c0165e59d0b35c1731ad73656915fbd6291deba80299bc358fc944a72978c","input_tokens":52,"truncated":false,"calibrated_raw_logits":[1.0107421875,-2.423828125,0.060760498046875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5475-noul","group_id":"massive-alarm-e7a25c599f07a27b07a8","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"264847e9ef6d6c6f317ea9dea8f6f8a17935af1b5fc265f2802d522dbf5af17d","option_order":["false","true"],"gold":"true","raw_logits":[-2.3203125,1.548828125],"temperature":1.0,"type":"noul","source_id":"5475","source_row_hash":"dc6c0165e59d0b35c1731ad73656915fbd6291deba80299bc358fc944a72978c","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.3203125,1.548828125],"calibrated_temperature":5.0},+ {"id":"massive-5568-choice","group_id":"massive-alarm-9d8d8e3f6fb99e20fed8","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8af7c9ac22d839179599a532150c0575129bb3db346ed4fcaf23996f2727c981","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.759765625,-1.7978515625,-1.2451171875],"temperature":1.0,"type":"choice","source_id":"5568","source_row_hash":"e2678a9b8f76e5325e37853048a1af7ef56128900590959c918238a2a0a87f67","input_tokens":54,"truncated":false,"calibrated_raw_logits":[3.759765625,-1.7978515625,-1.2451171875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5568-noul","group_id":"massive-alarm-9d8d8e3f6fb99e20fed8","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"6bb9d7bc2c8f0080617c2a6a730f73b45ea25f3c7a0e4fafdef8dbbfc6067d91","option_order":["false","true"],"gold":"true","raw_logits":[-2.484375,1.4775390625],"temperature":1.0,"type":"noul","source_id":"5568","source_row_hash":"e2678a9b8f76e5325e37853048a1af7ef56128900590959c918238a2a0a87f67","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.484375,1.4775390625],"calibrated_temperature":5.0},+ {"id":"massive-560-choice","group_id":"massive-alarm-3914f2c167cd3bc916a9","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8b4bacc83095f32adfbe08922536f8e8f085e1c778c9ed57ed2a65ce83ee6ff5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.1005859375,-3.044921875,1.4658203125],"temperature":1.0,"type":"choice","source_id":"560","source_row_hash":"198c7a67f853f0a20207e974e8b9f5c52eb85e496988929af5c4cd1b2ba14b7a","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.1005859375,-3.044921875,1.4658203125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-560-noul","group_id":"massive-alarm-3914f2c167cd3bc916a9","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"de516021eb9a4d76b704b912aab4923d0355631dc535f055c7f97319615c832e","option_order":["false","true"],"gold":"false","raw_logits":[-2.03515625,-0.5439453125],"temperature":1.0,"type":"noul","source_id":"560","source_row_hash":"198c7a67f853f0a20207e974e8b9f5c52eb85e496988929af5c4cd1b2ba14b7a","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.03515625,-0.5439453125],"calibrated_temperature":5.0},+ {"id":"massive-5623-choice","group_id":"massive-alarm-ee24ef5af4d9c0920de7","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2a8568251fdcde56fbf31c4ee89e239a704fee954543e6ef90286e089bda6c6f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.417236328125,-1.1318359375,0.07928466796875],"temperature":1.0,"type":"choice","source_id":"5623","source_row_hash":"d74f814f601c02c5df4a0b5f23516c6bf156ad8051a476422eb6fdea1660ae5e","input_tokens":50,"truncated":false,"calibrated_raw_logits":[0.417236328125,-1.1318359375,0.07928466796875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5623-noul","group_id":"massive-alarm-ee24ef5af4d9c0920de7","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"6af842101731694a1cd8e31608ddc4a5a0e95ab625ae4ebe22e246073a8f6700","option_order":["false","true"],"gold":"true","raw_logits":[-1.201171875,1.654296875],"temperature":1.0,"type":"noul","source_id":"5623","source_row_hash":"d74f814f601c02c5df4a0b5f23516c6bf156ad8051a476422eb6fdea1660ae5e","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.201171875,1.654296875],"calibrated_temperature":5.0},+ {"id":"massive-5635-choice","group_id":"massive-alarm-4beeb5931613d60f1341","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cc5c973ab31e3d3df68e2c8c5178bc62cccae85d6a38bef00637b677f9b3e2e0","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.031829833984375,-2.861328125,-0.1290283203125],"temperature":1.0,"type":"choice","source_id":"5635","source_row_hash":"40eb075b4d858f1411258909d4d77625f091cbc21c25f9c1484232129528cb67","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.031829833984375,-2.861328125,-0.1290283203125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5635-noul","group_id":"massive-alarm-4beeb5931613d60f1341","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"91e98a9d1309dfdefc5644e876e4141869104d2d08ee388a23a75d4dd1689a6a","option_order":["false","true"],"gold":"true","raw_logits":[-1.052734375,1.1396484375],"temperature":1.0,"type":"noul","source_id":"5635","source_row_hash":"40eb075b4d858f1411258909d4d77625f091cbc21c25f9c1484232129528cb67","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.052734375,1.1396484375],"calibrated_temperature":5.0},+ {"id":"massive-59-choice","group_id":"massive-alarm-2276a9a439ac95d3573f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"89b470c0d2ae0a5f39585af5b6ff60255952f2dd6dc6ec3e4a84b0ae006b2ecc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-0.04351806640625,-2.509765625,1.7255859375],"temperature":1.0,"type":"choice","source_id":"59","source_row_hash":"cb6899117f503f233e7b525dd5cd06511f88ace9a9caaee665d48c413113a6d9","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.04351806640625,-2.509765625,1.7255859375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-59-noul","group_id":"massive-alarm-2276a9a439ac95d3573f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"872c7a88b6ca28b3a9194a3e4678c850b38ef642762b3fd454545679f04001f5","option_order":["false","true"],"gold":"false","raw_logits":[-1.94140625,0.13916015625],"temperature":1.0,"type":"noul","source_id":"59","source_row_hash":"cb6899117f503f233e7b525dd5cd06511f88ace9a9caaee665d48c413113a6d9","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.94140625,0.13916015625],"calibrated_temperature":5.0},+ {"id":"massive-5902-choice","group_id":"massive-alarm-fb6df7f77542683a3ff2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7fbcd8af2c716bdd66961270eacaa6a96622bbcf1372261fd4a96f855553ec8d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.564453125,-0.93017578125,0.293212890625],"temperature":1.0,"type":"choice","source_id":"5902","source_row_hash":"d1fd61eb591646e537e465ded07ee0a48eeb5db31db81e0c4407cac197d93495","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-0.564453125,-0.93017578125,0.293212890625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5902-noul","group_id":"massive-alarm-fb6df7f77542683a3ff2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"c4be7c11de56ddc45a0601df7c80095583e9c5a7b51a97188443ff20f73e590e","option_order":["false","true"],"gold":"false","raw_logits":[-1.3876953125,1.02734375],"temperature":1.0,"type":"noul","source_id":"5902","source_row_hash":"d1fd61eb591646e537e465ded07ee0a48eeb5db31db81e0c4407cac197d93495","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.3876953125,1.02734375],"calibrated_temperature":5.0},+ {"id":"massive-5974-choice","group_id":"massive-alarm-f7cd5de4d0683af203ce","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"e7e220e3f9639398f9c6e56b8f38093b2018ad88a562b3f898609eacfb615f75","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[6.62890625,-4.55078125,-3.935546875],"temperature":1.0,"type":"choice","source_id":"5974","source_row_hash":"b4c1084215b35f9d593d4300ec53920326c4f994cb5bbe678cbc274e8068d703","input_tokens":54,"truncated":false,"calibrated_raw_logits":[6.62890625,-4.55078125,-3.935546875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-5974-noul","group_id":"massive-alarm-f7cd5de4d0683af203ce","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"964d28a1e6cbe408f1d74d44572fe81dc51b8fac4e3784c40464b96c34439c4d","option_order":["false","true"],"gold":"false","raw_logits":[-2.146484375,1.2470703125],"temperature":1.0,"type":"noul","source_id":"5974","source_row_hash":"b4c1084215b35f9d593d4300ec53920326c4f994cb5bbe678cbc274e8068d703","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.146484375,1.2470703125],"calibrated_temperature":5.0},+ {"id":"massive-6083-choice","group_id":"massive-alarm-950ec71310e9ece1aa7a","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"a11eee2509927e63186f13c256d44d20ed64932038a23b4a0806cab08a89405b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.064453125,-1.720703125,-0.33642578125],"temperature":1.0,"type":"choice","source_id":"6083","source_row_hash":"c98e1b18be265f4f3518a3d4f075393c40074e8f1aae799dbcefe012b867cd89","input_tokens":53,"truncated":false,"calibrated_raw_logits":[1.064453125,-1.720703125,-0.33642578125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-6083-noul","group_id":"massive-alarm-950ec71310e9ece1aa7a","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"5a396304607490821ff2e74144e5202da5c16f6aa3b19624edec62d3cbc0496a","option_order":["false","true"],"gold":"true","raw_logits":[-1.982421875,1.3046875],"temperature":1.0,"type":"noul","source_id":"6083","source_row_hash":"c98e1b18be265f4f3518a3d4f075393c40074e8f1aae799dbcefe012b867cd89","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.982421875,1.3046875],"calibrated_temperature":5.0},+ {"id":"massive-6101-choice","group_id":"massive-alarm-a63b13c9add3acdf4c9f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"06d76dcf51d497e0590fe5b28fa322168bfb9dce9eb88440c989d8a502850fc5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.08184814453125,-1.6630859375,-0.9814453125],"temperature":1.0,"type":"choice","source_id":"6101","source_row_hash":"3dce7763719f2037e2c03a6492336b70e796b37c6eec38a0243a9e146c04f22e","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.08184814453125,-1.6630859375,-0.9814453125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-6101-noul","group_id":"massive-alarm-a63b13c9add3acdf4c9f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"2eccf49f3d9075b4515ed86d93c2ae48a73534099b204323ff87e723347cbfb7","option_order":["false","true"],"gold":"false","raw_logits":[-0.765625,1.0244140625],"temperature":1.0,"type":"noul","source_id":"6101","source_row_hash":"3dce7763719f2037e2c03a6492336b70e796b37c6eec38a0243a9e146c04f22e","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-0.765625,1.0244140625],"calibrated_temperature":5.0},+ {"id":"massive-612-choice","group_id":"massive-alarm-6d2b063ac90aec6042cb","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"df2193b6cb5257904a68c032afc1bbe241c266f2fd966aff1d5f2543ef419e58","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.40576171875,-2.451171875,-0.64404296875],"temperature":1.0,"type":"choice","source_id":"612","source_row_hash":"437786bbb8397c686ef9c55a8e67c91d16d8087753ecc83dccbe5e64726224a6","input_tokens":57,"truncated":false,"calibrated_raw_logits":[0.40576171875,-2.451171875,-0.64404296875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-612-noul","group_id":"massive-alarm-6d2b063ac90aec6042cb","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"aa455667dd535b15199206b1efc70f5485bcda3c2b32ff04ff3fb4a6bcb5437c","option_order":["false","true"],"gold":"true","raw_logits":[-0.074951171875,1.0908203125],"temperature":1.0,"type":"noul","source_id":"612","source_row_hash":"437786bbb8397c686ef9c55a8e67c91d16d8087753ecc83dccbe5e64726224a6","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.074951171875,1.0908203125],"calibrated_temperature":5.0},+ {"id":"massive-6162-choice","group_id":"massive-alarm-7c67db34bf4904167369","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6c494191c98e1d84f5a75d1fe41fc46f153edf386c185a7e596bf70f25d2bf15","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.43359375,-3.333984375,2.044921875],"temperature":1.0,"type":"choice","source_id":"6162","source_row_hash":"9476a44d1260488e881b3dc20d371d653c5e2f622f312a564be0ff9faebdc44a","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-3.43359375,-3.333984375,2.044921875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-6162-noul","group_id":"massive-alarm-7c67db34bf4904167369","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7078e785d1141162ba7cc4503f84962ca97b9c6fdb845b23e0fe12d393519eeb","option_order":["false","true"],"gold":"false","raw_logits":[-1.7138671875,0.31005859375],"temperature":1.0,"type":"noul","source_id":"6162","source_row_hash":"9476a44d1260488e881b3dc20d371d653c5e2f622f312a564be0ff9faebdc44a","input_tokens":43,"truncated":false,"calibrated_raw_logits":[-1.7138671875,0.31005859375],"calibrated_temperature":5.0},+ {"id":"massive-63-choice","group_id":"massive-alarm-3f054a8e6d320a808954","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"b27424721301b616f34476c934b889aeef8fe2d25823afb618470a06db45d9cd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.4794921875,3.935546875,-1.7333984375],"temperature":1.0,"type":"choice","source_id":"63","source_row_hash":"f9f143eecf685ee8dcf485e26ca404fb60e1e6244ad6133efb0ff64c6889fc7b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.4794921875,3.935546875,-1.7333984375],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-63-noul","group_id":"massive-alarm-3f054a8e6d320a808954","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"13c4f2f436a4a1cad7db316725fa8870b5829f21dd47c292beb942e925bd11d2","option_order":["false","true"],"gold":"false","raw_logits":[-0.765625,0.442626953125],"temperature":1.0,"type":"noul","source_id":"63","source_row_hash":"f9f143eecf685ee8dcf485e26ca404fb60e1e6244ad6133efb0ff64c6889fc7b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-0.765625,0.442626953125],"calibrated_temperature":5.0},+ {"id":"massive-796-choice","group_id":"massive-alarm-1b94ce3d96f2d223de77","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"01608bf1f2ba0a3fe29f6ae16e61c8ee20458e453613bf297000ffae65be5d0b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.71142578125,-2.318359375,-2.154296875],"temperature":1.0,"type":"choice","source_id":"796","source_row_hash":"c0c137c04060af51a4172b5b368190deb3009c12c64fea91b2be12f7b694f28f","input_tokens":51,"truncated":false,"calibrated_raw_logits":[0.71142578125,-2.318359375,-2.154296875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-796-noul","group_id":"massive-alarm-1b94ce3d96f2d223de77","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e658950e96a415d7a90f36a0e8dbb02edf10a90e0ec6b9bf9b85672831d343d4","option_order":["false","true"],"gold":"false","raw_logits":[-1.1796875,0.55810546875],"temperature":1.0,"type":"noul","source_id":"796","source_row_hash":"c0c137c04060af51a4172b5b368190deb3009c12c64fea91b2be12f7b694f28f","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.1796875,0.55810546875],"calibrated_temperature":5.0},+ {"id":"massive-861-choice","group_id":"massive-alarm-105aeeed3519053d3f55","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"adb310a263a979b824132b06ef4ca2fd6f40f379cafa5c0583d2893422e48ffe","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[5.2890625,-3.55078125,-2.4453125],"temperature":1.0,"type":"choice","source_id":"861","source_row_hash":"7c204b7b739698b3a6f70ee107ef52e531a47e8d7b51ed6b839e3583bc83351a","input_tokens":50,"truncated":false,"calibrated_raw_logits":[5.2890625,-3.55078125,-2.4453125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-861-noul","group_id":"massive-alarm-105aeeed3519053d3f55","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a55bf3c906f2bba70c4aa1a2f07dc2ea87f35341ef6fe8be3d694dbabe98589e","option_order":["false","true"],"gold":"false","raw_logits":[-2.2734375,1.072265625],"temperature":1.0,"type":"noul","source_id":"861","source_row_hash":"7c204b7b739698b3a6f70ee107ef52e531a47e8d7b51ed6b839e3583bc83351a","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.2734375,1.072265625],"calibrated_temperature":5.0},+ {"id":"massive-8731-choice","group_id":"massive-alarm-c773d4135b5f98edcb5c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"e992f3c9dd54dfa9e1f81ed8dea16513e434dff0114a353de45a25822f20fc4b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.36376953125,-4.3046875,-1.541015625],"temperature":1.0,"type":"choice","source_id":"8731","source_row_hash":"37b09493cdc06a1bc88cc3643150ae9782b3ed10eea61dba2cb9c6519f20d4a1","input_tokens":59,"truncated":false,"calibrated_raw_logits":[-0.36376953125,-4.3046875,-1.541015625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-8731-noul","group_id":"massive-alarm-c773d4135b5f98edcb5c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ad5e11d3a7f8e16dc300f0f163db1e89309331376cd7149fc62d155a16de6d35","option_order":["false","true"],"gold":"true","raw_logits":[-2.794921875,0.283203125],"temperature":1.0,"type":"noul","source_id":"8731","source_row_hash":"37b09493cdc06a1bc88cc3643150ae9782b3ed10eea61dba2cb9c6519f20d4a1","input_tokens":54,"truncated":false,"calibrated_raw_logits":[-2.794921875,0.283203125],"calibrated_temperature":5.0},+ {"id":"massive-8741-choice","group_id":"massive-alarm-79a03eb5dc7bab70336f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"12f704d636ca16e99b76ca5cea7307740f6125027d09a9c89c544fe86c2342f3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.1640625,-4.1875,-3.794921875],"temperature":1.0,"type":"choice","source_id":"8741","source_row_hash":"062a50b981a7bf96310a69f314e1b5e9e03ca5f3c1ff474bf49fbf780ad21844","input_tokens":52,"truncated":false,"calibrated_raw_logits":[6.1640625,-4.1875,-3.794921875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-8741-noul","group_id":"massive-alarm-79a03eb5dc7bab70336f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fb565c561562c1581b7292d775cfc16f8ebab4e949c1ac31e7d00f377e4d9834","option_order":["false","true"],"gold":"true","raw_logits":[-1.9677734375,1.234375],"temperature":1.0,"type":"noul","source_id":"8741","source_row_hash":"062a50b981a7bf96310a69f314e1b5e9e03ca5f3c1ff474bf49fbf780ad21844","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.9677734375,1.234375],"calibrated_temperature":5.0},+ {"id":"massive-884-choice","group_id":"massive-alarm-42f057d6c35ee3122781","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"92579ff62a0442cf6f564f0ed96f983bec83f3f69dfe99d3ce8bb1259c1ac9d5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.1171875,-1.9990234375,-1.5869140625],"temperature":1.0,"type":"choice","source_id":"884","source_row_hash":"55b0f00bd02e71e600e190c2bc9e46f2a1d354fe59c818a9cb43fd1332486182","input_tokens":51,"truncated":false,"calibrated_raw_logits":[4.1171875,-1.9990234375,-1.5869140625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-884-noul","group_id":"massive-alarm-42f057d6c35ee3122781","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4ab83fec7f87887afa7cfc2310e2c6a492f625fe0400fac7bc6ed4813067f280","option_order":["false","true"],"gold":"true","raw_logits":[-2.265625,1.5087890625],"temperature":1.0,"type":"noul","source_id":"884","source_row_hash":"55b0f00bd02e71e600e190c2bc9e46f2a1d354fe59c818a9cb43fd1332486182","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.265625,1.5087890625],"calibrated_temperature":5.0},+ {"id":"massive-8867-choice","group_id":"massive-alarm-bfe246fb1d4b99a3f48a","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cbde715e981d6e934a2aac954045f8eae203ebf02c68432c7343a6bf8c3cc940","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.603515625,-3.880859375,-2.998046875],"temperature":1.0,"type":"choice","source_id":"8867","source_row_hash":"a513b328db26ac85863558c72614b0034a2f8177453c5d71df24760e4e766cd7","input_tokens":53,"truncated":false,"calibrated_raw_logits":[2.603515625,-3.880859375,-2.998046875],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-8867-noul","group_id":"massive-alarm-bfe246fb1d4b99a3f48a","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e2b236cc79afa43b3fc316499b938cc5aadf2150ff7a433da58df64c902ca760","option_order":["false","true"],"gold":"true","raw_logits":[-1.4541015625,0.333984375],"temperature":1.0,"type":"noul","source_id":"8867","source_row_hash":"a513b328db26ac85863558c72614b0034a2f8177453c5d71df24760e4e766cd7","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.4541015625,0.333984375],"calibrated_temperature":5.0},+ {"id":"massive-8882-choice","group_id":"massive-alarm-84c38cd9104789d07c40","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ff0063357703eca9276281b428c4dda0916cc9e39a13740943d5e2776399f15e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.3515625,-2.701171875,-1.8369140625],"temperature":1.0,"type":"choice","source_id":"8882","source_row_hash":"e1c5f172f9f02cade5685772af79ebded1b74b376026d93341832931a32676f9","input_tokens":57,"truncated":false,"calibrated_raw_logits":[2.3515625,-2.701171875,-1.8369140625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-8882-noul","group_id":"massive-alarm-84c38cd9104789d07c40","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"94a0ac44c2af3923c53e676b3d98f5ee321b0cc3a3a31c051d034ec2c89445ce","option_order":["false","true"],"gold":"true","raw_logits":[-2.046875,1.22265625],"temperature":1.0,"type":"noul","source_id":"8882","source_row_hash":"e1c5f172f9f02cade5685772af79ebded1b74b376026d93341832931a32676f9","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-2.046875,1.22265625],"calibrated_temperature":5.0},+ {"id":"massive-920-choice","group_id":"massive-alarm-84e033aebf70598798ff","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"d2f20e3d91af6e06b4f1acf721dcd73de93d27d83c024195da8ad52b1bceecdd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.517578125,-4.140625,4.17578125],"temperature":1.0,"type":"choice","source_id":"920","source_row_hash":"ed0a9ccdf3b6e428b4770ab41b3c2b59fa4327b9580291f077f18f49a54b44b2","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-3.517578125,-4.140625,4.17578125],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-920-noul","group_id":"massive-alarm-84e033aebf70598798ff","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4cc035ea2dff281db57f6de7c194ec38bfe20021c39aadf5fc817997e722b652","option_order":["false","true"],"gold":"false","raw_logits":[-2.470703125,-0.1820068359375],"temperature":1.0,"type":"noul","source_id":"920","source_row_hash":"ed0a9ccdf3b6e428b4770ab41b3c2b59fa4327b9580291f077f18f49a54b44b2","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.470703125,-0.1820068359375],"calibrated_temperature":5.0},+ {"id":"massive-939-choice","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"b39b50798b10226d8212f2e2f9e3a8929c244b25102bcf6e856c5df4b13d4ee8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.4375,-3.669921875,-0.99853515625],"temperature":1.0,"type":"choice","source_id":"939","source_row_hash":"936109e1a09f4aecb014f31c6bbb213ce6de4023624f13dfe11e9c083dd0320e","input_tokens":54,"truncated":false,"calibrated_raw_logits":[5.4375,-3.669921875,-0.99853515625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-939-noul","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"dd6f955f414596fad6478d08e6f36bea3627253a74ec2d137249375a66c63afb","option_order":["false","true"],"gold":"true","raw_logits":[-2.650390625,0.978515625],"temperature":1.0,"type":"noul","source_id":"939","source_row_hash":"936109e1a09f4aecb014f31c6bbb213ce6de4023624f13dfe11e9c083dd0320e","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.650390625,0.978515625],"calibrated_temperature":5.0},+ {"id":"massive-940-choice","group_id":"massive-alarm-5b03b657dea06a387027","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"58112a614bee3f995d34f5cd7698b4fc18f8696a65393c5d108082656961fb29","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.388671875,-2.8046875,-0.0709228515625],"temperature":1.0,"type":"choice","source_id":"940","source_row_hash":"78635f925b3274baf503c5823a54dfb84e1a144e79ad8a8ef3ffbd50edfc6c4b","input_tokens":50,"truncated":false,"calibrated_raw_logits":[1.388671875,-2.8046875,-0.0709228515625],"calibrated_temperature":3.1355035588563744},+ {"id":"massive-940-noul","group_id":"massive-alarm-5b03b657dea06a387027","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"d534399a37dd93763633f7d9628ca5918f84cacf599473cd36b168ef8fcf2198","option_order":["false","true"],"gold":"true","raw_logits":[-2.1015625,1.1630859375],"temperature":1.0,"type":"noul","source_id":"940","source_row_hash":"78635f925b3274baf503c5823a54dfb84e1a144e79ad8a8ef3ffbd50edfc6c4b","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.1015625,1.1630859375],"calibrated_temperature":5.0}+ ]+}diff --git a/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.md b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.mdnew file mode 100644index 00000000..2360a9c0--- /dev/null+++ b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.md@@ -0,0 +1,43 @@+# MASSIVE Chinese alarm calibration evaluation++Measurement date: **2026-09-24**. Execution base: `23a17522aa4942da6cce53a995a275760320b691`; contribution: `37cc58465da5cdabddf1ff543234397d666ad546`.++Historical inference was executed on 2026-09-24 from base commit 23a17522aa4942da6cce53a995a275760320b691 with a dirty contribution worktree subsequently recorded in 37cc58465da5cdabddf1ff543234397d666ad546. The recorded execution source-file hashes are authoritative for the measured code. These measurements were not rerun on the later upstream rebase or evaluation-only refactor.++E0 and E1 use the same multilingual checkpoint and raw decision logits. E1 fits one temperature per question type on calibration data. No model weights are updated.++| Condition | Slice | n | Accuracy | Macro F1 | NLL | Brier | ECE |+|---|---|---:|---:|---:|---:|---:|---:|+| E0 | overall | 186 | 0.532258 | 0.440840 | 1.383747 | 0.730071 | 0.366488 |+| E0 | choice | 93 | 0.634409 | 0.580928 | 1.296043 | 0.520825 | 0.253426 |+| E0 | noul | 93 | 0.430108 | 0.300752 | 1.471450 | 0.939317 | 0.494611 |+| E1 | overall | 186 | 0.532258 | 0.440840 | 0.785033 | 0.506898 | 0.197529 |+| E1 | choice | 93 | 0.634409 | 0.580928 | 0.837271 | 0.474032 | 0.169905 |+| E1 | noul | 93 | 0.430108 | 0.300752 | 0.732796 | 0.539765 | 0.289635 |++Positive scalar temperature preserves argmax: accuracy and F1 are unchanged.++| Type | Calibration n | Temperature | Bound reached |+|---|---:|---:|---|+| choice | 64 | 3.135504 | False |+| noul | 64 | 5.000000 | True |++Paired differences are E1 minus E0; negative NLL, Brier and ECE differences are better.++| Metric | Difference | 95% group bootstrap interval |+|---|---:|---|+| accuracy | +0.000000 | [+0.000000, +0.000000] |+| macro_f1 | +0.000000 | [+0.000000, +0.000000] |+| nll | -0.598713 | [-0.865774, -0.369099] |+| brier | -0.223173 | [-0.298826, -0.152890] |+| ece | -0.168958 | [-0.224995, -0.098547] |++## Scope and limits++- Test: 93 source utterances, 89 heuristic groups, 186 correlated decisions. Calibration: 64 source utterances, 62 groups, 128 decisions. The paired question types are not independent samples.+- This is a derived MASSIVE 1.1 alarm subset with upstream human intent-review evidence, not the full benchmark, naturally occurring customer-service logs, or independent local human review.+- One fixed split/seed; 1,000 group bootstrap draws. Intervals condition on the fixed checkpoint and fitted temperatures, omit calibration uncertainty, and are descriptive without multiplicity correction.+- Unknown checkpoint exposure to MASSIVE/SLURP; grouping may miss semantic dependencies. These results do not establish general Chinese improvement or unchanged behavior on other tasks.+- The baseline noul fit reaches T=5, the permitted upper bound. Calibration settings were fixed without choosing on test. Risk/coverage does not improve at every coverage.+- Configured max_len=512, head_max_len=192, FP16. Historical test inputs were at most 60 tokens; this is not a 512-token stress measurement.+- The compact evidence omits source text and retains source IDs/hashes. It can recompute numerical results and check pairing; full source-schema/annotation checks require the prepared public dataset.diff --git a/research/benchmarks/zh_reliability/runtime.py b/research/benchmarks/zh_reliability/runtime.pynew file mode 100644index 00000000..0eb6b5ae--- /dev/null+++ b/research/benchmarks/zh_reliability/runtime.py@@ -0,0 +1,277 @@+"""Environment diagnostics and a thin adapter over the repository's actual Agent."""+import importlib.metadata+import json+import os+from pathlib import Path+import platform+import shutil+import subprocess+import sys+import time++from .data import atomic_json, digest, file_hash, options++MODEL_ID = 'convaiinnovations/laya-multilingual'+INPUT_KEYS = ('input_ids', 'attention_mask', 'marker_pos', 'marker_mask', 'qtype')+++def command(args):+ try:+ p = subprocess.run(args, capture_output=True, text=True, timeout=20)+ return p.stdout.strip() if p.returncode == 0 else p.stderr.strip()+ except (OSError, subprocess.TimeoutExpired) as e:+ return str(e)+++def environment():+ result = {'python': sys.version, 'executable': sys.executable, 'os': platform.platform(),+ 'commit': command(['git', 'rev-parse', 'HEAD']),+ 'dirty': bool(command(['git', 'status', '--porcelain'])),+ 'source_hashes': {str(p): file_hash(p) for p in sorted(Path(__file__).parent.glob('*.py'))},+ 'environment': {k: os.environ[k] for k in ('CUDA_VISIBLE_DEVICES', 'CUDA_DEVICE_ORDER',+ 'HF_HUB_OFFLINE', 'USE_TF', 'USE_TORCH', 'TOKENIZERS_PARALLELISM', 'OMP_NUM_THREADS')+ if k in os.environ}, 'dependencies': {},+ 'nvidia_smi': command(['nvidia-smi', '--query-gpu=index,name,uuid,memory.total,driver_version',+ '--format=csv,noheader']), 'gpus': []}+ for package in ('torch', 'transformers', 'numpy', 'safetensors', 'huggingface_hub', 'tokenizers'):+ try:+ result['dependencies'][package] = importlib.metadata.version(package)+ except importlib.metadata.PackageNotFoundError:+ result['dependencies'][package] = None+ try:+ import torch+ result.update(cuda=torch.version.cuda, cudnn=torch.backends.cudnn.version(),+ pytorch_arches=torch.cuda.get_arch_list())+ for i in range(torch.cuda.device_count()):+ p = torch.cuda.get_device_properties(i)+ gpu = {'process_index': i, 'name': p.name, 'uuid': str(getattr(p, 'uuid', 'unavailable')),+ 'memory_bytes': p.total_memory, 'capability': [p.major, p.minor]}+ if gpu['uuid'] != 'unavailable' and not gpu['uuid'].startswith('GPU-'):+ gpu['uuid'] = 'GPU-' + gpu['uuid']+ try:+ with torch.cuda.device(i):+ x = torch.ones((16, 16), device=f'cuda:{i}')+ value = (x @ x).sum().item()+ torch.cuda.synchronize(i)+ if value != 4096:+ raise RuntimeError('CUDA tensor result mismatch')+ del x+ gpu['status'] = 'SUCCESS'+ except Exception as e:+ gpu.update(status='FAILED', reason=str(e))+ # UUID matching avoids confusing process-local index with physical index.+ matches = [line.split(',')[0].strip() for line in result['nvidia_smi'].splitlines()+ if gpu['uuid'] != 'unavailable' and gpu['uuid'] in line]+ gpu['physical_index'] = matches[0] if len(matches) == 1 else None+ result['gpus'].append(gpu)+ result['gpu_status'] = ('SUCCESS' if result['gpus'] and all(g['status'] == 'SUCCESS' for g in result['gpus'])+ else 'FAILED' if result['gpus'] else 'NOT_RUN')+ except Exception as e:+ result.update(gpu_status='NOT_RUN', reason=str(e))+ return result+++def select_device(selector, env):+ matches = [g for g in env['gpus'] if selector.lower() in g['name'].lower() or selector == g['uuid']+ or selector == f"cuda:{g['process_index']}"]+ if len(matches) != 1 or matches[0]['status'] != 'SUCCESS':+ raise RuntimeError('device must uniquely match one visible GPU with a successful CUDA operation: ' + selector)+ return 'cuda:' + str(matches[0]['process_index'])+++def resolve_model(source, revision, offline, work):+ """Make a private snapshot so Agent's tokenizer compatibility patch cannot touch shared caches."""+ source_path = Path(source)+ sha = None+ if not source_path.is_dir():+ if source != MODEL_ID:+ raise ValueError('only the multilingual model is supported')+ from huggingface_hub import HfApi, snapshot_download+ if not offline:+ sha = HfApi().model_info(MODEL_ID, revision=revision).sha+ path = snapshot_download(MODEL_ID, revision=sha or revision, local_files_only=offline,+ allow_patterns=['rl_agent_config.json', 'model.safetensors',+ 'encoder/*.json', 'tokenizer/*'])+ source_path = Path(path)+ sha = source_path.name+ lineage_path = source_path / 'zh_lineage.json'+ lineage = json.loads(lineage_path.read_text()) if lineage_path.exists() else None+ if lineage and (lineage['model_id'] != MODEL_ID or lineage.get('trained')):+ raise ValueError('evaluation requires an original multilingual checkpoint')+ cfg = json.loads((source_path / 'rl_agent_config.json').read_text())+ # Local imports require explicit, inspectable multilingual identity, not just a folder name.+ if not lineage and not any(x in str(cfg.get('encoder', '')).lower() for x in ('mmbert', 'multilingual')):+ raise ValueError('local checkpoint config does not identify a multilingual encoder')+ files = [source_path / 'model.safetensors', source_path / 'rl_agent_config.json']+ files += sorted((source_path / 'encoder').glob('*.json')) + sorted((source_path / 'tokenizer').glob('*'))+ hashes = {str(p.relative_to(source_path)): file_hash(p) for p in files if p.is_file()}+ if lineage and lineage.get('weights_hash') and lineage['weights_hash'] != hashes['model.safetensors']:+ raise ValueError('checkpoint weight hash mismatch')+ # Preserve the base identity shape so existing calibration artifacts still match.+ identity = {'model_id': MODEL_ID, 'revision': sha or (lineage or {}).get('base_revision'),+ 'content_hash': digest(hashes), 'files': hashes,+ 'trained': False}+ target = Path(work) / 'model_input'+ target.mkdir()+ for name in hashes:+ dst = target / name+ dst.parent.mkdir(parents=True, exist_ok=True)+ shutil.copyfile(source_path / name, dst)+ atomic_json(Path(work) / 'model_identity.json', identity)+ return target, identity+++def ensure_device(agent, device):+ import torch+ expected = torch.device(device)+ if agent.device != expected or any(p.device != expected for p in agent.model.parameters()):+ raise RuntimeError('device mismatch / SDK CPU fallback: requested ' + device)+++class Adapter:+ def __init__(self, agent, identity, max_len=512, head_max_len=192, dtype='fp16'):+ import torch+ self.agent, self.identity = agent, identity+ self.max_len, self.head_max_len, self.dtype = max_len, head_max_len, dtype+ self.device = str(agent.device)+ if head_max_len < 16 or max_len <= head_max_len + 4:+ raise ValueError('max_len must leave state space beyond head_max_len + 4')+ agent.amp_enabled = dtype == 'fp16' and agent.device.type == 'cuda'+ agent.dtype = torch.float16 if agent.amp_enabled else torch.float32+ self.original = {'temperature_raw': agent.temperature_raw,+ 'temperature_by_options_raw': agent.temperature_by_options_raw,+ 'temperature': list(agent.temperature),+ 'temperature_by_options': dict(agent.temperature_by_options),+ 'lang_temperatures': dict(agent.lang_temperatures)}++ def temperature(self, row):+ from laya.common import QTYPES, temp_bucket+ qt = QTYPES[row['question']['type']]+ cfg = self.agent.lang_temperatures.get('zh', {'temperature': self.agent.temperature,+ 'temperature_by_options': self.agent.temperature_by_options})+ return cfg['temperature_by_options'].get(temp_bucket(qt, len(options(row))), cfg['temperature'][qt])++ def prepare(self, row):+ from laya.common import build_sequence, render_options+ a, q = self.agent, row['question']+ a._check_question(row['id'], q)+ internal = a._to_internal(q)+ item = a._encode_state(row['state'], [row['id']], {row['id']: internal},+ self.max_len, self.head_max_len)[0]+ state_ids = a.tok(row['state'].replace(a.tok.mask_token, ' '), add_special_tokens=False)['input_ids']+ empty, markers = build_sequence(a.tok, '', internal, self.max_len, self.head_max_len, state_ids=[])+ if len(markers) != len(options(row)) or any(item['ids'][m] != a.tok.mask_token_id for m in markers):+ raise ValueError('option marker truncated')+ retained = max(0, len(item['ids']) - len(empty))+ head = a.tok(f"{internal['t']} question: {internal['ins'].replace(a.tok.mask_token, ' ')}",+ add_special_tokens=False)['input_ids']+ opt_full = [a.tok(' ' + o.replace(a.tok.mask_token, ' '), add_special_tokens=False)['input_ids']+ for o in render_options(internal)]+ head_cut = len(head) > markers[0] - 2+ opt_cut = any(len(full) > (markers[i + 1] if i + 1 < len(markers) else len(empty) - 2) - m - 1+ for i, (m, full) in enumerate(zip(markers, opt_full)))+ item['label'] = options(row).index(row['label'])+ tokens = {'input_tokens': len(item['ids']), 'state_tokens': len(state_ids), 'state_retained': retained,+ 'option_count': len(markers), 'state_truncated': retained < len(state_ids),+ 'instructions_truncated': head_cut, 'options_truncated': opt_cut,+ 'truncated': retained < len(state_ids) or head_cut or opt_cut}+ return item, tokens++ def batch(self, items):+ from laya.common import collate_items+ return collate_items([items], self.agent.tok.pad_token_id)++ def forward(self, batch):+ import torch+ ensure_device(self.agent, self.device)+ with torch.autocast(device_type=self.agent.device.type, dtype=torch.float16,+ enabled=self.dtype == 'fp16' and self.agent.device.type == 'cuda'):+ logits, _action_logits = self.agent.model(+ **{k: batch[k].to(self.agent.device) for k in INPUT_KEYS})+ if not torch.isfinite(logits).all():+ raise RuntimeError('nonfinite decision logits')+ return logits++ def predict(self, rows, condition, split):+ import torch+ from .metrics import probability_record+ self.agent.model.eval()+ results = []+ # One state has exactly one decision sequence in this schema.+ with torch.no_grad():+ for r in rows:+ item, tokens = self.prepare(r)+ logits = self.forward(self.batch([item]))[0, :len(options(r))].float().cpu().tolist()+ rec = probability_record(r, logits, self.temperature(r))+ rec.update(tokens=tokens, condition=condition, split=split['assignments'][r['id']], model=self.identity)+ results.append(rec)+ return results++ def parity(self, rows):+ import numpy as np+ from .metrics import probability_record+ import torch+ errors = []+ self.agent.model.eval()+ for row in rows:+ item, _ = self.prepare(row)+ with torch.no_grad():+ z = self.forward(self.batch([item]))[0, :len(options(row))].cpu().tolist()+ p = probability_record(row, z, self.temperature(row))['probabilities']+ fallback_count = self.agent.cpu_fallback_count+ answer = self.agent.system_one(row['state'], {row['id']: row['question']}, lang='zh',+ max_len=self.max_len, head_max_len=self.head_max_len)['answers'][row['id']]+ ensure_device(self.agent, self.device)+ if self.agent.cpu_fallback_count != fallback_count:+ raise RuntimeError('Agent parity used CPU fallback')+ public = ([1 - answer['noul'], answer['noul']] if row['question']['type'] == 'noul'+ else [answer['probabilities'][k] for k in options(row)])+ error = float(np.max(np.abs(np.array(p) - public)))+ if error > .000051:+ raise AssertionError('Agent parity exceeds four-decimal rounding: ' + str(error))+ errors.append(error)+ return {'n': len(rows), 'max_absolute_error': max(errors), 'rounding_tolerance': .000051}+++def measure(adapter, rows, warmup=10, batches=30):+ import numpy as np+ import torch+ if adapter.agent.device.type != 'cuda':+ raise RuntimeError('GPU measurements prohibit CPU fallback')+ if warmup < 1 or batches < 1 or not rows:+ raise ValueError('measurement needs positive warmup/batches and nonempty samples')+ device = adapter.agent.device+ adapter.agent.model.eval()+ prepared = [adapter.batch([adapter.prepare(r)[0]]) for r in rows]+ prepared_gpu = [{k: b[k].to(device) for k in INPUT_KEYS} for b in prepared]+ result = {'warmup_batches': warmup, 'measured_batches': batches, 'batch_states': 1,+ 'batch_decisions': 1, 'independent_samples': len(rows),+ 'tokens': [adapter.prepare(r)[1] for r in rows], 'repetitions': batches / len(rows),+ 'model_stage_transfers': 'excludes H2D/D2H; inputs already on device'}+ from .metrics import probability_record+ with torch.no_grad():+ for stage in ('end_to_end', 'model_stage'):+ times = []+ torch.cuda.synchronize(device)+ torch.cuda.reset_peak_memory_stats(device)+ for i in range(warmup + batches):+ row = rows[i % len(rows)]+ torch.cuda.synchronize(device)+ start = time.perf_counter()+ if stage == 'end_to_end':+ item, _ = adapter.prepare(row)+ z = adapter.forward(adapter.batch([item]))[0, :len(options(row))].cpu().tolist()+ probability_record(row, z, adapter.temperature(row))+ else:+ adapter.forward(prepared_gpu[i % len(rows)])+ torch.cuda.synchronize(device)+ elapsed = time.perf_counter() - start+ if i >= warmup:+ times.append(elapsed)+ result[stage] = {'p50_batch_ms': float(np.quantile(times, .5) * 1000),+ 'p95_batch_ms': float(np.quantile(times, .95) * 1000),+ 'state_per_s': batches / sum(times), 'decision_per_s': batches / sum(times),+ 'max_memory_allocated': torch.cuda.max_memory_allocated(device),+ 'max_memory_reserved': torch.cuda.max_memory_reserved(device),+ 'memory_scope': 'inference including warmup, resident model and prepared inputs; excludes model loading'}+ return resultdiff --git a/tests/test_zh_annotation_provenance.py b/tests/test_zh_annotation_provenance.pynew file mode 100644index 00000000..04e92219--- /dev/null+++ b/tests/test_zh_annotation_provenance.py@@ -0,0 +1,150 @@+"""Offline regressions for upstream review evidence; all rows here are synthetic."""+import copy+import json+from pathlib import Path+import sys+import tempfile+import unittest++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))+from research.benchmarks.zh_reliability.data import partition, read_data+++class UpstreamAnnotationProvenanceTests(unittest.TestCase):+ def setUp(self):+ self.directory = tempfile.TemporaryDirectory()+ self.addCleanup(self.directory.cleanup)+ self.path = Path(self.directory.name) / 'data.jsonl'+ self.row = {+ 'id': 'synthetic_alarm_1', 'group_id': 'synthetic_source_1', 'task_id': 'alarm_operation',+ 'state': '设置明早的闹钟', 'split': 'train',+ 'question': {'type': 'choice', 'instructions': '选择闹钟操作',+ 'criteria': {'alarm_set': '设置闹钟', 'alarm_remove': '删除闹钟'}},+ 'label': 'alarm_set', 'tags': [],+ 'provenance': {+ 'kind': 'public_human_annotated', 'human_verified': False,+ 'source': 'synthetic source for offline validation only', 'license': 'CC-BY-4.0',+ 'annotation_status': 'upstream intent judgments; no local human review',+ 'source_intent': 'alarm_set',+ 'upstream_annotation': {+ 'dataset': 'MASSIVE', 'version': '1.1', 'source_id': '1', 'intent': 'alarm_set',+ 'source_row_hash': 'b' * 64,+ 'evidence_url': 'https://github.com/alexa/massive/blob/' + 'a' * 40 + '/README.md#data-format',+ 'judgments': [{'worker_id': str(i), 'intent_score': 1} for i in range(3)],+ },+ },+ }++ def read(self, rows, formal=True):+ self.path.write_text(''.join(json.dumps(r, ensure_ascii=False) + '\n' for r in rows), encoding='utf-8')+ return read_data(self.path, formal=formal)++ def test_source_review_keeps_local_review_false_and_preserves_group_splits(self):+ rows = []+ for i, split in enumerate(('train', 'dev', 'calibration', 'test')):+ source = copy.deepcopy(self.row)+ source.update(id=str(i), group_id='source_' + str(i), split=split)+ source['provenance']['upstream_annotation']['source_id'] = str(i)+ binary = copy.deepcopy(source)+ binary.update(id=str(i) + '_binary', task_id='alarm_set_binary', label='true')+ binary['question'] = {'type': 'noul', 'instructions': '是否设置闹钟?',+ 'criteria': {'false': '不是设置闹钟', 'true': '设置闹钟'}}+ rows.extend((source, binary))+ loaded = self.read(rows)+ self.assertTrue(all(r['provenance']['human_verified'] is False for r in loaded))+ manifest = partition(loaded, formal=True)+ self.assertEqual(manifest['group_counts'], {s: 1 for s in ('train', 'dev', 'calibration', 'test')})+ self.assertEqual(manifest['assignments'], {r['id']: r['split'] for r in rows})+ rows[-1]['split'] = 'train'+ with self.assertRaisesRegex(ValueError, 'split a group'):+ partition(self.read(rows), formal=True)++ def test_missing_or_invalid_identity_evidence_is_rejected(self):+ changes = {+ 'dataset': (None, 'Other'), 'version': (None, '1.0'),+ 'source_id': (None, 1, '', ' '), 'intent': (None, '', 'alarm_remove'),+ 'source_row_hash': (None, '', 'b' * 63, 'g' * 64),+ 'evidence_url': (None, 'https://github.com/alexa/massive/blob/main/README.md',+ 'https://example.com/' + 'a' * 40 + '/README.md',+ 'https://github.com/alexa/massive/blob/' + 'a' * 40 + '/README.md?branch=main'),+ }+ for field, values in changes.items():+ for value in values:+ with self.subTest(field=field, value=value):+ row = copy.deepcopy(self.row)+ row['provenance']['upstream_annotation'][field] = value+ with self.assertRaisesRegex(ValueError, 'source-review evidence'):+ self.read([row])+ for value in (None, [], 'claimed review'):+ row = copy.deepcopy(self.row)+ row['provenance']['upstream_annotation'] = value+ with self.assertRaisesRegex(ValueError, 'source-review evidence'):+ self.read([row])++ def test_contested_missing_or_duplicate_judgments_are_rejected(self):+ valid = self.row['provenance']['upstream_annotation']['judgments']+ invalid = [None, {}, [], valid[:2], valid[:2] + [valid[0]], valid + [None]]+ for score in (0, 2, True, 1.0, '1', None):+ judgments = copy.deepcopy(valid)+ judgments[2]['intent_score'] = score+ invalid.append(judgments)+ for worker in ('', ' ', None, 2):+ judgments = copy.deepcopy(valid)+ judgments[2]['worker_id'] = worker+ invalid.append(judgments)+ invalid.append(valid + [{'worker_id': 'fourth', 'intent_score': 0}])+ for judgments in invalid:+ with self.subTest(judgments=judgments):+ row = copy.deepcopy(self.row)+ row['provenance']['upstream_annotation']['judgments'] = judgments+ with self.assertRaisesRegex(ValueError, 'source-review evidence'):+ self.read([row])++ def test_public_kind_cannot_claim_local_human_review(self):+ row = copy.deepcopy(self.row)+ row['provenance']['human_verified'] = True+ for formal in (False, True):+ with self.subTest(formal=formal), self.assertRaisesRegex(ValueError, 'source-review evidence'):+ self.read([row], formal=formal)++ def test_formal_source_license_status_and_split_are_still_required(self):+ for field in ('source', 'license', 'annotation_status'):+ row = copy.deepcopy(self.row)+ del row['provenance'][field]+ with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'formal data requires'):+ self.read([row])+ row = copy.deepcopy(self.row)+ del row['split']+ with self.assertRaisesRegex(ValueError, 'formal data requires'):+ self.read([row])+ row = copy.deepcopy(self.row)+ del row['provenance']['source_intent']+ with self.assertRaisesRegex(ValueError, 'source-review evidence'):+ self.read([row])++ def test_pinned_raw_document_url_is_accepted(self):+ row = copy.deepcopy(self.row)+ row['provenance']['upstream_annotation']['evidence_url'] = (+ 'https://raw.githubusercontent.com/alexa/massive/' + 'a' * 40 + '/README.md')+ self.assertEqual(len(self.read([row])), 1)++ def test_existing_local_verification_and_fixture_rules_are_preserved(self):+ row = copy.deepcopy(self.row)+ row['provenance']['kind'] = 'user_annotated'+ row['provenance']['human_verified'] = True+ del row['provenance']['upstream_annotation']+ self.assertEqual(len(self.read([row])), 1)+ row['provenance']['kind'] = 'agent_authored_fixture'+ with self.assertRaisesRegex(ValueError, 'cannot claim human verification'):+ self.read([row])+ row['provenance']['human_verified'] = False+ self.assertEqual(len(self.read([row], formal=False)), 1)+ with self.assertRaisesRegex(ValueError, 'formal data requires'):+ self.read([row])+ row['provenance']['kind'] = 'unverified_external'+ with self.assertRaisesRegex(ValueError, 'formal data requires'):+ self.read([row])+++if __name__ == '__main__':+ unittest.main(verbosity=2)diff --git a/tests/test_zh_massive.py b/tests/test_zh_massive.pynew file mode 100644index 00000000..5b149544--- /dev/null+++ b/tests/test_zh_massive.py@@ -0,0 +1,440 @@+"""Offline MASSIVE converter checks using fabricated, test-only source records."""+import copy+import io+from pathlib import Path+import sys+import tarfile+import tempfile+import unittest+from unittest import mock++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))+from research.benchmarks.zh_reliability import prepare_massive as massive+from research.benchmarks.zh_reliability.data import atomic_json, digest, file_hash, partition, read_data+++class MassiveConversionTests(unittest.TestCase):+ def setUp(self):+ self.directory = tempfile.TemporaryDirectory()+ self.addCleanup(self.directory.cleanup)+ self.path = Path(self.directory.name)+ self.source = {'archive_sha256': 'b' * 64}+ self.chinese, self.english = [], []+ for i, split in enumerate(('train', 'train', 'train', 'train', 'dev', 'test')):+ self.add(str(i), split, tuple(massive.INTENTS)[i % 3])++ def add(self, source_id, split, intent='alarm_set', utterance=None, template=None):+ chinese = {+ 'id': source_id, 'locale': 'zh-CN', 'partition': split, 'scenario': 'alarm', 'intent': intent,+ 'utt': utterance or '仅供测试的合成句子' + source_id,+ 'annot_utt': '仅供测试的合成句子' + source_id,+ 'judgments': [{'worker_id': str(i), 'intent_score': 1} for i in range(3)],+ }+ english = {**copy.deepcopy(chinese), 'locale': 'en-US',+ 'utt': 'Fabricated source for test only ' + source_id,+ 'annot_utt': template or 'Fabricated source for test only ' + source_id}+ self.chinese.append(chinese)+ self.english.append(english)+ return chinese, english++ def convert(self, **kwargs):+ return massive.convert(self.chinese, self.english, self.source, **kwargs)++ @staticmethod+ def by_source(rows):+ result = {}+ for row in rows:+ source_id = row['provenance']['upstream_annotation']['source_id']+ result.setdefault(source_id, {})[row['question']['type']] = row+ return result++ def test_exact_labels_shared_groups_and_upstream_provenance(self):+ rows, diagnostics, summary = self.convert()+ self.assertEqual(diagnostics, [])+ self.assertEqual(summary['derived_decisions'], 2 * len(self.chinese))+ for source in self.chinese:+ pair = self.by_source(rows)[source['id']]+ choice, noul = pair['choice'], pair['noul']+ self.assertEqual(choice['label'], source['intent'])+ self.assertEqual(choice['question']['criteria'], massive.INTENTS)+ self.assertEqual(noul['label'], 'true' if source['intent'] == 'alarm_set' else 'false')+ self.assertEqual(list(noul['question']['criteria']), ['false', 'true'])+ self.assertEqual(choice['group_id'], noul['group_id'])+ self.assertEqual(choice['split'], noul['split'])+ self.assertEqual(choice['state'], source['utt'])+ self.assertEqual(noul['state'], source['utt'])+ provenance = choice['provenance']+ self.assertIs(provenance['human_verified'], False)+ self.assertIs(provenance['local_human_review'], False)+ self.assertEqual(provenance['kind'], 'public_human_annotated')+ evidence = provenance['upstream_annotation']+ self.assertEqual(evidence['intent'], source['intent'])+ self.assertEqual(evidence['source_row_hash'], digest(source))+ self.assertEqual(evidence['judgments'], source['judgments'])+ self.assertEqual(evidence['evidence_url'], massive.EVIDENCE_URL)+ massive.write_jsonl(self.path / 'derived.jsonl', rows)+ validated = read_data(self.path / 'derived.jsonl', formal=True)+ self.assertEqual(partition(validated, formal=True)['counts'],+ {s: summary['by_split'][s]['decisions'] for s in summary['by_split']})++ def test_official_dev_test_are_preserved_and_calibration_is_only_from_train(self):+ rows, _, summary = self.convert(calibration_fraction=.5)+ pairs = self.by_source(rows)+ self.assertEqual(pairs['4']['choice']['split'], 'dev')+ self.assertEqual(pairs['5']['choice']['split'], 'test')+ source_splits = {r['id']: r['partition'] for r in self.chinese}+ for source_id, pair in pairs.items():+ if pair['choice']['split'] == 'calibration':+ self.assertEqual(source_splits[source_id], 'train')+ else:+ self.assertEqual(pair['choice']['split'], source_splits[source_id])+ self.assertEqual(summary['by_split']['train']['source_rows'], 2)+ self.assertEqual(summary['by_split']['calibration']['source_rows'], 2)+ self.assertEqual(sum(x['source_rows'] for x in summary['by_split'].values()), summary['retained_source_rows'])+ self.assertEqual(sum(x['decisions'] for x in summary['by_split'].values()), len(rows))+ self.assertEqual(sum(x['groups'] for x in summary['by_split'].values()), len({r['group_id'] for r in rows}))++ def test_english_delexicalized_template_overlap_quarantines_train(self):+ self.add('overlap_train', 'train', template='Test only: set an alarm at [time : seven]')+ self.add('overlap_test', 'test', template='Test only: set an alarm at [time : eleven]')+ rows, diagnostics, summary = self.convert()+ pairs = self.by_source(rows)+ self.assertNotIn('overlap_train', pairs)+ self.assertEqual(pairs['overlap_test']['choice']['split'], 'test')+ excluded = next(r for r in diagnostics if r['source_id'] == 'overlap_train')+ self.assertIn('shares_source_template_or_text_with_test', excluded['reasons'])+ self.assertEqual(excluded['group_id'], pairs['overlap_test']['choice']['group_id'])+ self.assertEqual(summary['retained_source_rows'] + summary['diagnostic_source_rows'], len(self.chinese))++ def test_chinese_normalization_and_transitive_template_grouping(self):+ self.add('chain_train', 'train', utterance='测试:ABC!', template='Synthetic bridge one [time : seven]')+ self.add('chain_dev', 'dev', utterance='测 试 abc', template='Synthetic bridge two [time : eight]')+ self.add('chain_test', 'test', utterance='仅供测试的第三句', template='Synthetic bridge two [time : ten]')+ rows, diagnostics, _ = self.convert()+ pairs = self.by_source(rows)+ self.assertNotIn('chain_train', pairs)+ self.assertNotIn('chain_dev', pairs)+ owner_group = pairs['chain_test']['choice']['group_id']+ excluded = [r for r in diagnostics if r['source_id'].startswith('chain_')]+ self.assertEqual({r['source_id'] for r in excluded}, {'chain_train', 'chain_dev'})+ self.assertTrue(all(r['group_id'] == owner_group for r in excluded))+ self.assertTrue(all('shares_source_template_or_text_with_test' in r['reasons'] for r in excluded))++ def test_duplicate_chinese_sources_in_one_partition_share_a_group(self):+ self.add('duplicate_a', 'test', utterance='测试:相同句子。')+ self.add('duplicate_b', 'test', utterance='测试 相同句子')+ rows, diagnostics, summary = self.convert()+ pairs = self.by_source(rows)+ self.assertEqual(pairs['duplicate_a']['choice']['group_id'], pairs['duplicate_b']['choice']['group_id'])+ self.assertEqual(summary['by_split']['test']['source_rows'], 3)+ self.assertEqual(summary['by_split']['test']['groups'], 2)+ self.assertEqual(diagnostics, [])++ def test_conflicting_chinese_labels_are_retained_only_in_diagnostics(self):+ self.add('conflict_a', 'test', intent='alarm_set', utterance='测试:相同但冲突的句子')+ self.add('conflict_b', 'test', intent='alarm_remove', utterance='测试 相同但冲突的句子。')+ rows, diagnostics, summary = self.convert()+ pairs = self.by_source(rows)+ self.assertNotIn('conflict_a', pairs)+ self.assertNotIn('conflict_b', pairs)+ excluded = [r for r in diagnostics if r['source_id'].startswith('conflict_')]+ self.assertEqual(len(excluded), 2)+ self.assertTrue(all('conflicting_intents_for_identical_normalized_chinese' in r['reasons'] for r in excluded))+ self.assertEqual(summary['excluded_reasons']['conflicting_intents_for_identical_normalized_chinese'], 2)++ def test_contested_or_missing_reviews_are_preserved_in_diagnostics(self):+ for source_id, score in (('contested', 0), ('reasonable_only', 2), ('boolean_not_score', True)):+ row, _ = self.add(source_id, 'test')+ row['judgments'][2]['intent_score'] = score+ missing, _ = self.add('missing_review', 'test')+ del missing['judgments']+ duplicate, _ = self.add('duplicate_review', 'test')+ duplicate['judgments'][2] = copy.deepcopy(duplicate['judgments'][0])+ original = copy.deepcopy(self.chinese)+ rows, diagnostics, summary = self.convert()+ expected = {'contested', 'reasonable_only', 'boolean_not_score', 'missing_review', 'duplicate_review'}+ self.assertEqual({r['source_id'] for r in diagnostics}, expected)+ self.assertTrue(expected.isdisjoint(self.by_source(rows)))+ self.assertTrue(all('upstream_intent_review_not_unanimous_or_missing' in r['reasons'] for r in diagnostics))+ source_lookup = {r['id']: r for r in original}+ self.assertTrue(all(d['source_row'] == source_lookup[d['source_id']] for d in diagnostics))+ self.assertEqual(self.chinese, original)+ self.assertEqual(summary['diagnostic_source_rows'], len(expected))++ def test_source_mismatch_is_rejected(self):+ variants = [('locale', 'zh-TW'), ('partition', 'test'), ('intent', 'alarm_remove')]+ for field, value in variants:+ english = copy.deepcopy(self.english)+ english[0][field] = value+ with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'source mismatch'):+ massive.convert(self.chinese, english, self.source)+ with self.assertRaisesRegex(ValueError, 'source mismatch'):+ massive.convert(self.chinese, self.english[1:], self.source)+ chinese = copy.deepcopy(self.chinese)+ chinese[0]['locale'] = 'zh-TW'+ with self.assertRaisesRegex(ValueError, 'source mismatch'):+ massive.convert(chinese, self.english, self.source)++ def test_deterministic_rerun_and_source_input_not_mutated(self):+ source_before = copy.deepcopy((self.chinese, self.english, self.source))+ first = self.convert(seed=7, calibration_fraction=.5)+ second = self.convert(seed=7, calibration_fraction=.5)+ self.assertEqual(first, second)+ self.assertEqual((self.chinese, self.english, self.source), source_before)+ reversed_result = massive.convert(list(reversed(self.chinese)), list(reversed(self.english)), self.source,+ seed=7, calibration_fraction=.5)+ self.assertEqual(first, reversed_result)++ def test_insufficient_groups_and_invalid_calibration_fraction_fail(self):+ for fraction in (0, 1, -.1, 1.1):+ with self.subTest(fraction=fraction), self.assertRaisesRegex(ValueError, 'calibration fraction'):+ self.convert(calibration_fraction=fraction)+ with self.assertRaisesRegex(ValueError, 'insufficient independent train groups'):+ massive.convert(self.chinese[3:], self.english[3:], self.source)+++class MassiveSourceChecksTests(unittest.TestCase):+ def setUp(self):+ self.directory = tempfile.TemporaryDirectory()+ self.addCleanup(self.directory.cleanup)+ self.path = Path(self.directory.name)++ def make_source(self):+ source = self.path / 'source'+ source.mkdir()+ for name in massive.SOURCE_FILES:+ text = 'FABRICATED TEST-ONLY DOCUMENT ' + name+ if name == 'NOTICE.md':+ text += '\nCC BY 4.0\n'+ if name == 'DATA_LICENSE':+ text += '\nAttribution 4.0 International\n'+ (source / name).write_text(text, encoding='utf-8')+ archive = source / 'massive-1.1.tar.gz'+ archive.write_bytes(b'FABRICATED TEST-ONLY ARCHIVE')+ metadata = {'archive_url': massive.ARCHIVE_URL, 'publisher_revision': massive.PUBLISHER_REVISION,+ 'archive_sha256': file_hash(archive),+ 'files': {name: file_hash(source / name) for name in massive.SOURCE_FILES}}+ atomic_json(source / 'source_manifest.json', metadata)+ return source, metadata++ def test_source_file_and_archive_checksum_mismatch_are_rejected(self):+ source, metadata = self.make_source()+ self.assertEqual(massive.check_source(source), metadata)+ text = (source / 'zh-CN.jsonl').read_bytes()+ (source / 'zh-CN.jsonl').write_bytes(text + b'changed')+ with self.assertRaisesRegex(ValueError, 'source file hash mismatch: zh-CN.jsonl'):+ massive.check_source(source)+ (source / 'zh-CN.jsonl').write_bytes(text)+ (source / 'massive-1.1.tar.gz').write_bytes(b'changed archive')+ with self.assertRaisesRegex(ValueError, 'source archive hash mismatch'):+ massive.check_source(source)++ def test_version_and_notice_mismatch_are_rejected(self):+ source, metadata = self.make_source()+ for field, value in (('archive_url', 'https://example.com/not-massive'), ('publisher_revision', 'main')):+ changed = {**metadata, field: value}+ atomic_json(source / 'source_manifest.json', changed)+ with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'unexpected dataset version'):+ massive.check_source(source)+ (source / 'NOTICE.md').write_text('test-only document without required notice')+ metadata['files']['NOTICE.md'] = file_hash(source / 'NOTICE.md')+ atomic_json(source / 'source_manifest.json', metadata)+ with self.assertRaisesRegex(ValueError, 'missing publisher dataset-license notice'):+ massive.check_source(source)++ @staticmethod+ def archive_bytes(extra=None):+ out = io.BytesIO()+ members = [('release/data/zh-CN.jsonl', b'{"test_only":true}\n'),+ ('release/data/en-US.jsonl', b'{"test_only":true}\n'),+ ('release/LICENSE', b'FABRICATED TEST ONLY Attribution 4.0 International')]+ if extra:+ members.append(extra)+ with tarfile.open(fileobj=out, mode='w:gz') as archive:+ for name, value in members:+ member = tarfile.TarInfo(name)+ if value is None:+ member.type = tarfile.SYMTYPE+ member.linkname = '../../outside'+ archive.addfile(member)+ else:+ member.size = len(value)+ archive.addfile(member, io.BytesIO(value))+ return out.getvalue()++ def responses(self, archive):+ def open_url(url, timeout):+ self.assertGreater(timeout, 0)+ if url == massive.ARCHIVE_URL:+ return io.BytesIO(archive)+ self.assertTrue(url.startswith(massive.RAW_DOCS))+ return io.BytesIO(b'FABRICATED TEST-ONLY DOCUMENT CC BY 4.0')+ return open_url++ def test_download_reads_selected_regular_members_without_extracting_paths(self):+ archive = self.archive_bytes(('../outside', b'not allowed outside source'))+ destination = self.path / 'download'+ with mock.patch.object(massive.urllib.request, 'urlopen', side_effect=self.responses(archive)):+ massive.fetch_source(destination)+ self.assertFalse((self.path / 'outside').exists())+ self.assertFalse((destination / 'release').exists())+ self.assertEqual((destination / 'zh-CN.jsonl').read_text(), '{"test_only":true}\n')+ self.assertEqual(massive.check_source(destination)['archive_sha256'], file_hash(destination / 'massive-1.1.tar.gz'))++ def test_download_rejects_selected_symlinks_and_duplicate_members(self):+ for i, extra in enumerate((('other/data/zh-CN.jsonl', None),+ ('other/data/en-US.jsonl', b'duplicate test-only member'))):+ archive = self.archive_bytes(extra)+ with self.subTest(extra=extra), mock.patch.object(massive.urllib.request, 'urlopen',+ side_effect=self.responses(archive)):+ with self.assertRaisesRegex(ValueError, 'unexpected/duplicate archive member'):+ massive.fetch_source(self.path / ('download_' + str(i)))+++class MassiveParquetTests(unittest.TestCase):+ """Fabricated HF-shaped records and transport mocks; no Parquet library or network."""+ def setUp(self):+ self.directory = tempfile.TemporaryDirectory()+ self.addCleanup(self.directory.cleanup)+ self.path = Path(self.directory.name)+ self.features = {+ 'scenario': {'_type': 'ClassLabel', 'names': ['music', 'alarm']},+ 'intent': {'_type': 'ClassLabel', 'names': ['alarm_query', 'alarm_remove', 'alarm_set']},+ }+ self.row = {+ 'id': 'test-only-1', 'locale': 'zh-CN', 'partition': 'dev', 'scenario': 1, 'intent': 2,+ 'utt': '仅供测试:请设置AM七点的闹钟!',+ 'annot_utt': '仅供测试:请设置[time : AM七点]的闹钟!',+ 'judgments': {'worker_id': ['0', '1', '2'], 'intent_score': [1, 1, 1],+ 'grammar_score': [3, 4, 3]},+ 'slot_method': {'slot': ['time'], 'method': ['localization']},+ }++ def decode(self, row=None, features=None):+ return massive.decode_parquet_row(self.row if row is None else row,+ self.features if features is None else features,+ 'zh-CN', 'validation')++ def test_decoder_uses_embedded_labels_and_transposes_sequences_without_text_changes(self):+ original = copy.deepcopy(self.row)+ result = self.decode()+ self.assertEqual(result['scenario'], 'alarm')+ self.assertEqual(result['intent'], 'alarm_set')+ self.assertEqual(result['utt'], original['utt'])+ self.assertEqual(result['annot_utt'], original['annot_utt'])+ self.assertEqual(result['partition'], 'dev')+ self.assertEqual(result['judgments'], [+ {'worker_id': '0', 'intent_score': 1, 'grammar_score': 3},+ {'worker_id': '1', 'intent_score': 1, 'grammar_score': 4},+ {'worker_id': '2', 'intent_score': 1, 'grammar_score': 3},+ ])+ self.assertEqual(result['slot_method'], [{'slot': 'time', 'method': 'localization'}])+ self.assertEqual(self.row, original)+ self.row['slot_method'] = {'slot': [], 'method': []}+ self.assertEqual(self.decode()['slot_method'], [])++ def test_decoder_rejects_invalid_label_indices_and_feature_type(self):+ for field in ('scenario', 'intent'):+ for value in (-1, len(self.features[field]['names']), True, 1.0, '1', None):+ row = copy.deepcopy(self.row)+ row[field] = value+ with self.subTest(field=field, value=value), self.assertRaisesRegex(ValueError, 'invalid parquet class label'):+ self.decode(row)+ features = copy.deepcopy(self.features)+ features[field]['_type'] = 'Value'+ with self.assertRaisesRegex(ValueError, 'invalid parquet class label'):+ self.decode(features=features)++ def test_decoder_rejects_wrong_locale_or_partition(self):+ for field, value in (('locale', 'en-US'), ('partition', 'validation'), ('partition', 'test')):+ row = copy.deepcopy(self.row)+ row[field] = value+ with self.subTest(field=field, value=value), self.assertRaisesRegex(ValueError, 'locale/partition mismatch'):+ self.decode(row)+ row = copy.deepcopy(self.row)+ row['partition'] = 'train'+ self.assertEqual(massive.decode_parquet_row(row, self.features, 'zh-CN', 'train')['partition'], 'train')++ def test_decoder_rejects_malformed_or_unequal_sequence_columns(self):+ for field in ('judgments', 'slot_method'):+ for value in (None, [], {}, {'one': 'not a list'}, {'one': [1], 'two': [1, 2]}):+ row = copy.deepcopy(self.row)+ row[field] = value+ with self.subTest(field=field, value=value), self.assertRaisesRegex(ValueError, 'parquet sequence'):+ self.decode(row)++ def make_source(self):+ directory = self.path / 'source'+ directory.mkdir()+ for name in massive.SOURCE_FILES + massive.PARQUET_FILES:+ text = 'FABRICATED TEST-ONLY SOURCE ' + name+ if name == 'NOTICE.md':+ text += ' CC BY 4.0'+ if name == 'DATA_LICENSE':+ text += ' Attribution 4.0 International'+ (directory / name).write_text(text)+ metadata = {+ 'archive_url': massive.ARCHIVE_URL, 'publisher_revision': massive.PUBLISHER_REVISION,+ 'transport': 'hf_parquet', 'mirror_revision': massive.PARQUET_REVISION,+ 'mirror_base_url': massive.PARQUET_BASE, 'license_text_url': massive.LICENSE_TEXT_URL,+ 'files': {name: file_hash(directory / name) for name in massive.SOURCE_FILES + massive.PARQUET_FILES},+ }+ atomic_json(directory / 'source_manifest.json', metadata)+ return directory, metadata++ def test_source_check_requires_fixed_mirror_metadata_and_parquet_hashes(self):+ directory, metadata = self.make_source()+ self.assertEqual(massive.check_source(directory), metadata)+ for field, value in (('mirror_revision', 'main'), ('mirror_base_url', 'https://example.com/'),+ ('license_text_url', 'https://example.com/LICENSE')):+ atomic_json(directory / 'source_manifest.json', {**metadata, field: value})+ with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'unexpected parquet source revision'):+ massive.check_source(directory)+ atomic_json(directory / 'source_manifest.json', metadata)+ for name in massive.PARQUET_FILES:+ target = directory / name+ original = target.read_bytes()+ target.write_bytes(original + b'CHANGED')+ with self.subTest(name=name), self.assertRaisesRegex(ValueError, 'source file hash mismatch'):+ massive.check_source(directory)+ target.write_bytes(original)++ def test_mirror_fetch_uses_pinned_urls_and_publishes_completed_files(self):+ directory = self.path / 'download'+ expected = {f'{locale}-{split}.parquet': f'{massive.PARQUET_BASE}{locale}/{split}/0000.parquet'+ for locale in ('zh-CN', 'en-US') for split in ('train', 'validation', 'test')}+ expected.update({name: massive.RAW_DOCS + name for name in massive.SOURCE_FILES[3:]})+ expected['DATA_LICENSE'] = massive.LICENSE_TEXT_URL+ def open_url(url, timeout):+ self.assertIn(url, expected.values())+ self.assertGreater(timeout, 0)+ return io.BytesIO(('FABRICATED TEST-ONLY ' + url).encode())+ with mock.patch.object(massive.urllib.request, 'urlopen', side_effect=open_url) as download, \+ mock.patch.object(massive, 'prepare_parquet_source', return_value=directory) as prepare:+ self.assertEqual(massive.fetch_parquet_source(directory), directory)+ self.assertEqual(download.call_count, len(expected))+ prepare.assert_called_once_with(directory)+ self.assertEqual({p.name for p in directory.iterdir()}, set(expected))+ self.assertFalse(list(directory.glob('*.part')))+ for name, url in expected.items():+ self.assertEqual((directory / name).read_bytes(), ('FABRICATED TEST-ONLY ' + url).encode())++ def test_interrupted_mirror_download_is_not_published_or_prepared(self):+ class InterruptedStream(io.BytesIO):+ def read(self, size=-1):+ data = super().read(size)+ if not data:+ raise OSError('fabricated download interruption')+ return data+ directory = self.path / 'download'+ with mock.patch.object(massive.urllib.request, 'urlopen', return_value=InterruptedStream(b'test only')), \+ mock.patch.object(massive, 'prepare_parquet_source') as prepare:+ with self.assertRaisesRegex(OSError, 'fabricated download interruption'):+ massive.fetch_parquet_source(directory)+ prepare.assert_not_called()+ self.assertFalse((directory / massive.PARQUET_FILES[0]).exists())+ self.assertEqual((directory / (massive.PARQUET_FILES[0] + '.part')).read_bytes(), b'test only')+++if __name__ == '__main__':+ unittest.main(verbosity=2)diff --git a/tests/test_zh_reliability.py b/tests/test_zh_reliability.pynew file mode 100644index 00000000..e58ee703--- /dev/null+++ b/tests/test_zh_reliability.py@@ -0,0 +1,383 @@+"""Offline logic tests; tiny random BERT is NOT a real multilingual Laya experiment."""+import copy+import contextlib+import io+import json+import math+import os+from pathlib import Path+import sys+import tempfile+import unittest+from unittest.mock import patch++os.environ.setdefault('USE_TF', '0')+os.environ.setdefault('USE_TORCH', '1')+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))++import numpy as np+import torch+from safetensors.torch import save_file+from tokenizers import Tokenizer+from tokenizers.models import WordLevel+from tokenizers.pre_tokenizers import Whitespace+from transformers import BertConfig, BertModel, PreTrainedTokenizerFast++from research.benchmarks.zh_reliability.calibration import calibrate, fit_temperature, validate_calibration+from research.benchmarks.zh_reliability.data import FIXTURES, atomic_json, options, partition, read_data+from research.benchmarks.zh_reliability.metrics import log_probs, paired_bootstrap, probability_record, quality, summarize+from research.benchmarks.zh_reliability.runtime import Adapter, ensure_device, select_device+from laya.agent import Agent+from laya.common import DecisionModel, collate_items+++class DataMetricsTests(unittest.TestCase):+ def setUp(self):+ self.rows = read_data(FIXTURES)+ self.split = partition(self.rows)+ self.model = {'content_hash': 'mock-only', 'trained': False}++ def prediction(self, row=None, logits=(0., 0.), t=1.):+ row = row or self.rows[0]+ p = probability_record(row, logits, t)+ p.update(split=self.split['assignments'][row['id']], model=self.model)+ return p++ def test_schema_groups_manifest_and_relations(self):+ self.assertEqual(len(self.rows), 80)+ self.assertEqual(len(self.split['groups']), 40)+ self.assertEqual(self.split['counts'], {'test': 12, 'train': 48, 'dev': 8, 'calibration': 12})+ with tempfile.TemporaryDirectory() as tmp:+ path = Path(tmp) / 'split.json'+ saved = partition(self.rows, path)+ self.assertEqual(partition(self.rows, path, seed=1), saved)+ changed = copy.deepcopy(self.rows)+ changed[0]['state'] += '!'+ with self.assertRaisesRegex(ValueError, 'hash'):+ partition(changed, path)+ saved['assignments'][self.rows[0]['id']] = 'train'+ atomic_json(path, saved)+ with self.assertRaisesRegex(ValueError, 'leakage'):+ partition(self.rows, path)++ def test_schema_rejections(self):+ for change in ({'label': 'unknown'}, {'provenance': {}}, {'tags': 'negation'},+ {'question': {'type': 'score', 'instructions': 'bad'}},+ {'provenance': {'kind': 'agent_authored', 'human_verified': True}}):+ with self.subTest(change=change), tempfile.TemporaryDirectory() as tmp:+ rows = copy.deepcopy(self.rows)+ rows[0].update(change)+ path = Path(tmp) / 'data.jsonl'+ path.write_text('\n'.join(json.dumps(r) for r in rows))+ with self.assertRaises(ValueError):+ read_data(path)+ with self.assertRaises(ValueError):+ read_data(FIXTURES, formal=True)++ def test_template_family_may_not_cross_groups(self):+ rows = copy.deepcopy(self.rows)+ rows[2]['provenance']['template_family'] = rows[0]['provenance']['template_family']+ with tempfile.TemporaryDirectory() as tmp:+ p = Path(tmp) / 'data.jsonl'+ p.write_text('\n'.join(json.dumps(r) for r in rows))+ with self.assertRaisesRegex(ValueError, 'family crosses'):+ read_data(p)++ def test_formal_requires_fixed_licensed_verified_data(self):+ rows = copy.deepcopy(self.rows)+ for r in rows:+ r['provenance'].update(kind='user_supplied', human_verified=True, source='test source',+ license='CC0', annotation_status='double reviewed')+ r['split'] = self.split['assignments'][r['id']]+ with tempfile.TemporaryDirectory() as tmp:+ p = Path(tmp) / 'real.jsonl'+ p.write_text('\n'.join(json.dumps(r) for r in rows))+ accepted = read_data(p, formal=True)+ self.assertEqual(partition(accepted, formal=True)['assignments'], self.split['assignments'])+ accepted[0]['split'] = 'train'+ with self.assertRaisesRegex(ValueError, 'split a group'):+ partition(accepted, formal=True)++ def test_choice_permutation_preserves_semantic_mapping(self):+ row = self.rows[40]+ before = probability_record(row, [4., 3., 2., 1.], 1.)+ perm = copy.deepcopy(row)+ perm['question']['criteria'] = dict(reversed(list(perm['question']['criteria'].items())))+ after = probability_record(perm, [1., 2., 3., 4.], 1.)+ self.assertEqual(before['predicted'], after['predicted'])+ self.assertEqual(before['gold'], after['gold'])+ for key, value in zip(before['option_order'], before['probabilities']):+ self.assertAlmostEqual(value, dict(zip(after['option_order'], after['probabilities']))[key])++ def test_noul_probability_is_true_despite_display_labels(self):+ row = copy.deepcopy(self.rows[0])+ row['question']['labels'] = {'false': '拒绝', 'true': '接受'}+ row['question']['criteria'] = dict(reversed(list(row['question']['criteria'].items())))+ p = self.prediction(row, [0., math.log(3)])+ self.assertEqual(p['option_order'], ['false', 'true'])+ self.assertAlmostEqual(p['p_true'], .75)+ self.assertAlmostEqual(p['confidence'], .75)++ def test_valid_mask_and_temperature_invariance(self):+ expected = log_probs([1., 2.], 1.)+ np.testing.assert_allclose(expected, log_probs([1., 99., 2.], 1., [True, False, True]))+ np.testing.assert_allclose(np.exp(expected), torch.softmax(torch.tensor([1., 2.], dtype=torch.float64), 0))+ for t in (.5, 1., 2., 5.):+ self.assertEqual(log_probs([1., 4., -2.], t).argmax(), 1)+ for t in (0., -.1, .1, 6., float('nan'), float('inf')):+ with self.assertRaises(ValueError):+ log_probs([0., 1.], t)++ def test_hand_computable_nll_brier_ece_and_confidence(self):+ p = self.prediction(logits=(0., math.log(3))) # gold true, correct with p=.75+ m = quality([p])+ self.assertAlmostEqual(m['nll'], -math.log(.75))+ self.assertAlmostEqual(m['brier'], .125)+ self.assertAlmostEqual(m['ece'], .25)+ self.assertAlmostEqual(p['confidence'], .75) # not 1 - entropy/log(2)+ self.assertEqual(m['accuracy'], 1.)+ uniform = self.prediction()+ self.assertEqual(uniform['confidence'], .5)+ self.assertIsNone(quality([])['nll'])+ json.dumps(summarize([]), allow_nan=False)++ def test_nll_does_not_underflow_via_probabilities(self):+ p = self.prediction(logits=(10000., -10000.))+ self.assertEqual(quality([p])['nll'], 20000.)++ def test_risk_ties_do_not_depend_on_labels(self):+ a, b = self.prediction(), self.prediction()+ a.update(id='a', gold='false')+ b.update(id='b', gold='true')+ self.assertEqual(quality([b, a])['risk_coverage'][0]['risk'], 0)+ a['gold'], b['gold'] = 'true', 'false'+ self.assertEqual(quality([b, a])['risk_coverage'][0]['risk'], 1)+ self.assertEqual(quality([a, b])['risk_coverage'], quality([b, a])['risk_coverage'])++ def test_task_local_macro_f1(self):+ a, b = self.prediction(), self.prediction()+ a.update(task_id='a', gold='false', predicted='false')+ b.update(task_id='b', gold='true', predicted='true')+ # Each task has one perfectly predicted class and one absent class: F1=.5 each.+ self.assertEqual(quality([a, b])['macro_f1'], .5)++ def calibration_rows(self):+ return [self.prediction(r, [0.] * len(options(r))) for r in self.rows+ if self.split['assignments'][r['id']] == 'calibration']++ def test_fit_export_reload_no_double_scaling_or_bucket_shadowing(self):+ rows = self.calibration_rows()+ original = {'temperature': [2., 3., 4.], 'temperature_by_options': {'choice:3-5': 5., 'score:2': 2.},+ 'lang_temperatures': {}}+ a = calibrate(rows, self.model, self.split, original, minimum=1, smoke=True)+ with tempfile.TemporaryDirectory() as tmp:+ p = Path(tmp) / 'cal.json'+ atomic_json(p, a)+ a = json.loads(p.read_text())+ cfg = validate_calibration(a, self.model, self.split, True)+ self.assertEqual(cfg['zh']['temperature'][1], 3.)+ self.assertNotIn('choice:3-5', cfg['zh']['temperature_by_options'])+ self.assertEqual(cfg['zh']['temperature_by_options']['score:2'], 2.)+ altered = copy.deepcopy(rows)+ for r in altered:+ r['temperature'] = 5.+ r['probabilities'] = [0.] * len(r['raw_logits'])+ self.assertEqual(fit_temperature(rows, 1), fit_temperature(altered, 1))+ with self.assertRaisesRegex(ValueError, 'smoke'):+ validate_calibration(a, self.model, self.split)+ with self.assertRaisesRegex(ValueError, 'model or data'):+ validate_calibration(a, {'content_hash': 'other'}, self.split, True)+ bad = copy.deepcopy(self.split)+ bad['data_hash'] = 'wrong'+ with self.assertRaises(ValueError):+ validate_calibration(a, self.model, bad, True)+ bad = copy.deepcopy(a)+ bad['calibration_hash'] = 'wrong'+ with self.assertRaises(ValueError):+ validate_calibration(bad, self.model, self.split, True)+ bad = copy.deepcopy(a)+ bad['lang_temperatures']['zh']['temperature'][0] = .1+ with self.assertRaises(ValueError):+ validate_calibration(bad, self.model, self.split, True)++ def test_fit_bounds_skipped_and_partition_guard(self):+ rows = self.calibration_rows()+ self.assertEqual(fit_temperature(rows)['status'], 'SKIPPED')+ a = self.prediction(logits=(10., -10.))+ result = fit_temperature([a] * 30)+ self.assertEqual(result['temperature'], 5.)+ self.assertTrue(result['boundary_hit'])+ self.assertLessEqual(result['nll_fitted'], result['nll_t1'])+ rows[0]['split'] = 'test'+ with self.assertRaisesRegex(ValueError, 'calibration partition'):+ calibrate(rows, self.model, self.split, {})++ def test_bootstrap_pairs_and_insufficient_groups(self):+ rows = self.calibration_rows()+ self.assertEqual(paired_bootstrap(rows, rows)['status'], 'SKIPPED')+ result = paired_bootstrap(rows, rows, repeats=10, min_groups=2)+ self.assertEqual(result['after_minus_before']['nll']['ci95'], [0., 0.])+ with self.assertRaises(ValueError):+ paired_bootstrap(rows, rows[:-1])++ def test_visible_device_selector_and_cpu_fallback(self):+ env = {'gpus': [{'name': 'TITAN RTX', 'process_index': 0, 'uuid': 'GPU-b', 'status': 'SUCCESS'},+ {'name': 'RTX 2080 Ti', 'process_index': 1, 'uuid': 'GPU-a', 'status': 'FAILED'}]}+ self.assertEqual(select_device('GPU-b', env), 'cuda:0')+ with self.assertRaises(RuntimeError):+ select_device('2080', env)+ with self.assertRaises(RuntimeError):+ select_device('physical:0', env)+ fake = type('A', (), {'device': torch.device('cpu')})()+ with self.assertRaisesRegex(RuntimeError, 'fallback'):+ ensure_device(fake, 'cuda:0')++ def test_cli_failure_is_nonzero_and_cannot_reuse_success(self):+ from research.benchmarks.zh_reliability.__main__ import main+ with tempfile.TemporaryDirectory() as tmp:+ run = Path(tmp) / 'failed'+ with patch('research.benchmarks.zh_reliability.__main__.environment', return_value={'gpus': [], 'gpu_status': 'NOT_RUN'}):+ self.assertEqual(main(['import-data', '--data', str(run / 'missing'), '--run-dir', str(run)]), 1)+ m = json.loads((run / 'manifest.json').read_text())+ self.assertEqual(m['status'], 'FAILED')+ self.assertEqual(m['error']['stage'], 'data')+ self.assertEqual(main(['import-data', '--run-dir', str(run)]), 2)++ def test_no_cuda_smoke_and_report_saved_results(self):+ from research.benchmarks.zh_reliability.__main__ import main+ with tempfile.TemporaryDirectory() as tmp:+ run, out = Path(tmp) / 'no-gpu', Path(tmp) / 'report'+ with patch('research.benchmarks.zh_reliability.__main__.environment', return_value={'gpus': [], 'gpu_status': 'NOT_RUN'}):+ self.assertEqual(main(['smoke', '--gpu', '--run-dir', str(run)]), 0)+ self.assertEqual(json.loads((run / 'manifest.json').read_text())['status'], 'SKIPPED')+ self.assertEqual(main(['report', '--source', str(run), '--run-dir', str(out)]), 0)+ self.assertEqual((run / 'split_manifest.json').read_bytes(), (out / 'split_manifest.json').read_bytes())+ self.assertEqual((run / 'metrics.json').read_bytes(), (out / 'metrics.json').read_bytes())+ self.assertIn('SKIPPED', (out / 'report.md').read_text())++ def test_all_cli_help(self):+ from research.benchmarks.zh_reliability.__main__ import parser+ for name in ('check-env', 'import-data', 'smoke', 'eval', 'calibrate', 'report'):+ with self.subTest(name=name), contextlib.redirect_stdout(io.StringIO()), self.assertRaises(SystemExit) as e:+ parser().parse_args([name, '--help'])+ self.assertEqual(e.exception.code, 0)+++class TinyEvaluationTests(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ torch.set_num_threads(1)+ cls.tmp = tempfile.TemporaryDirectory()+ cls.addClassCleanup(cls.tmp.cleanup)+ cls.base = Path(cls.tmp.name) / 'tiny'+ cls.base.mkdir()+ torch.manual_seed(7)+ config = BertConfig(vocab_size=16, hidden_size=16, num_hidden_layers=1,+ num_attention_heads=1, intermediate_size=32)+ config.save_pretrained(cls.base / 'encoder')+ backend = Tokenizer(WordLevel({'[PAD]': 0, '[UNK]': 1, '[CLS]': 2, '[SEP]': 3, '[MASK]': 4,+ 'false': 5, 'true': 6, 'billing': 7, 'technical': 8,+ 'shipping': 9, 'other': 10}, unk_token='[UNK]'))+ backend.pre_tokenizer = Whitespace()+ tok = PreTrainedTokenizerFast(tokenizer_object=backend, pad_token='[PAD]', unk_token='[UNK]',+ cls_token='[CLS]', sep_token='[SEP]', mask_token='[MASK]')+ tok.save_pretrained(cls.base / 'tokenizer')+ model = DecisionModel(BertModel(config), head_layers=1)+ save_file(model.state_dict(), cls.base / 'model.safetensors')+ atomic_json(cls.base / 'rl_agent_config.json', {'encoder': 'tiny-test-only', 'head_layers': 1,+ 'act_costs': {'act': 0}, 'temperature': [2., 1., 3.],+ 'temperature_by_options': {'choice:3-5': 1.5}, 'max_len': 128, 'head_max_len': 64})+ cls.rows = read_data(FIXTURES)+ cls.split = partition(cls.rows)++ def adapter(self):+ return Adapter(Agent(str(self.base), device='cpu', fast=False, compile=False),+ {'model_id': 'tiny-mock-only', 'revision': None, 'content_hash': 'tiny', 'trained': False},+ max_len=128, head_max_len=64, dtype='fp32')++ def test_raw_forward_official_parity_and_mask(self):+ a = self.adapter()+ result = a.parity([self.rows[0], self.rows[40]])+ self.assertLessEqual(result['max_absolute_error'], .000051)+ items = [a.prepare(r)[0] for r in (self.rows[0], self.rows[40])]+ b = collate_items([items], a.agent.tok.pad_token_id)+ self.assertEqual(b['marker_mask'].tolist(), [[True, True, False, False], [True] * 4])+ a.agent.model.eval()+ with torch.no_grad():+ z = a.forward(b)+ self.assertTrue(torch.equal(z[0, 2:], torch.full((2,), -1e4)))++ def test_mock_decision_not_action_and_noul_display_labels(self):+ a = self.adapter()+ row = copy.deepcopy(self.rows[0])+ row['question']['labels'] = {'false': '不', 'true': '是'}+ with patch.object(a.agent.model, 'forward', return_value=(torch.tensor([[0., 3.]]), torch.tensor([[999., -999.]]))):+ p = a.predict([row], 'E0', self.split)[0]+ self.assertAlmostEqual(p['p_true'], 1 / (1 + math.exp(-1)))+ a.parity([row])++ def test_calibrated_evaluation_keeps_logits_and_restores_baseline(self):+ from research.benchmarks.zh_reliability.__main__ import evaluate+ adapter = self.adapter()+ weights = {name: value.clone() for name, value in adapter.agent.model.state_dict().items()}+ baseline, base_metrics = evaluate(adapter, self.rows, self.split)+ calibration_rows, _ = evaluate(adapter, self.rows, self.split, selection='calibration')+ artifact = calibrate(calibration_rows, adapter.identity, self.split, adapter.original,+ minimum=1, smoke=True)+ calibrated, calibrated_metrics = evaluate(adapter, self.rows, self.split, artifact, smoke=True)+ self.assertTrue(all(row['condition'] == 'E0' and row['split'] == 'test' for row in baseline))+ self.assertTrue(all(row['condition'] == 'E1' and row['split'] == 'test' for row in calibrated))+ self.assertEqual([row['raw_logits'] for row in baseline], [row['raw_logits'] for row in calibrated])+ self.assertEqual([row['predicted'] for row in baseline], [row['predicted'] for row in calibrated])+ self.assertEqual(base_metrics['overall']['accuracy'], calibrated_metrics['overall']['accuracy'])+ for before, after in zip(baseline, calibrated):+ fitted = artifact['fitted'][after['question']['type']]['temperature']+ np.testing.assert_allclose(after['log_probabilities'], log_probs(before['raw_logits'], fitted))+ restored, restored_metrics = evaluate(adapter, self.rows, self.split)+ self.assertEqual(restored, baseline)+ self.assertEqual(restored_metrics, base_metrics)+ self.assertTrue(all(torch.equal(value, weights[name])+ for name, value in adapter.agent.model.state_dict().items()))++ def test_official_parity_rejects_restored_device_cpu_fallback(self):+ adapter = self.adapter()+ original = adapter.agent.system_one+ def fallback(*args, **kwargs):+ result = original(*args, **kwargs)+ adapter.agent.cpu_fallback_count += 1+ return result+ with patch.object(adapter.agent, 'system_one', side_effect=fallback):+ with self.assertRaisesRegex(RuntimeError, 'fallback'):+ adapter.parity(self.rows[:1])++ def test_token_lengths_state_truncation_and_missing_marker(self):+ a = self.adapter()+ r = copy.deepcopy(self.rows[0])+ r['state'] = 'hello ' * 500+ item, tokens = a.prepare(r)+ self.assertTrue(tokens['state_truncated'])+ self.assertEqual(tokens['input_tokens'], 128)+ self.assertLess(tokens['state_retained'], tokens['state_tokens'])+ a.max_len = 4+ with self.assertRaises(ValueError):+ a.prepare(r)++[email protected](os.environ.get('LAYA_ZH_REAL_MODEL'), 'opt-in: set LAYA_ZH_REAL_MODEL to multilingual local weights')+class RealCheckpointTests(unittest.TestCase):+ def test_official_parity(self):+ if not torch.cuda.is_available():+ self.skipTest('CUDA unavailable')+ # Keep opt-in load within a private copy; never let Agent patch a shared snapshot.+ from research.benchmarks.zh_reliability.runtime import resolve_model+ with tempfile.TemporaryDirectory() as tmp:+ source, identity = resolve_model(os.environ['LAYA_ZH_REAL_MODEL'], 'main', True, tmp)+ agent = Agent(str(source), device='cuda:0', fast=False, compile=False)+ ensure_device(agent, 'cuda:0')+ a = Adapter(agent, identity, 256, 192, 'fp16')+ rows = read_data(FIXTURES)+ a.parity(rows[:2] + rows[40:42])+++if __name__ == '__main__':+ unittest.main()diff --git a/tests/test_zh_results.py b/tests/test_zh_results.pynew file mode 100644index 00000000..35931d15--- /dev/null+++ b/tests/test_zh_results.py@@ -0,0 +1,95 @@+"""Recompute the published E0/E1 evidence and reject inconsistent archived results."""+import copy+import json+from pathlib import Path+import sys+import unittest++sys.path.insert(0, str(Path(__file__).resolve().parents[1]))++from research.benchmarks.zh_reliability.aggregate_massive import render_report, verify_artifact+from research.benchmarks.zh_reliability.data import digest++RESULTS = Path(__file__).resolve().parents[1] / 'research/benchmarks/zh_reliability/results'+++class PublishedResultsTests(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ cls.artifact = json.loads((RESULTS / 'massive_alarm_2026-09-24.json').read_text(encoding='utf-8'))+ cls.result = verify_artifact(cls.artifact)++ def altered(self):+ return copy.deepcopy(self.artifact)++ def test_published_table_recomputes_from_decisions(self):+ self.assertEqual(self.result['counts']['test_source_utterances'], 93)+ self.assertEqual(self.result['counts']['test_groups'], 89)+ self.assertEqual(self.result['counts']['test_decisions'], 186)+ before, after = (self.result[c]['overall'] for c in ('E0', 'E1'))+ self.assertAlmostEqual(before['accuracy'], 99 / 186)+ self.assertEqual(before['accuracy'], after['accuracy'])+ self.assertEqual(before['macro_f1'], after['macro_f1'])+ self.assertAlmostEqual(before['nll'], 1.383747, places=6)+ self.assertAlmostEqual(after['nll'], .785033, places=6)+ self.assertEqual(self.result['fitted']['noul']['temperature'], 5.)+ self.assertTrue(self.result['fitted']['noul']['boundary_hit'])+ paired = self.result['paired']['E1-E0']+ self.assertEqual(paired['repeats'], 1000)+ self.assertEqual(paired['after_minus_before']['accuracy']['ci95'], [0., 0.])+ self.assertLess(paired['after_minus_before']['nll']['ci95'][1], 0.)+ self.assertEqual(render_report(self.artifact, self.result),+ (RESULTS / 'massive_alarm_2026-09-24.md').read_text(encoding='utf-8'))++ def test_records_hash_rejects_changed_logits(self):+ artifact = self.altered()+ artifact['records'][0]['raw_logits'][0] += 1.+ with self.assertRaisesRegex(ValueError, 'record hash mismatch'):+ verify_artifact(artifact, bootstrap=False)++ def test_rehashed_logits_must_match_between_conditions(self):+ artifact = self.altered()+ record = next(r for r in artifact['records'] if r['split'] == 'test')+ record['calibrated_raw_logits'][0] += 1.+ artifact['records_sha256'] = digest(artifact['records'])+ with self.assertRaisesRegex(ValueError, 'changed raw decision logits'):+ verify_artifact(artifact, bootstrap=False)++ def test_rehashed_duplicate_ids_are_rejected(self):+ artifact = self.altered()+ artifact['records'][1]['id'] = artifact['records'][0]['id']+ artifact['records_sha256'] = digest(artifact['records'])+ with self.assertRaisesRegex(ValueError, 'duplicate decision IDs'):+ verify_artifact(artifact, bootstrap=False)++ def test_rehashed_cross_partition_group_is_rejected(self):+ artifact = self.altered()+ calibration = next(r for r in artifact['records'] if r['split'] == 'calibration')+ record = next(r for r in artifact['records'] if r['split'] == 'test')+ record['group_id'] = calibration['group_id']+ artifact['records_sha256'] = digest(artifact['records'])+ with self.assertRaisesRegex(ValueError, 'group leakage'):+ verify_artifact(artifact, bootstrap=False)++ def test_source_pair_hash_must_match(self):+ artifact = self.altered()+ artifact['records'][0]['source_row_hash'] = '0' * 64+ artifact['records_sha256'] = digest(artifact['records'])+ with self.assertRaisesRegex(ValueError, 'source utterance split/group/hash mismatch'):+ verify_artifact(artifact, bootstrap=False)++ def test_temperature_is_refitted_from_calibration_only(self):+ artifact = self.altered()+ artifact['calibration']['fitted']['choice']['temperature'] += .1+ with self.assertRaisesRegex(ValueError, 'fitted temperature/objective mismatch'):+ verify_artifact(artifact, bootstrap=False)++ def test_stored_metric_cannot_override_recomputed_value(self):+ artifact = self.altered()+ artifact['summary']['E1']['overall']['accuracy'] = 1.+ with self.assertRaisesRegex(ValueError, 'stored summary mismatch: E1'):+ verify_artifact(artifact, bootstrap=False)+++if __name__ == '__main__':+ unittest.main()