Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(benchmarks): add Chinese reliability evaluation

Fanrito7c8b8316410d
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 39843b20..ef8a8415 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -148,6 +148,7 @@ jobs:
python tests/test_langchain.py
python tests/test_portability.py
python tests/test_training.py
+ python -m unittest discover -s tests -p 'test_zh_*.py' -v
python tests/test_example_server_limits.py
python tests/test_blank_lang_routing.py
python tests/test_export_onnx_safety.py
@@ -355,9 +356,9 @@ jobs:
python-version: "3.11"
- run: pip install ruff
- name: Ruff
- run: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
+ run: ruff check laya/ research/benchmarks/zh_reliability tests/test_zh_*.py --select=E9,F63,F7,F82,F401,F811 --line-length=120
- name: Byte-compile every module
- run: python -m compileall -q laya/ tests/
+ run: python -m compileall -q laya/ tests/ research/benchmarks/zh_reliability
typescript-sdk:
name: TypeScript SDK (node${{ matrix.node }})
diff --git a/.gitignore b/.gitignore
index 75442a6f..42c9377a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -80,3 +80,6 @@ research/*_benchmark_results.json
# local checkpoint weights, kept out of the tree
# (fetch them with: python verify/checkpoints.py --fetch)
models/
+
+# Local Chinese reliability runs (model copies, data, diagnostics)
+/runs/
diff --git a/research/benchmarks/zh_reliability/README.md b/research/benchmarks/zh_reliability/README.md
new file mode 100644
index 00000000..45f6aa7d
--- /dev/null
+++ b/research/benchmarks/zh_reliability/README.md
@@ -0,0 +1,183 @@
+# Chinese alarm-intent evaluation and temperature calibration
+
+This optional research benchmark measures `choice` and `noul` decisions through Laya's
+existing local Agent, sequence builder and multilingual checkpoint. E0 uses the checkpoint's
+runtime temperatures; E1 fits one bounded scalar temperature per question type on a separate
+calibration partition. It does not update model weights or change the public runtime API.
+
+The contribution is deliberately narrower than a general Chinese benchmark: a three-intent
+MASSIVE alarm subset, grouping checks, auditable calibration/test separation, and paired
+before/after reliability metrics. It complements the broader Chinese evaluation in
+[#557](https://github.com/NandhaKishorM/laya/pull/557), short-command prompt ablations in
+[#364](https://github.com/NandhaKishorM/laya/pull/364), and the generic evaluation harness in
+[#382](https://github.com/NandhaKishorM/laya/pull/382).
+
+## Inspect and recompute the historical result on CPU
+
+[The result table](results/massive_alarm_2026-09-24.md) is generated from
+[compact decision evidence](results/massive_alarm_2026-09-24.json). The JSON contains 128
+calibration and 186 test records with source IDs, group IDs, labels, option order, raw logits,
+actual temperatures and source hashes. Test records retain both runs' raw logits so the verifier
+checks that calibration changed only temperature. It includes original file hashes, model
+identity, selected dependency versions and numerical summaries. Source text, weights, local
+absolute paths, GPU UUIDs and credentials are omitted.
+
+**These are historical measurements from 2026-09-24**, executed on base commit `23a1752`
+with the contribution subsequently recorded in `37cc584`. The execution worktree was dirty;
+its recorded source-file hashes identify the measured implementation. The results were not
+measured against the later upstream rebase or this evaluation-only refactor.
+
+From the repository root, with the project's Python dependencies installed:
+
+```bash
+python -m research.benchmarks.zh_reliability.aggregate_massive \
+ --artifact research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json \
+ --report /tmp/massive-alarm-result.md
+```
+
+This performs no inference or downloads and needs no CUDA/model checkpoint. It verifies record
+integrity, partition/group/source pairing and original temperatures; refits temperatures from
+calibration logits; rebuilds probabilities and metrics; recomputes 1,000 paired group bootstrap
+draws; and compares all numerical summaries with the saved values. Hashes bind the archived
+artifacts; they are not independent authentication of data provenance. Auditing original text
+and annotation declarations additionally requires the prepared public dataset.
+
+| Condition | Accuracy | Macro F1 | NLL | Brier | ECE |
+|---|---:|---:|---:|---:|---:|
+| E0 original temperatures | 0.532258 | 0.440840 | 1.383747 | 0.730071 | 0.366488 |
+| E1 calibrated temperatures | 0.532258 | 0.440840 | 0.785033 | 0.506898 | 0.197529 |
+
+Accuracy and F1 are unchanged because positive scalar temperature preserves argmax. The NLL
+difference is −0.598713, with a group bootstrap 95% interval [−0.865774, −0.369099]. This
+supports a calibration result on this subset, not general Chinese capability improvement.
+Choice temperature is 3.135504; noul reaches the permitted upper bound of 5.0. Risk/coverage
+does not improve at every coverage.
+
+## Prepare public data
+
+Amazon's MASSIVE 1.1 is human-localized virtual-assistant text, licensed CC BY 4.0
+([pinned publisher notice](https://github.com/alexa/massive/blob/f966f21846043aabef9b0f974fa7970027f43738/NOTICE.md)).
+It is not naturally occurring Chinese customer-service logs. The derived tasks use the original
+`alarm_set`, `alarm_query`, and `alarm_remove` labels: choice selects the operation; noul asks
+whether the intent is setting an alarm. No paraphrases or new semantic labels are generated.
+
+```bash
+# Optional dependency for the small Parquet download route only.
+uv pip install --python .venv/bin/python pyarrow
+.venv/bin/python -m research.benchmarks.zh_reliability.prepare_massive \
+ --download-parquet --run-dir runs/massive-alarm-data
+.venv/bin/python -m research.benchmarks.zh_reliability import-data \
+ --data runs/massive-alarm-data/data.jsonl --formal \
+ --split-manifest runs/massive-alarm-data/split_manifest.json \
+ --run-dir runs/massive-import
+```
+
+The Parquet route downloads six Chinese/English files, about 2 MB, from the publisher's
+`AmazonScience/massive` repository at conversion revision
+`ed58ac423a2f4121720918bf5301577edce4ffd3`. It executes no dataset loading script. Embedded
+ClassLabel indices are decoded, and column-oriented annotation lists are transposed without
+changing text or labels. Saved manifests identify transport, decoding rules, reader version,
+file hashes and pinned attribution/legal text. Decoded source-row hashes are not archive-byte
+hashes. Alternatively, `--download` uses the publisher's approximately 40 MB archive and
+records its hash; only selected regular members are read. `--source-dir PATH` converts an
+already downloaded, hash-checked source directory offline. Every output directory must be new.
+
+The fixed admission policy requires at least three distinct upstream reviewers whose intent
+scores all equal 1. It does not imply unanimous grammar/spelling/slot approval. Provenance
+retains these upstream judgments and `human_verified=false`; no independent local review is
+claimed. Other records remain in `diagnostic.jsonl`, with reasons. No model scores determine
+admission, grouping or labels.
+
+Connected groups link identical source IDs, normalized Chinese text and delexicalized English
+source templates. Cross-partition groups keep the highest-priority original partition
+(test > dev > train); lower-priority members are quarantined. Conflicting-label groups are
+quarantined. Calibration uses 20% of the remaining training groups with seed `20260924`.
+Grouping is heuristic and can miss semantic dependencies.
+
+The retained 471 source utterances generate 942 correlated decisions. Fixed partition decision
+counts are train 522, dev 106, calibration 128 and test 186, with 247/53/62/89 groups.
+**This evaluation consumes only calibration and test**; the other source partitions remain
+reserved. Test has 93 utterances, each represented by both question types.
+
+## Run a new evaluation
+
+Use a local copy of `convaiinnovations/laya-multilingual` revision
+`e4e9ddf21a7b1903b7acffd8814ad4307bf63a67`, including the recorded `zh_lineage.json` identifying
+that original revision. The script verifies fixed data/group counts and records paths/hashes
+before predictions. Set `LAYA_ZH_PYTHON` if the interpreter is not `.venv/bin/python`.
+
+On a clean clone, explicitly download that pinned revision and record its original-checkpoint
+lineage. This downloads only runtime files to the Hugging Face cache, copies them into a new
+local model directory, and performs no inference. It does not fall back to `main`:
+
+```bash
+.venv/bin/python - <<'PY_DOWNLOAD'
+from pathlib import Path
+from research.benchmarks.zh_reliability.data import atomic_json
+from research.benchmarks.zh_reliability.runtime import resolve_model
+
+revision = 'e4e9ddf21a7b1903b7acffd8814ad4307bf63a67'
+work = Path('runs/massive-model-download')
+work.mkdir(parents=True) # Refuse an existing output directory.
+model, identity = resolve_model('convaiinnovations/laya-multilingual', revision, False, work)
+if identity['revision'] != revision:
+ raise ValueError('download resolved to an unexpected revision')
+atomic_json(model / 'zh_lineage.json', {
+ 'model_id': identity['model_id'], 'base_revision': revision,
+ 'trained': False, 'weights_hash': identity['files']['model.safetensors']})
+print(model)
+PY_DOWNLOAD
+```
+
+The resulting path is `runs/massive-model-download/model_input`:
+
+```bash
+bash research/benchmarks/zh_reliability/reproduce_massive.sh \
+ runs/new-massive-calibration runs/massive-model-download/model_input \
+ runs/massive-alarm-data/data.jsonl runs/massive-alarm-data/split_manifest.json
+```
+
+The fixed recipe runs only `calibrate`, baseline `eval`, calibrated `eval`, and CPU aggregation.
+It defaults to process-visible `cuda:0`, FP16, max_len=512, and head_max_len=192. Set
+`LAYA_ZH_DEVICE` to a visible device selector, for example `LAYA_ZH_DEVICE="RTX 2080 Ti"`,
+to use another device. The selected value is recorded before predictions in the generated
+protocol; the historical measurements used RTX 2080 Ti. Selectors refer to visible CUDA devices; no device mask or driver is changed. Downloading models is
+not implicit. A failed stage returns nonzero and records its stage in a FAILED manifest.
+Run directories hold local predictions, environment, split, calibration and model identities;
+private tokenizer copies protect the input checkpoint from compatibility patches.
+
+Standalone commands are available with `python -m research.benchmarks.zh_reliability --help`.
+`eval` and `calibrate` require consistent data, `--formal`, `--split-manifest`, model, dtype and
+length settings. Calibration defaults to at least 30 samples per type. Artifact validation
+rejects a different model/data/split or unapproved smoke-only fit. A new completed recipe can
+be checked and its report rebuilt without GPU inference:
+
+```bash
+python -m research.benchmarks.zh_reliability.aggregate_massive runs/new-massive-calibration
+```
+
+The short agent-authored examples in `fixtures.jsonl` remain unverified regression diagnostics;
+passing fixture tests is not a measured public-data result.
+
+## Definitions and limits
+
+- Only valid decision logits are used, never action outputs or rounded public probabilities.
+ Noul option order is `[false,true]`; confidence is max(p). Existing temperatures are not
+ applied twice. The bounded fit minimizes calibration NLL in the runtime range [0.5,5.0].
+- NLL uses float64 log-softmax. Brier is the mean sum of squared class errors. Macro F1 averages
+ task-local macro F1 over the full option set. ECE has 15 equal bins, the last including 1.
+ Risk/coverage breaks confidence ties by lexical sample ID. Type/task/truncation slices and
+ official-Agent probability parity are retained by standalone evaluation.
+- Bootstrap resamples the 89 test groups together across both conditions and question types.
+ Its 1,000 draws use seed `20260924`. Intervals condition on this checkpoint and fitted
+ temperatures, omit calibration uncertainty, and make no corrected multiple-testing claim.
+- This is one task subset and one split/seed. Checkpoint exposure to MASSIVE/SLURP is unknown.
+ It does not establish full MASSIVE performance, general Chinese improvement, or unchanged
+ behavior in other domains. Action outputs are outside scope.
+- Historical test sequences were at most 60 tokens with no truncation. max_len=512 is a limit,
+ not a 512-token stress test. Optional timings synchronize CUDA, distinguish loading and
+ inference memory, and report single-decision batch p50/p95; GPUs are not exclusively reserved.
+
+Offline checks use `python -m unittest discover -s tests -p 'test_zh_*.py' -v`. The archive
+recomputation command above additionally verifies the submitted numerical evidence. Repository
+Ruff and compileall gates apply; no hosted inference service or new runtime dependency is added.
diff --git a/research/benchmarks/zh_reliability/__init__.py b/research/benchmarks/zh_reliability/__init__.py
new file mode 100644
index 00000000..b634e621
--- /dev/null
+++ b/research/benchmarks/zh_reliability/__init__.py
@@ -0,0 +1 @@
+"""Opt-in local Chinese decision evaluation and calibration; not a runtime API."""
diff --git a/research/benchmarks/zh_reliability/__main__.py b/research/benchmarks/zh_reliability/__main__.py
new file mode 100644
index 00000000..c7568bf7
--- /dev/null
+++ b/research/benchmarks/zh_reliability/__main__.py
@@ -0,0 +1,297 @@
+"""Reproducible opt-in experiments: python -m research.benchmarks.zh_reliability --help."""
+import argparse
+from datetime import datetime, timezone
+import json
+from pathlib import Path
+import sys
+import traceback
+import uuid
+
+from .data import FIXTURES, SPLITS, atomic_json, partition, read_data
+from .runtime import environment, select_device
+
+
+def parser():
+ p = argparse.ArgumentParser(description=__doc__)
+ sub = p.add_subparsers(dest='command', required=True)
+ for name in ('check-env', 'import-data', 'smoke', 'eval', 'calibrate', 'report'):
+ s = sub.add_parser(name)
+ s.add_argument('--run-dir', type=Path, help='fresh output directory; existing runs are never overwritten')
+ s.add_argument('--data', type=Path, default=FIXTURES)
+ s.add_argument('--formal', action='store_true', help='require licensed data, documented human annotation and fixed splits')
+ s.add_argument('--split-manifest', type=Path)
+ s.add_argument('--seed', type=int, default=20260924)
+ if name in ('smoke', 'eval', 'calibrate'):
+ s.add_argument('--model', default='convaiinnovations/laya-multilingual')
+ s.add_argument('--revision', default='main', help='resolved to immutable HF SHA before download')
+ s.add_argument('--download', action='store_true', help='allow target-model-only download; default offline')
+ s.add_argument('--device', default='cuda:0', help='unique visible GPU name, UUID or process cuda:index')
+ s.add_argument('--dtype', choices=('fp16', 'fp32'), default='fp16')
+ s.add_argument('--max-len', type=int, default=256 if name == 'smoke' else 512)
+ s.add_argument('--head-max-len', type=int, default=192)
+ if name in ('eval', 'smoke'):
+ s.add_argument('--warmup', type=int, default=10)
+ s.add_argument('--batches', type=int, default=30)
+ if name == 'eval':
+ s.add_argument('--split', choices=SPLITS, default='test')
+ s.add_argument('--calibration', type=Path)
+ s.add_argument('--allow-smoke-calibration', action='store_true')
+ s.add_argument('--measure', action='store_true')
+ if name in ('calibrate', 'smoke'):
+ s.add_argument('--minimum', type=int, default=1 if name == 'smoke' else 30)
+ if name == 'calibrate':
+ s.add_argument('--smoke-only', action='store_true')
+ if name == 'smoke':
+ s.add_argument('--gpu', action='store_true', help='opt in to actual Laya inference, calibration and precision measurements')
+ if name == 'report':
+ s.add_argument('--source', type=Path, required=True, help='existing result directory (read only)')
+ return p
+
+
+def write_predictions(path, rows):
+ # Runs are private until manifest finalization; each row retains raw unrounded decision logits.
+ with Path(path).open('w', encoding='utf-8') as f:
+ for row in rows:
+ f.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + '\n')
+
+
+def report(run, destination=None):
+ run = Path(run)
+ manifest = json.loads((run / 'manifest.json').read_text())
+ metrics = json.loads((run / 'metrics.json').read_text())
+ text = '# Chinese decision reliability run\n\n'
+ text += f"Status: **{manifest['status']}**. Run: `{run.name}`.\n\n"
+ data_info = manifest.get('data_provenance', manifest.get('source_manifest', {}).get('data_provenance', {}))
+ if 'public_human_annotated' in data_info.get('kinds', []):
+ text += 'Public human-localized data with retained upstream annotation evidence; no independent local human review. '
+ text += 'This derived task/subset is not the full source benchmark or evidence of general Chinese improvement. '
+ elif data_info.get('formal'):
+ text += 'Source/annotation declarations and fixed splits were validated; this does not establish representative sampling or independent label correctness. '
+ else:
+ text += 'Agent-authored fixtures are unverified diagnostics, not a standard evaluation or evidence of general Chinese improvement. '
+ text += 'Action outputs are outside this evaluation. No test-set temperature selection.\n\n'
+ text += 'Full slices, per-sample evidence and configuration: `metrics.json`, `predictions.jsonl`, '
+ text += '`split_manifest.json`, `environment.json`, `manifest.json`.\n\n'
+ conditions = [(k, v['overall']) for k, v in metrics.items() if isinstance(v, dict) and 'overall' in v]
+ if 'overall' in metrics:
+ conditions = [('evaluated split', metrics['overall'])]
+ if conditions:
+ text += '| Condition | n | Accuracy | Task macro F1 | NLL | Brier | ECE |\n'
+ text += '|---|---:|---:|---:|---:|---:|---:|\n'
+ for k, m in conditions:
+ values = [str(m['n'])] + [f"{m[x]:.6f}" if m[x] is not None else 'null'
+ for x in ('accuracy', 'macro_f1', 'nll', 'brier', 'ece')]
+ text += '| ' + k + ' | ' + ' | '.join(values) + ' |\n'
+ text += '\n'
+ timing = [(k, v) for k, v in metrics.items() if isinstance(v, dict) and 'end_to_end' in v]
+ if timing:
+ text += '| Device / dtype | E2E batch p50 ms | E2E batch p95 ms | state/s | Model p50 ms | Peak allocated MiB |\n'
+ text += '|---|---:|---:|---:|---:|---:|\n'
+ for k, v in timing:
+ e, m = v['end_to_end'], v['model_stage']
+ text += (f"| {k} | {e['p50_batch_ms']:.3f} | {e['p95_batch_ms']:.3f} | "
+ f"{e['state_per_s']:.2f} | {m['p50_batch_ms']:.3f} | {e['max_memory_allocated']/2**20:.1f} |\n")
+ text += '\nLatency is per batch (one state/decision); model stage excludes H2D/D2H.\n\n'
+ details = {k: v for k, v in metrics.items() if k not in {x[0] for x in conditions + timing}}
+ text += '## Verification and numerical diagnostics\n\n```json\n'
+ text += json.dumps(details, ensure_ascii=False, indent=2, allow_nan=False) + '\n```\n'
+ text += '\n## Execution\n\n```json\n' + json.dumps(manifest, ensure_ascii=False, indent=2) + '\n```\n'
+ target = Path(destination) if destination else run / 'report.md'
+ target.write_text(text, encoding='utf-8')
+
+
+def load_adapter(args, env, run):
+ import torch
+ from laya.agent import Agent
+ from .runtime import Adapter, ensure_device, resolve_model
+ device = select_device(args.device, env)
+ model_input, identity = resolve_model(args.model, args.revision, not args.download, run)
+ torch.cuda.synchronize(device)
+ torch.cuda.reset_peak_memory_stats(device)
+ agent = Agent(str(model_input), device=device, fast=False, compile=False)
+ ensure_device(agent, device)
+ atomic_json(Path(run) / 'loading_memory.json', {'device': device,
+ 'max_memory_allocated': torch.cuda.max_memory_allocated(device),
+ 'max_memory_reserved': torch.cuda.max_memory_reserved(device), 'scope': 'model loading only'})
+ return Adapter(agent, identity, args.max_len, args.head_max_len, args.dtype), model_input
+
+
+def evaluate(adapter, rows, split, calibration=None, smoke=False, selection='test'):
+ import copy
+ from .calibration import validate_calibration
+ from .metrics import summarize
+ adapter.agent.lang_temperatures = (validate_calibration(calibration, adapter.identity, split, smoke)
+ if calibration else copy.deepcopy(adapter.original['lang_temperatures']))
+ condition = 'E1' if calibration else 'E0'
+ selected = [r for r in rows if split['assignments'][r['id']] == selection]
+ predictions = adapter.predict(selected, condition, split)
+ return predictions, summarize(predictions)
+
+
+def gpu_smoke(args, env, run, rows, split, manifest):
+ import copy
+ import numpy as np
+ import torch
+ from .calibration import calibrate
+ from .metrics import paired_bootstrap
+ from .runtime import measure
+ device = select_device(args.device, env)
+ results = {'devices': {args.device: {'status': 'AVAILABLE', 'device': device}}}
+ manifest['gpu_experiment'] = 'PARTIAL'
+ base_dir = run / 'base'
+ base_dir.mkdir()
+ adapter, _ = load_adapter(args, env, base_dir)
+ manifest['model'] = adapter.identity
+ subset = [
+ r for kind in ('choice', 'noul') for r in [x for x in rows if x['question']['type'] == kind][:4]]
+ results['official_parity'] = adapter.parity(subset)
+ cal_rows = [r for r in rows if split['assignments'][r['id']] == 'calibration']
+ base_preds, results['E0'] = evaluate(adapter, rows, split)
+ cal_preds = adapter.predict(cal_rows, 'E0', split)
+ artifact = calibrate(cal_preds, adapter.identity, split, adapter.original,
+ run / 'calibration.json', args.minimum, True)
+ results['calibration_E1'] = artifact['fitted']
+ if any(x['status'] == 'SKIPPED' for x in artifact['fitted'].values()):
+ manifest['status'] = 'PARTIAL'
+ fitted_preds, results['E1'] = evaluate(adapter, rows, split, artifact, True)
+ all_predictions = base_preds + fitted_preds + cal_preds
+ results['E1-E0'] = (paired_bootstrap(base_preds, fitted_preds) if args.formal
+ else {'status': 'SKIPPED', 'reason': 'unverified fixtures cannot support population inference'})
+ manifest['completed'].append('E0/E1')
+ write_predictions(run / 'predictions.jsonl', all_predictions)
+ atomic_json(run / 'metrics.json', results)
+ atomic_json(run / 'manifest.json', manifest)
+ # Precision comparisons use the original checkpoint temperatures and fixed samples.
+ adapter.agent.lang_temperatures = copy.deepcopy(adapter.original['lang_temperatures'])
+ numeric = {}
+ for dtype in ('fp16', 'fp32'):
+ key = args.device + '/' + dtype
+ adapter.dtype = dtype
+ adapter.agent.amp_enabled = dtype == 'fp16'
+ adapter.agent.dtype = torch.float16 if dtype == 'fp16' else torch.float32
+ numeric[dtype] = adapter.predict(subset, 'numeric-' + dtype, split)
+ results[key] = measure(adapter, subset, args.warmup, args.batches)
+ all_predictions += numeric[dtype]
+ write_predictions(run / 'predictions.jsonl', all_predictions)
+ atomic_json(run / 'metrics.json', results)
+ delta = np.concatenate([np.abs(np.array(x['probabilities']) - y['probabilities'])
+ for x, y in zip(numeric['fp16'], numeric['fp32'])])
+ results['numerical_comparisons'] = {args.device + '/fp16 vs ' + args.device + '/fp32': {
+ 'n': len(subset), 'argmax_agreement': float(np.mean([x['predicted'] == y['predicted']
+ for x, y in zip(numeric['fp16'], numeric['fp32'])])),
+ 'probability_abs_mean': float(delta.mean()), 'probability_abs_max': float(delta.max()),
+ 'probability_abs_p50_p95_p99': np.quantile(delta, [.5, .95, .99]).tolist()}}
+ manifest['completed'].append('FP16/FP32 measurements on requested device')
+ return results
+
+
+def main(argv=None):
+ args = parser().parse_args(argv)
+ run = args.run_dir or Path('runs') / (datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ') + '-' + uuid.uuid4().hex[:8])
+ if run.exists():
+ print('refusing existing run directory: ' + str(run), file=sys.stderr)
+ return 2
+ run.mkdir(parents=True)
+ manifest = {'run_id': run.name, 'status': 'RUNNING', 'command': args.command, 'completed': [],
+ 'config': {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()},
+ 'smoke_only': args.command == 'smoke' or getattr(args, 'smoke_only', False),
+ 'gpu_experiment': 'NOT_RUN'}
+ atomic_json(run / 'manifest.json', manifest)
+ atomic_json(run / 'metrics.json', {'status': 'NOT_RUN'})
+ write_predictions(run / 'predictions.jsonl', [])
+ stage = 'environment'
+ try:
+ env = environment()
+ atomic_json(run / 'environment.json', env)
+ if args.command == 'report':
+ import shutil
+ stage = 'report'
+ manifest['source_manifest'] = json.loads((args.source / 'manifest.json').read_text())
+ for name in ('metrics.json', 'predictions.jsonl', 'split_manifest.json'):
+ shutil.copyfile(args.source / name, run / name)
+ shutil.copyfile(args.source / 'environment.json', run / 'source_environment.json')
+ manifest['completed'].append('report generated from ' + str(args.source))
+ else:
+ stage = 'data'
+ rows = read_data(args.data, args.formal)
+ manifest['data_provenance'] = {
+ 'formal': args.formal,
+ 'kinds': sorted({r['provenance']['kind'] for r in rows}),
+ 'licenses': sorted({r['provenance'].get('license', 'unspecified') for r in rows}),
+ 'human_verified_rows': sum(r['provenance']['human_verified'] for r in rows),
+ 'upstream_review_rows': sum('upstream_annotation' in r['provenance'] for r in rows)}
+ split = partition(rows, args.split_manifest, args.seed, args.formal)
+ atomic_json(run / 'split_manifest.json', split)
+ manifest['completed'].append('data/schema/group partitions')
+ atomic_json(run / 'metrics.json', {'data_counts': split['counts'], 'groups': split['group_counts']})
+ if args.command == 'check-env':
+ manifest['completed'].append('CUDA execution probes')
+ if env['gpu_status'] == 'FAILED':
+ raise RuntimeError('CUDA execution probe failed; see environment.json')
+ elif args.command == 'smoke' and not args.gpu:
+ import subprocess
+ stage = 'offline tests'
+ completed = subprocess.run([sys.executable, '-m', 'unittest', 'discover', '-s', 'tests',
+ '-p', 'test_zh_*.py', '-v'], capture_output=True, text=True)
+ (run / 'offline-tests.log').write_text(completed.stdout + completed.stderr)
+ if completed.returncode:
+ raise RuntimeError('offline tests failed; see offline-tests.log')
+ atomic_json(run / 'metrics.json', {'offline_tests': 'SUCCESS', 'gpu': 'NOT_RUN',
+ 'reason': 'default smoke is offline; opt in using --gpu', 'data_counts': split['counts']})
+ manifest['completed'].append('offline tests (tiny/mock inference only)')
+ manifest['status'] = 'PARTIAL'
+ elif args.command == 'smoke':
+ stage = 'GPU smoke'
+ if not env['gpus']:
+ manifest['status'] = 'SKIPPED'
+ manifest['reason'] = 'CUDA unavailable; run default offline smoke'
+ else:
+ results = gpu_smoke(args, env, run, rows, split, manifest)
+ atomic_json(run / 'metrics.json', results)
+ manifest['gpu_experiment'] = ('SUCCESS' if manifest['status'] == 'RUNNING'
+ else manifest['status'])
+ elif args.command not in ('check-env', 'import-data'):
+ stage = 'model loading'
+ adapter, _ = load_adapter(args, env, run)
+ manifest['model'] = adapter.identity
+ stage = args.command
+ if args.command == 'eval':
+ calibration = json.loads(args.calibration.read_text()) if args.calibration else None
+ preds, metrics = evaluate(adapter, rows, split, calibration, args.allow_smoke_calibration, args.split)
+ metrics['official_parity'] = adapter.parity([r for r in rows if split['assignments'][r['id']] == args.split][:4])
+ if args.measure:
+ from .runtime import measure
+ metrics['measurement'] = measure(adapter, [r for r in rows if split['assignments'][r['id']] == args.split],
+ args.warmup, args.batches)
+ write_predictions(run / 'predictions.jsonl', preds)
+ elif args.command == 'calibrate':
+ from .calibration import calibrate
+ preds, metrics = evaluate(adapter, rows, split, selection='calibration')
+ artifact = calibrate(preds, adapter.identity, split, adapter.original, run / 'calibration.json',
+ args.minimum, args.smoke_only)
+ metrics['fitted'] = artifact['fitted']
+ if any(x['status'] == 'SKIPPED' for x in artifact['fitted'].values()):
+ manifest['status'] = 'PARTIAL'
+ write_predictions(run / 'predictions.jsonl', preds)
+ atomic_json(run / 'metrics.json', metrics)
+ manifest['completed'].append(args.command)
+ manifest['gpu_experiment'] = 'SUCCESS'
+ if manifest['status'] == 'RUNNING':
+ manifest['status'] = 'SUCCESS'
+ atomic_json(run / 'manifest.json', manifest)
+ report(run)
+ print(str(run.resolve()))
+ return 0
+ except Exception as e:
+ if manifest['gpu_experiment'] != 'NOT_RUN':
+ manifest['gpu_experiment'] = 'FAILED'
+ manifest.update(status='FAILED', error={'stage': stage, 'type': type(e).__name__, 'reason': str(e)})
+ atomic_json(run / 'manifest.json', manifest)
+ (run / 'error.log').write_text(traceback.format_exc())
+ report(run)
+ print(f'{stage}: {e}; diagnostics: {run}', file=sys.stderr)
+ return 1
+
+
+if __name__ == '__main__':
+ raise SystemExit(main())
diff --git a/research/benchmarks/zh_reliability/aggregate_massive.py b/research/benchmarks/zh_reliability/aggregate_massive.py
new file mode 100644
index 00000000..afadf689
--- /dev/null
+++ b/research/benchmarks/zh_reliability/aggregate_massive.py
@@ -0,0 +1,320 @@
+"""Audit baseline/calibrated runs, or recompute the compact historical evidence on CPU."""
+import argparse
+from collections import Counter
+from datetime import datetime, timezone
+import json
+import math
+from pathlib import Path
+import sys
+import traceback
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
+from research.benchmarks.zh_reliability.calibration import fit_temperature, validate_calibration
+from research.benchmarks.zh_reliability.data import atomic_json, digest, file_hash, partition, read_data
+from research.benchmarks.zh_reliability.metrics import paired_bootstrap, probability_record, summarize
+
+CHILDREN = ('base-calibration', 'E0', 'E1')
+CONDITIONS = ('E0', 'E1')
+SCHEMA = 'massive-alarm-calibration-evidence-v1'
+
+
+def require(value, message):
+ if not value:
+ raise ValueError(message)
+
+
+def read_json(path):
+ return json.loads(Path(path).read_text(encoding='utf-8'))
+
+
+def read_jsonl(path):
+ return [json.loads(line) for line in Path(path).read_text(encoding='utf-8').splitlines() if line.strip()]
+
+
+def numerically_equal(a, b):
+ if isinstance(a, dict) and isinstance(b, dict):
+ return a.keys() == b.keys() and all(numerically_equal(a[k], b[k]) for k in a)
+ if isinstance(a, list) and isinstance(b, list):
+ return len(a) == len(b) and all(numerically_equal(x, y) for x, y in zip(a, b))
+ if isinstance(a, float) and isinstance(b, (int, float)):
+ return math.isclose(a, b, rel_tol=1e-10, abs_tol=1e-10)
+ return a == b
+
+
+def evidence_prediction(record, condition):
+ """Rebuild probabilities without source text, a tokenizer, model weights or CUDA."""
+ row = {'id': record['id'], 'group_id': record['group_id'], 'task_id': record['task_id'],
+ 'question': {'type': record['type'], 'criteria': dict.fromkeys(record['option_order'], '')},
+ 'label': record['gold'], 'tags': ['public_localized', 'alarm']}
+ logits = record['raw_logits'] if condition == 'E0' else record['calibrated_raw_logits']
+ temperature = record['temperature'] if condition == 'E0' else record['calibrated_temperature']
+ result = probability_record(row, logits, temperature)
+ result.update(sample_hash=record['sample_hash'], split=record['split'],
+ tokens={'input_tokens': record['input_tokens'], 'truncated': record['truncated']})
+ return result
+
+
+def compact_summary(rows):
+ result = summarize(rows)
+ return {key: result[key] for key in ('overall', 'by_type', 'definitions')}
+
+
+def recompute(artifact, bootstrap=True):
+ """Recompute evidence and validate pairing/fit; do not trust stored summary values."""
+ require(artifact['schema'] == SCHEMA, 'unsupported evidence schema')
+ records = artifact['records']
+ model = artifact['provenance']['model']
+ require(model['content_hash'] == digest(model['files']), 'model identity hash mismatch')
+ require(model['model_id'] == 'convaiinnovations/laya-multilingual'
+ and model['revision'] == 'e4e9ddf21a7b1903b7acffd8814ad4307bf63a67'
+ and model['trained'] is False, 'expected pinned original multilingual checkpoint')
+ require(artifact['calibration']['minimum_per_type'] == 30 and artifact['calibration']['bounds'] == [.5, 5.],
+ 'calibration protocol mismatch')
+ from laya.common import QTYPES, temp_bucket
+ original = artifact['calibration']['original']
+ temperature_config = original['lang_temperatures'].get('zh', original)
+ require(artifact['records_sha256'] == digest(records), 'decision record hash mismatch')
+ require(len({r['id'] for r in records}) == len(records), 'duplicate decision IDs')
+ require(all(r['split'] in ('calibration', 'test') for r in records), 'unexpected evidence split')
+ groups = {}
+ source_types = {}
+ source_bindings = {}
+ for rec in records:
+ require(rec['type'] in ('choice', 'noul'), 'unsupported question type')
+ require(rec['gold'] in rec['option_order'] and len(set(rec['option_order'])) == len(rec['option_order']),
+ 'invalid gold/options')
+ require(len(rec['raw_logits']) == len(rec['option_order']), 'logit/options mismatch')
+ require(len(rec['sample_hash']) == 64 and len(rec['source_row_hash']) == 64, 'invalid source hash')
+ require(rec['group_id'] not in groups or groups[rec['group_id']] == rec['split'], 'group leakage')
+ groups[rec['group_id']] = rec['split']
+ source = (rec['split'], rec['source_id'])
+ binding = (rec['split'], rec['group_id'], rec['source_row_hash'])
+ require(rec['source_id'] not in source_bindings or source_bindings[rec['source_id']] == binding,
+ 'source utterance split/group/hash mismatch')
+ source_bindings[rec['source_id']] = binding
+ qt = QTYPES[rec['type']]
+ expected_temperature = temperature_config['temperature_by_options'].get(
+ temp_bucket(qt, len(rec['option_order'])), temperature_config['temperature'][qt])
+ require(rec['temperature'] == expected_temperature, 'baseline temperature mismatch')
+ source_types.setdefault(source, []).append(rec['type'])
+ if rec['split'] == 'test':
+ require(rec['calibrated_raw_logits'] == rec['raw_logits'], 'calibration changed raw decision logits')
+ require(all(sorted(types) == ['choice', 'noul'] for types in source_types.values()),
+ 'source utterances must keep both question types together')
+ counts = Counter(r['split'] for r in records)
+ group_counts = Counter(groups.values())
+ require(counts == {'calibration': 128, 'test': 186}, 'fixed subset decision counts mismatch')
+ require(group_counts == {'calibration': 62, 'test': 89}, 'fixed subset group counts mismatch')
+ calibration = [evidence_prediction(r, 'E0') for r in records if r['split'] == 'calibration']
+ fitted = {}
+ for kind in ('choice', 'noul'):
+ selected = [r for r in calibration if r['question']['type'] == kind]
+ require(len(selected) == 64, 'calibration type count mismatch')
+ fitted[kind] = fit_temperature(selected, minimum=30)
+ require(numerically_equal(fitted[kind], artifact['calibration']['fitted'][kind]),
+ 'fitted temperature/objective mismatch: ' + kind)
+ test = [r for r in records if r['split'] == 'test']
+ for rec in test:
+ require(rec['calibrated_temperature'] == artifact['calibration']['fitted'][rec['type']]['temperature'],
+ 'test calibration temperature mismatch')
+ predictions = {c: [evidence_prediction(r, c) for r in test] for c in CONDITIONS}
+ require(all(a['predicted'] == b['predicted'] for a, b in zip(predictions['E0'], predictions['E1'])),
+ 'positive scalar temperature changed argmax')
+ result = {c: compact_summary(predictions[c]) for c in CONDITIONS}
+ result['fitted'] = fitted
+ result['counts'] = {'calibration_decisions': 128, 'calibration_groups': 62,
+ 'test_decisions': 186, 'test_groups': 89, 'test_source_utterances': 93}
+ if bootstrap:
+ config = artifact['analysis']['paired_group_bootstrap']
+ require(config == {'repeats': 1000, 'seed': 20260924, 'min_groups': 20}, 'bootstrap protocol mismatch')
+ result['paired'] = {'E1-E0': paired_bootstrap(predictions['E0'], predictions['E1'], **config)}
+ return result
+
+
+def verify_artifact(artifact, bootstrap=True):
+ """Return recomputed results; reject changed records, fit, pairing or stored metrics."""
+ result = recompute(artifact, bootstrap)
+ for key, value in result.items():
+ require(numerically_equal(value, artifact['summary'][key]), 'stored summary mismatch: ' + key)
+ return result
+
+
+def render_report(artifact, result):
+ provenance = artifact['provenance']
+ lines = ['# MASSIVE Chinese alarm calibration evaluation', '',
+ f"Measurement date: **{provenance['measurement_date']}**. "
+ f"Execution base: `{provenance['execution_base_commit']}`; "
+ f"contribution: `{provenance['contribution_commit']}`.", '',
+ provenance['version_note'], '',
+ 'E0 and E1 use the same multilingual checkpoint and raw decision logits. E1 fits one '
+ 'temperature per question type on calibration data. No model weights are updated.', '',
+ '| Condition | Slice | n | Accuracy | Macro F1 | NLL | Brier | ECE |',
+ '|---|---|---:|---:|---:|---:|---:|---:|']
+ for condition in CONDITIONS:
+ for kind, scores in [('overall', result[condition]['overall']), *result[condition]['by_type'].items()]:
+ values = ' | '.join(f'{scores[k]:.6f}' for k in ('accuracy', 'macro_f1', 'nll', 'brier', 'ece'))
+ lines.append(f"| {condition} | {kind} | {scores['n']} | {values} |")
+ lines += ['', 'Positive scalar temperature preserves argmax: accuracy and F1 are unchanged.', '',
+ '| Type | Calibration n | Temperature | Bound reached |', '|---|---:|---:|---|']
+ for kind, fit in result['fitted'].items():
+ lines.append(f"| {kind} | {fit['n']} | {fit['temperature']:.6f} | {fit['boundary_hit']} |")
+ if 'paired' in result:
+ lines += ['', 'Paired differences are E1 minus E0; negative NLL, Brier and ECE differences are better.', '',
+ '| Metric | Difference | 95% group bootstrap interval |', '|---|---:|---|']
+ for name, value in result['paired']['E1-E0']['after_minus_before'].items():
+ lo, hi = value['ci95']
+ lines.append(f"| {name} | {value['delta']:+.6f} | [{lo:+.6f}, {hi:+.6f}] |")
+ lines += ['', '## Scope and limits', '',
+ '- Test: 93 source utterances, 89 heuristic groups, 186 correlated decisions. Calibration: '
+ '64 source utterances, 62 groups, 128 decisions. The paired question types are not independent samples.',
+ '- This is a derived MASSIVE 1.1 alarm subset with upstream human intent-review evidence, '
+ 'not the full benchmark, naturally occurring customer-service logs, or independent local human review.',
+ '- One fixed split/seed; 1,000 group bootstrap draws. Intervals condition on the fixed checkpoint '
+ 'and fitted temperatures, omit calibration uncertainty, and are descriptive without multiplicity correction.',
+ '- Unknown checkpoint exposure to MASSIVE/SLURP; grouping may miss semantic dependencies. '
+ 'These results do not establish general Chinese improvement or unchanged behavior on other tasks.',
+ '- The baseline noul fit reaches T=5, the permitted upper bound. Calibration settings were fixed '
+ 'without choosing on test. Risk/coverage does not improve at every coverage.',
+ '- Configured max_len=512, head_max_len=192, FP16. Historical test inputs were at most 60 tokens; '
+ 'this is not a 512-token stress measurement.',
+ '- The compact evidence omits source text and retains source IDs/hashes. It can recompute numerical '
+ 'results and check pairing; full source-schema/annotation checks require the prepared public dataset.', '']
+ return '\n'.join(lines)
+
+
+def verify_run_predictions(predictions, rows, split, model, selection, condition):
+ expected = {r['id']: r for r in rows if split['assignments'][r['id']] == selection}
+ require(len(predictions) == len(expected) == len({r['id'] for r in predictions}), 'prediction count/duplicate mismatch')
+ require({r['id'] for r in predictions} == set(expected), 'prediction membership mismatch')
+ for rec in predictions:
+ row = expected[rec['id']]
+ require(rec['split'] == selection and rec['condition'] == condition, 'prediction partition/condition mismatch')
+ require(rec['model'] == model and rec['sample_hash'] == digest(row) == split['row_hashes'][rec['id']],
+ 'prediction model/source mismatch')
+ require(all(rec.get(k) == v for k, v in row.items()), 'prediction source row changed')
+ rebuilt = probability_record(row, rec['raw_logits'], rec['temperature'])
+ require(all(numerically_equal(rec[k], rebuilt[k]) for k in
+ ('option_order', 'gold', 'predicted', 'probabilities', 'log_probabilities', 'confidence', 'p_true')),
+ 'probability reconstruction mismatch')
+
+
+def collect_run(root):
+ """Read only the three evaluation children; never modify the supplied run."""
+ root = Path(root)
+ protocol = read_json(root / 'protocol.json')
+ manifest = read_json(root / 'manifest.json')
+ require(file_hash(root / 'protocol.json') == manifest['protocol_sha256'], 'protocol hash mismatch')
+ rows = read_data(protocol['data'], formal=True)
+ require(file_hash(protocol['data']) == protocol['data_file_sha256'], 'source file hash mismatch')
+ require(file_hash(protocol['split_manifest']) == protocol['split_file_sha256'], 'split file hash mismatch')
+ split = partition(rows, protocol['split_manifest'], 20260924, True)
+ children, predictions, hashes = {}, {}, {}
+ for name in CHILDREN:
+ directory = root / name
+ child = read_json(directory / 'manifest.json')
+ require(child['status'] == 'SUCCESS' and child['gpu_experiment'] == 'SUCCESS' and not child['smoke_only'],
+ name + ': unsuccessful/formally ineligible run')
+ expected = {'formal': True, 'seed': 20260924, 'max_len': 512, 'head_max_len': 192, 'dtype': 'fp16'}
+ require(all(child['config'].get(k) == v for k, v in expected.items()), name + ': config mismatch')
+ require(child['command'] == ('calibrate' if name == 'base-calibration' else 'eval'), 'wrong command')
+ require(read_json(directory / 'split_manifest.json') == split, name + ': split mismatch')
+ require(read_json(directory / 'model_identity.json') == child['model'], name + ': model metadata mismatch')
+ children[name] = child
+ predictions[name] = read_jsonl(directory / 'predictions.jsonl')
+ hashes.update({name + '/' + file: file_hash(directory / file) for file in
+ ('manifest.json', 'predictions.jsonl', 'metrics.json', 'split_manifest.json',
+ 'model_identity.json', 'environment.json')})
+ model = children['E0']['model']
+ require(all(child['model'] == model for child in children.values()), 'model differs between conditions')
+ require(model['content_hash'] == digest(model['files']), 'model identity content hash mismatch')
+ require(model['trained'] is False, 'expected original checkpoint')
+ for name in CHILDREN:
+ verify_run_predictions(predictions[name], rows, split, model,
+ 'calibration' if name == 'base-calibration' else 'test',
+ 'E0' if name == 'base-calibration' else name)
+ calibration = read_json(root / 'base-calibration/calibration.json')
+ validate_calibration(calibration, model, split)
+ require(calibration['minimum_per_type'] == 30 and calibration['smoke_only'] is False, 'invalid calibration minimum/mode')
+ require(calibration['calibration_ids'] == [r['id'] for r in predictions['base-calibration']], 'calibration IDs differ')
+ hashes['base-calibration/calibration.json'] = file_hash(root / 'base-calibration/calibration.json')
+ after = {r['id']: r for r in predictions['E1']}
+ records = []
+ for rec in predictions['base-calibration'] + predictions['E0']:
+ upstream = rec['provenance']['upstream_annotation']
+ compact = {key: rec[key] for key in ('id', 'group_id', 'task_id', 'split', 'sample_hash', 'option_order',
+ 'gold', 'raw_logits', 'temperature')}
+ compact.update(type=rec['question']['type'], source_id=upstream['source_id'],
+ source_row_hash=upstream['source_row_hash'], input_tokens=rec['tokens']['input_tokens'],
+ truncated=rec['tokens']['truncated'])
+ if rec['split'] == 'test':
+ paired = after[rec['id']]
+ require(paired['gold'] == rec['gold'] and paired['group_id'] == rec['group_id'], 'pairing changed')
+ compact.update(calibrated_raw_logits=paired['raw_logits'], calibrated_temperature=paired['temperature'])
+ records.append(compact)
+ environment = read_json(root / 'E0/environment.json')
+ safe_environment = {k: environment[k] for k in ('python', 'os', 'dependencies', 'cuda', 'cudnn')}
+ source_hashes = {Path(k).name: v for k, v in environment.get('source_hashes', {}).items()
+ if Path(k).name in ('__main__.py', 'data.py', 'metrics.py', 'calibration.py', 'runtime.py', 'prepare_massive.py')}
+ artifact = {'schema': SCHEMA, 'provenance': {
+ 'measurement_date': protocol['created_utc'][:10], 'execution_base_commit': environment['commit'],
+ 'contribution_commit': 'unrecorded', 'version_note': 'Source version recorded by the execution environment; '
+ 'recomputing this artifact does not rerun model inference.', 'execution_worktree_dirty': environment['dirty'],
+ 'execution_source_hashes': source_hashes, 'original_file_sha256': hashes,
+ 'dataset': 'MASSIVE 1.1 zh-CN alarm subset', 'dataset_license': 'CC-BY-4.0',
+ 'publisher_notice': 'https://github.com/alexa/massive/blob/f966f21846043aabef9b0f974fa7970027f43738/NOTICE.md',
+ 'data_hash': split['data_hash'], 'split_hash': split['split_hash'],
+ 'data_file_sha256': protocol['data_file_sha256'], 'split_file_sha256': protocol['split_file_sha256'],
+ 'model': model, 'environment': safe_environment,
+ 'evaluation_config': {k: children['E0']['config'][k] for k in
+ ('seed', 'max_len', 'head_max_len', 'dtype', 'device')}},
+ 'analysis': {'paired_group_bootstrap': {'repeats': 1000, 'seed': 20260924, 'min_groups': 20}},
+ 'calibration': {'minimum_per_type': 30, 'bounds': [.5, 5.], 'fitted': calibration['fitted'],
+ 'original': calibration['original']}, 'records_sha256': digest(records), 'records': records}
+ artifact['summary'] = recompute(artifact)
+ for name in CONDITIONS:
+ source = read_json(root / name / 'metrics.json')
+ require(numerically_equal(artifact['summary'][name], {k: source[k] for k in artifact['summary'][name]}),
+ 'recomputed metrics differ from measured child: ' + name)
+ return artifact
+
+
+def main(argv=None):
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument('run_dir', type=Path, nargs='?')
+ parser.add_argument('--artifact', type=Path, help='verify compact evidence without model/data downloads or CUDA')
+ parser.add_argument('--report', type=Path, help='write Markdown rebuilt from verified values')
+ args = parser.parse_args(argv)
+ if (args.run_dir is None) == (args.artifact is None):
+ parser.error('provide either RUN_DIR or --artifact')
+ try:
+ artifact = read_json(args.artifact) if args.artifact else collect_run(args.run_dir)
+ result = verify_artifact(artifact) if args.artifact else artifact['summary']
+ if args.artifact:
+ if args.report:
+ args.report.write_text(render_report(artifact, result), encoding='utf-8')
+ else:
+ root = args.run_dir
+ atomic_json(root / 'calibration_evidence.json', artifact)
+ atomic_json(root / 'metrics.json', result)
+ (args.report or root / 'report.md').write_text(render_report(artifact, result), encoding='utf-8')
+ manifest = read_json(root / 'manifest.json')
+ manifest.update(status='SUCCESS', gpu_experiment='SUCCESS', active_stage='complete', completed=list(CHILDREN),
+ completed_utc=datetime.now(timezone.utc).isoformat(),
+ evidence_sha256=file_hash(root / 'calibration_evidence.json'))
+ atomic_json(root / 'manifest.json', manifest)
+ print('SUCCESS: E0/E1 pairing, fitted temperatures, metrics and 1,000 group bootstrap draws verified')
+ return 0
+ except Exception as error:
+ if args.run_dir:
+ manifest_path = args.run_dir / 'manifest.json'
+ manifest = read_json(manifest_path) if manifest_path.exists() else {}
+ manifest.update(status='FAILED', gpu_experiment='FAILED', active_stage='aggregation',
+ error={'stage': 'aggregation', 'type': type(error).__name__, 'reason': str(error)})
+ atomic_json(manifest_path, manifest)
+ (args.run_dir / 'aggregation-error.log').write_text(traceback.format_exc(), encoding='utf-8')
+ print('verification failed: ' + str(error), file=sys.stderr)
+ return 1
+
+
+if __name__ == '__main__':
+ raise SystemExit(main())
diff --git a/research/benchmarks/zh_reliability/calibration.py b/research/benchmarks/zh_reliability/calibration.py
new file mode 100644
index 00000000..b015a44c
--- /dev/null
+++ b/research/benchmarks/zh_reliability/calibration.py
@@ -0,0 +1,86 @@
+"""Fit bounded scalar temperatures from unscaled calibration logits only."""
+import copy
+import math
+from .data import atomic_json, digest
+from .metrics import log_probs
+
+
+def fit_temperature(rows, minimum=30):
+ from laya.common import TEMP_MIN, TEMP_MAX
+ if len(rows) < minimum:
+ return {'status': 'SKIPPED', 'n': len(rows), 'reason': 'insufficient calibration samples',
+ 'temperature': None, 'boundary_hit': False}
+ def loss(beta):
+ return sum(-log_probs(r['raw_logits'], 1 / beta)[r['option_order'].index(r['gold'])]
+ for r in rows) / len(rows)
+ # NLL is convex in inverse temperature. Bounded golden-section, no new dependency.
+ lo, hi = 1 / TEMP_MAX, 1 / TEMP_MIN
+ ratio = (math.sqrt(5) - 1) / 2
+ for _ in range(80):
+ x, y = hi - ratio * (hi - lo), lo + ratio * (hi - lo)
+ if loss(x) < loss(y):
+ hi = y
+ else:
+ lo = x
+ candidates = [TEMP_MIN, TEMP_MAX, 1., 1 / ((lo + hi) / 2)]
+ t = min(candidates, key=lambda v: (loss(1 / v), abs(v - 1)))
+ return {'status': 'SUCCESS', 'n': len(rows), 'temperature': t,
+ 'boundary_hit': t in (TEMP_MIN, TEMP_MAX), 'nll_t1': loss(1.), 'nll_fitted': loss(1 / t)}
+
+
+def calibrate(rows, model, split, original, path=None, minimum=30, smoke=False):
+ if minimum < 30 and not smoke:
+ raise ValueError('minimum < 30 is smoke_only')
+ if not rows or any(r['split'] != 'calibration' for r in rows):
+ raise ValueError('fit only on calibration partition')
+ if len({r['id'] for r in rows}) != len(rows):
+ raise ValueError('duplicate calibration predictions')
+ for r in rows:
+ if (split['assignments'].get(r['id']) != 'calibration' or r.get('model') != model
+ or r.get('sample_hash') != split['row_hashes'].get(r['id'])):
+ raise ValueError('calibration model/partition mismatch')
+ cfg = copy.deepcopy(original)
+ base = cfg.get('lang_temperatures', {}).get('zh', {
+ 'temperature': cfg['temperature'], 'temperature_by_options': cfg['temperature_by_options']})
+ override = copy.deepcopy(base)
+ fitted = {}
+ for kind, index in [('choice', 0), ('noul', 2)]:
+ fitted[kind] = fit_temperature([r for r in rows if r['question']['type'] == kind], minimum)
+ if fitted[kind]['status'] == 'SUCCESS':
+ override['temperature'][index] = fitted[kind]['temperature']
+ override['temperature_by_options'] = {
+ k: v for k, v in override['temperature_by_options'].items() if not k.startswith(kind + ':')}
+ out = {'model': model, 'data_hash': split['data_hash'], 'split_hash': split['split_hash'],
+ 'calibration_hash': digest([split['row_hashes'][r['id']] for r in rows]),
+ 'calibration_ids': [r['id'] for r in rows], 'smoke_only': smoke,
+ 'minimum_per_type': minimum, 'fitted': fitted, 'original': original,
+ 'lang_temperatures': {**cfg.get('lang_temperatures', {}), 'zh': override}}
+ if path:
+ atomic_json(path, out)
+ return out
+
+
+def validate_calibration(artifact, model, split, allow_smoke=False):
+ from laya.common import clamp_temperature
+ if artifact['model'] != model or artifact['data_hash'] != split['data_hash']:
+ raise ValueError('calibration model or data hash mismatch')
+ ids = artifact['calibration_ids']
+ if (artifact['split_hash'] != split['split_hash'] or len(set(ids)) != len(ids)
+ or any(split['assignments'].get(i) != 'calibration' for i in ids)
+ or artifact['calibration_hash'] != digest([split['row_hashes'][i] for i in ids])):
+ raise ValueError('calibration partition/hash mismatch')
+ if artifact['smoke_only'] and not allow_smoke:
+ raise ValueError('smoke calibration requires explicit --allow-smoke-calibration')
+ for cfg in artifact['lang_temperatures'].values():
+ if len(cfg['temperature']) != 3:
+ raise ValueError('invalid temperature vector')
+ for t in list(cfg['temperature']) + list(cfg['temperature_by_options'].values()):
+ if not isinstance(t, (int, float)) or not math.isfinite(t) or clamp_temperature(t) != t:
+ raise ValueError('illegal exported temperature')
+ for kind, index in [('choice', 0), ('noul', 2)]:
+ fit = artifact['fitted'][kind]
+ cfg = artifact['lang_temperatures']['zh']
+ if fit['status'] == 'SUCCESS' and (cfg['temperature'][index] != fit['temperature'] or
+ any(k.startswith(kind + ':') for k in cfg['temperature_by_options'])):
+ raise ValueError('fitted temperature shadowed by stale option bucket')
+ return artifact['lang_temperatures']
diff --git a/research/benchmarks/zh_reliability/data.py b/research/benchmarks/zh_reliability/data.py
new file mode 100644
index 00000000..60ad8a55
--- /dev/null
+++ b/research/benchmarks/zh_reliability/data.py
@@ -0,0 +1,189 @@
+"""Strict fixture/real-data validation and immutable group partitions."""
+import hashlib
+import json
+import os
+from pathlib import Path
+import random
+import re
+import tempfile
+from collections import Counter
+
+SPLITS = ('train', 'dev', 'calibration', 'test')
+FIXTURES = Path(__file__).with_name('fixtures.jsonl')
+
+
+def digest(value):
+ # Preserve criteria insertion order: order is part of the actual model input.
+ return hashlib.sha256(json.dumps(value, ensure_ascii=False, separators=(',', ':')).encode()).hexdigest()
+
+
+def file_hash(path):
+ h = hashlib.sha256()
+ with open(path, 'rb') as f:
+ for chunk in iter(lambda: f.read(1024 * 1024), b''):
+ h.update(chunk)
+ return h.hexdigest()
+
+
+def atomic_json(path, value):
+ path = Path(path)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ fd, tmp = tempfile.mkstemp(dir=path.parent, prefix='.' + path.name)
+ try:
+ with os.fdopen(fd, 'w', encoding='utf-8') as f:
+ json.dump(value, f, ensure_ascii=False, indent=2, allow_nan=False)
+ f.write('\n')
+ os.replace(tmp, path)
+ finally:
+ if os.path.exists(tmp):
+ os.unlink(tmp)
+
+
+def options(row):
+ return ['false', 'true'] if row['question']['type'] == 'noul' else list(row['question']['criteria'])
+
+
+def upstream_annotation_verified(provenance):
+ """Validate supplied source-review evidence, not a local review of the derived task."""
+ if provenance.get('kind') != 'public_human_annotated' or provenance.get('human_verified') is not False:
+ return False
+ evidence = provenance.get('upstream_annotation')
+ if not isinstance(evidence, dict):
+ return False
+ if (evidence.get('dataset') != 'MASSIVE' or evidence.get('version') != '1.1'
+ or not isinstance(evidence.get('source_id'), str) or not evidence['source_id'].strip()
+ or not isinstance(provenance.get('source_intent'), str) or not provenance['source_intent'].strip()
+ or evidence.get('intent') != provenance['source_intent']):
+ return False
+ source_hash = evidence.get('source_row_hash')
+ evidence_url = evidence.get('evidence_url')
+ if (not isinstance(source_hash, str) or not re.fullmatch(r'[0-9a-f]{64}', source_hash)
+ or not isinstance(evidence_url, str) or not re.fullmatch(
+ r'https://(?:github\.com/alexa/massive/blob/|raw\.githubusercontent\.com/alexa/massive/)'
+ r'[0-9a-f]{40}/README\.md(?:#[A-Za-z0-9_-]+)?', evidence_url)):
+ return False
+ judgments = evidence.get('judgments')
+ if not isinstance(judgments, list) or len(judgments) < 3:
+ return False
+ workers = set()
+ for judgment in judgments:
+ if (not isinstance(judgment, dict) or not isinstance(judgment.get('worker_id'), str)
+ or not judgment['worker_id'].strip() or judgment['worker_id'] in workers
+ or type(judgment.get('intent_score')) is not int or judgment['intent_score'] != 1):
+ return False
+ workers.add(judgment['worker_id'])
+ return True
+
+
+def read_data(path, formal=False):
+ rows = [json.loads(line) for line in Path(path).read_text(encoding='utf-8').splitlines() if line.strip()]
+ if not rows:
+ raise ValueError('empty dataset')
+ seen, tasks, families, relations = set(), {}, {}, {}
+ for r in rows:
+ for key in ('id', 'group_id', 'task_id', 'state'):
+ if not isinstance(r.get(key), str) or not r[key].strip():
+ raise ValueError('nonempty string required: ' + key)
+ if r['id'] in seen:
+ raise ValueError('duplicate id: ' + r['id'])
+ seen.add(r['id'])
+ q = r.get('question', {})
+ if q.get('type') not in ('choice', 'noul') or not isinstance(q.get('instructions'), str):
+ raise ValueError('only choice/noul with string instructions are supported')
+ c = q.get('criteria')
+ if not isinstance(c, dict) or len(c) < 2 or any(not isinstance(v, str) for v in c.values()):
+ raise ValueError('criteria must map at least two stable option IDs to string descriptions')
+ if any(not isinstance(k, str) or not k.strip() for k in c):
+ raise ValueError('invalid option_id')
+ if q['type'] == 'noul' and set(c) != {'false', 'true'}:
+ raise ValueError('noul criteria must be [false, true]')
+ if 'labels' in q:
+ labels = q['labels']
+ if (q['type'] != 'noul' or not isinstance(labels, dict) or set(labels) != {'false', 'true'}
+ or any(not isinstance(x, str) or not x.strip() for x in labels.values())
+ or len(set(labels.values())) != 2):
+ raise ValueError('invalid noul display labels')
+ if r.get('label') not in options(r):
+ raise ValueError('label not in semantic options')
+ if not isinstance(r.get('tags'), list) or any(not isinstance(t, str) for t in r['tags']):
+ raise ValueError('tags must be a list of strings')
+ p = r.get('provenance', {})
+ if not isinstance(p.get('kind'), str) or type(p.get('human_verified')) is not bool:
+ raise ValueError('provenance kind and human_verified required')
+ if p['kind'].startswith('agent_authored') and p['human_verified']:
+ raise ValueError('agent fixtures cannot claim human verification')
+ upstream_verified = upstream_annotation_verified(p)
+ if p['kind'] == 'public_human_annotated' and not upstream_verified:
+ raise ValueError('public human annotations require uncontested pinned source-review evidence')
+ if formal and (p['kind'].startswith('agent_authored') or not (p['human_verified'] or upstream_verified)
+ or any(not isinstance(p.get(k), str) or not p[k].strip()
+ for k in ('source', 'license', 'annotation_status'))
+ or r.get('split') not in SPLITS):
+ raise ValueError('formal data requires source, license, verified annotation and fixed split')
+ task = (q['type'], tuple(sorted(c.items())), q['instructions'])
+ if r['task_id'] in tasks and tasks[r['task_id']] != task:
+ raise ValueError('task_id must have consistent semantic criteria/instructions')
+ tasks[r['task_id']] = task
+ family = p.get('template_family', r['group_id'])
+ if family in families and families[family] != r['group_id']:
+ raise ValueError('template family crosses groups')
+ families[family] = r['group_id']
+ rel = r.get('relation')
+ if rel:
+ if rel.get('kind') not in ('preserve', 'flip') or not isinstance(rel.get('pair_id'), str):
+ raise ValueError('invalid relation')
+ relations.setdefault(rel['pair_id'], []).append(r)
+ for pair in relations.values():
+ if len(pair) != 2 or len({r['group_id'] for r in pair}) != 1:
+ raise ValueError('each relation must pair exactly two rows in the same group')
+ if len({r['relation']['kind'] for r in pair}) != 1 or len({r['task_id'] for r in pair}) != 1:
+ raise ValueError('inconsistent pair')
+ if (pair[0]['label'] == pair[1]['label']) != (pair[0]['relation']['kind'] == 'preserve'):
+ raise ValueError('relation contradicts labels')
+ return rows
+
+
+def partition(rows, path=None, seed=20260924, formal=False):
+ hashes = {r['id']: digest(r) for r in rows}
+ content_hash = digest(hashes)
+ groups = sorted({r['group_id'] for r in rows})
+ if path and Path(path).exists():
+ out = json.loads(Path(path).read_text())
+ if out['data_hash'] != content_hash or out['row_hashes'] != hashes:
+ raise ValueError('split manifest data hash mismatch')
+ else:
+ if formal:
+ mapping = {}
+ for r in rows:
+ if r['group_id'] in mapping and mapping[r['group_id']] != r['split']:
+ raise ValueError('fixed partitions split a group')
+ mapping[r['group_id']] = r['split']
+ else:
+ random.Random(seed).shuffle(groups)
+ n = len(groups)
+ cuts = [round(n * .6), round(n * .7), round(n * .85), n]
+ mapping, start = {}, 0
+ for split, end in zip(SPLITS, cuts):
+ mapping.update({g: split for g in groups[start:end]})
+ start = end
+ out = {'seed': seed, 'formal': formal, 'data_hash': content_hash, 'row_hashes': hashes,
+ 'groups': mapping, 'assignments': {r['id']: mapping[r['group_id']] for r in rows}}
+ if set(out['groups']) != set(groups) or set(out['assignments']) != set(hashes):
+ raise ValueError('split manifest membership mismatch')
+ for r in rows:
+ s = out['assignments'][r['id']]
+ if s not in SPLITS or s != out['groups'][r['group_id']] or (formal and s != r['split']):
+ raise ValueError('group leakage or fixed split mismatch')
+ if out['formal'] != formal:
+ raise ValueError('split mode mismatch')
+ out['counts'] = dict(Counter(out['assignments'].values()))
+ out['group_counts'] = dict(Counter(out['groups'].values()))
+ split_hash = digest(out['assignments'])
+ if 'split_hash' in out and out['split_hash'] != split_hash:
+ raise ValueError('split manifest assignment hash mismatch')
+ out['split_hash'] = split_hash
+ if any(not out['counts'].get(s) for s in SPLITS):
+ raise ValueError('all four partitions must be nonempty')
+ if path and not Path(path).exists():
+ atomic_json(path, out)
+ return out
diff --git a/research/benchmarks/zh_reliability/fixtures.jsonl b/research/benchmarks/zh_reliability/fixtures.jsonl
new file mode 100644
index 00000000..dc37971e
--- /dev/null
+++ b/research/benchmarks/zh_reliability/fixtures.jsonl
@@ -0,0 +1,80 @@
+{"id": "explicit_refund_01_1", "group_id": "explicit_refund_01", "task_id": "explicit_refund", "state": "包裹还在路上,我决定不要了,请给我退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_01"}}
+{"id": "explicit_refund_01_2", "group_id": "explicit_refund_01", "task_id": "explicit_refund", "state": "包裹还在路上,我不想等了,麻烦退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_01"}}
+{"id": "explicit_refund_02_1", "group_id": "explicit_refund_02", "task_id": "explicit_refund", "state": "别退款,我只想知道维修进度。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_02"}}
+{"id": "explicit_refund_02_2", "group_id": "explicit_refund_02", "task_id": "explicit_refund", "state": "维修进度帮我查查,钱先别退。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_02"}}
+{"id": "explicit_refund_03_1", "group_id": "explicit_refund_03", "task_id": "explicit_refund", "state": "如果明天还没发货,我才申请退款,现在先等。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_03"}}
+{"id": "explicit_refund_03_2", "group_id": "explicit_refund_03", "task_id": "explicit_refund", "state": "明天还没发货再退,今天不要操作。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_03"}}
+{"id": "explicit_refund_04_1", "group_id": "explicit_refund_04", "task_id": "explicit_refund", "state": "我不是不想退款,请现在办理。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_04", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_04"}}
+{"id": "explicit_refund_04_2", "group_id": "explicit_refund_04", "task_id": "explicit_refund", "state": "别误会,我确实要把钱退回来。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_04", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_04"}}
+{"id": "explicit_refund_05_1", "group_id": "explicit_refund_05", "task_id": "explicit_refund", "state": "这单不要啦,退下款呗。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_05", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_05"}}
+{"id": "explicit_refund_05_2", "group_id": "explicit_refund_05", "task_id": "explicit_refund", "state": "这单不要啦,不用退款,送给朋友。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_05", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_05"}}
+{"id": "explicit_refund_06_1", "group_id": "explicit_refund_06", "task_id": "explicit_refund", "state": "Please refund 这笔重复扣款,谢谢。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_06", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_06"}}
+{"id": "explicit_refund_06_2", "group_id": "explicit_refund_06", "task_id": "explicit_refund", "state": "Please check 这笔重复扣款,暂时不退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_06", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_06"}}
+{"id": "explicit_refund_07_1", "group_id": "explicit_refund_07", "task_id": "explicit_refund", "state": "天气真热,快递员也辛苦,但买错尺寸了,请退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_07", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_07"}}
+{"id": "explicit_refund_07_2", "group_id": "explicit_refund_07", "task_id": "explicit_refund", "state": "天气真热,快递员也辛苦,尺寸刚好,不用退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_07", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_07"}}
+{"id": "explicit_refund_08_1", "group_id": "explicit_refund_08", "task_id": "explicit_refund", "state": "退款按钮在哪里?我只是看看,还没决定。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_08"}}
+{"id": "explicit_refund_08_2", "group_id": "explicit_refund_08", "task_id": "explicit_refund", "state": "退款按钮在哪里?我决定现在退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_08"}}
+{"id": "explicit_refund_09_1", "group_id": "explicit_refund_09", "task_id": "explicit_refund", "state": "客服说可以退款,但我选择换货,不退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_09", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_09"}}
+{"id": "explicit_refund_09_2", "group_id": "explicit_refund_09", "task_id": "explicit_refund", "state": "客服说可以退款,我选择退钱,不换货。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_09", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_09"}}
+{"id": "explicit_refund_10_1", "group_id": "explicit_refund_10", "task_id": "explicit_refund", "state": "我没有取消退款的打算,继续退吧。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_10", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_10"}}
+{"id": "explicit_refund_10_2", "group_id": "explicit_refund_10", "task_id": "explicit_refund", "state": "我打算取消退款,别继续退了。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_10", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_10"}}
+{"id": "explicit_refund_11_1", "group_id": "explicit_refund_11", "task_id": "explicit_refund", "state": "发票开错了,只重开发票,不涉及退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_11", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_11"}}
+{"id": "explicit_refund_11_2", "group_id": "explicit_refund_11", "task_id": "explicit_refund", "state": "发票不用重开了,这笔费用请直接退回。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_11", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_11"}}
+{"id": "explicit_refund_12_1", "group_id": "explicit_refund_12", "task_id": "explicit_refund", "state": "定金已经交了,活动也取消了,麻烦退定金。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_12"}}
+{"id": "explicit_refund_12_2", "group_id": "explicit_refund_12", "task_id": "explicit_refund", "state": "活动取消了,把我交的定金退回来吧。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_12"}}
+{"id": "explicit_refund_13_1", "group_id": "explicit_refund_13", "task_id": "explicit_refund", "state": "上次说退款是气话,这次只查物流。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_13"}}
+{"id": "explicit_refund_13_2", "group_id": "explicit_refund_13", "task_id": "explicit_refund", "state": "别按上次的气话退款,现在帮我查物流就好。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_13"}}
+{"id": "explicit_refund_14_1", "group_id": "explicit_refund_14", "task_id": "explicit_refund", "state": "会员自动续费我没用过,申清退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["typo"], "relation": {"pair_id": "explicit_refund_14", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_14"}}
+{"id": "explicit_refund_14_2", "group_id": "explicit_refund_14", "task_id": "explicit_refund", "state": "会员续费没用过,申请把费用退回。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["typo"], "relation": {"pair_id": "explicit_refund_14", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_14"}}
+{"id": "explicit_refund_15_1", "group_id": "explicit_refund_15", "task_id": "explicit_refund", "state": "不排除以后退款,但今天没有这个要求。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_15"}}
+{"id": "explicit_refund_15_2", "group_id": "explicit_refund_15", "task_id": "explicit_refund", "state": "以后也许会退款,今天先不退。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_15"}}
+{"id": "explicit_refund_16_1", "group_id": "explicit_refund_16", "task_id": "explicit_refund", "state": "你说不能退款,我不同意,仍要求退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_16", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_16"}}
+{"id": "explicit_refund_16_2", "group_id": "explicit_refund_16", "task_id": "explicit_refund", "state": "你说不能退款,我同意了,不再要求退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_16", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_16"}}
+{"id": "explicit_refund_17_1", "group_id": "explicit_refund_17", "task_id": "explicit_refund", "state": "订单 A 保留;订单 B 重复买了,请退 B。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_17", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_17"}}
+{"id": "explicit_refund_17_2", "group_id": "explicit_refund_17", "task_id": "explicit_refund", "state": "订单 A 和 B 都保留,重复那件送人。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_17", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_17"}}
+{"id": "explicit_refund_18_1", "group_id": "explicit_refund_18", "task_id": "explicit_refund", "state": "refund status 显示处理中,查一下即可,别新建申请。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_18", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_18"}}
+{"id": "explicit_refund_18_2", "group_id": "explicit_refund_18", "task_id": "explicit_refund", "state": "查一下已有 refund 的进度,不是要再申请。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_18", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_18"}}
+{"id": "explicit_refund_19_1", "group_id": "explicit_refund_19", "task_id": "explicit_refund", "state": "不退不是我的意思,我的意思就是退。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_19", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_19"}}
+{"id": "explicit_refund_19_2", "group_id": "explicit_refund_19", "task_id": "explicit_refund", "state": "我的意思就是不退,别理解反了。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_19", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_19"}}
+{"id": "explicit_refund_20_1", "group_id": "explicit_refund_20", "task_id": "explicit_refund", "state": "东西破了,赔个配件就行,不必退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_20"}}
+{"id": "explicit_refund_20_2", "group_id": "explicit_refund_20", "task_id": "explicit_refund", "state": "东西破了,不要配件了,请退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_20"}}
+{"id": "ticket_category_01_1", "group_id": "ticket_category_01", "task_id": "ticket_category", "state": "支付成功后网页又扣了一次钱,请核对账单。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_01"}}
+{"id": "ticket_category_01_2", "group_id": "ticket_category_01", "task_id": "ticket_category", "state": "账单上同一笔支付记了两遍,帮忙核对。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_01"}}
+{"id": "ticket_category_02_1", "group_id": "ticket_category_02", "task_id": "ticket_category", "state": "应用一点登录就闪退,换过网络也一样。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_02"}}
+{"id": "ticket_category_02_2", "group_id": "ticket_category_02", "task_id": "ticket_category", "state": "切了网络仍然登录闪退,看看程序哪里坏了。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_02"}}
+{"id": "ticket_category_03_1", "group_id": "ticket_category_03", "task_id": "ticket_category", "state": "新买的键盘一周没送到,查一下运输位置。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_03"}}
+{"id": "ticket_category_03_2", "group_id": "ticket_category_03", "task_id": "ticket_category", "state": "键盘寄出一周了还没到,帮我追踪包裹。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_03"}}
+{"id": "ticket_category_04_1", "group_id": "ticket_category_04", "task_id": "ticket_category", "state": "不是软件坏了,是账单金额不对。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_04", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_04"}}
+{"id": "ticket_category_04_2", "group_id": "ticket_category_04", "task_id": "ticket_category", "state": "不是账单问题,是软件打开就报错。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_04", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_04"}}
+{"id": "ticket_category_05_1", "group_id": "ticket_category_05", "task_id": "ticket_category", "state": "如果包裹今天到就不用查;现在请处理无法启动的程序。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_05", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_05"}}
+{"id": "ticket_category_05_2", "group_id": "ticket_category_05", "task_id": "ticket_category", "state": "程序无法启动请先处理,物流等今天过后再说。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_05", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_05"}}
+{"id": "ticket_category_06_1", "group_id": "ticket_category_06", "task_id": "ticket_category", "state": "不是没收到包裹,我收到了;现在问如何更改昵称。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_06", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_06"}}
+{"id": "ticket_category_06_2", "group_id": "ticket_category_06", "task_id": "ticket_category", "state": "包裹已经签收,和物流无关,昵称怎么改?", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_06", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_06"}}
+{"id": "ticket_category_07_1", "group_id": "ticket_category_07", "task_id": "ticket_category", "state": "俺的快递跑哪去啦,好几天莫动静。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["colloquial", "option_order"], "relation": {"pair_id": "ticket_category_07", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_07"}}
+{"id": "ticket_category_07_2", "group_id": "ticket_category_07", "task_id": "ticket_category", "state": "快递好几天没动静,帮忙找找在哪。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["colloquial", "option_order"], "relation": {"pair_id": "ticket_category_07", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_07"}}
+{"id": "ticket_category_08_1", "group_id": "ticket_category_08", "task_id": "ticket_category", "state": "App 报 error 503,无法加载首页。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_08"}}
+{"id": "ticket_category_08_2", "group_id": "ticket_category_08", "task_id": "ticket_category", "state": "App 正常,只是 invoice 金额算错。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_08"}}
+{"id": "ticket_category_09_1", "group_id": "ticket_category_09", "task_id": "ticket_category", "state": "昨天喝了咖啡,今天天气不错,顺便问你们周日几点营业?", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_09", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_09"}}
+{"id": "ticket_category_09_2", "group_id": "ticket_category_09", "task_id": "ticket_category", "state": "天气不错,想问周日营业时间。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_09", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_09"}}
+{"id": "ticket_category_10_1", "group_id": "ticket_category_10", "task_id": "ticket_category", "state": "地址写错了,请联系承运方改配送地址。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_10", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_10"}}
+{"id": "ticket_category_10_2", "group_id": "ticket_category_10", "task_id": "ticket_category", "state": "配送地址填错,麻烦帮我联系快递改一下。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_10", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_10"}}
+{"id": "ticket_category_11_1", "group_id": "ticket_category_11", "task_id": "ticket_category", "state": "优惠券抵扣没有显示在最终付款账单中。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_11", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_11"}}
+{"id": "ticket_category_11_2", "group_id": "ticket_category_11", "task_id": "ticket_category", "state": "付款账单没有减去优惠券金额。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_11", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_11"}}
+{"id": "ticket_category_12_1", "group_id": "ticket_category_12", "task_id": "ticket_category", "state": "导出文件时程序卡死,重装也不行。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_12"}}
+{"id": "ticket_category_12_2", "group_id": "ticket_category_12", "task_id": "ticket_category", "state": "重装后导出还是卡死,需要排查程序。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_12"}}
+{"id": "ticket_category_13_1", "group_id": "ticket_category_13", "task_id": "ticket_category", "state": "我想了解你们公司的招聘渠道。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_13"}}
+{"id": "ticket_category_13_2", "group_id": "ticket_category_13", "task_id": "ticket_category", "state": "请告诉我应聘岗位应该去哪里看。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_13"}}
+{"id": "ticket_category_14_1", "group_id": "ticket_category_14", "task_id": "ticket_category", "state": "快递没丢,不用找;发票税额有误,请核算。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_14", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_14"}}
+{"id": "ticket_category_14_2", "group_id": "ticket_category_14", "task_id": "ticket_category", "state": "税额没错,不用算;快递丢了,请查找。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_14", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_14"}}
+{"id": "ticket_category_15_1", "group_id": "ticket_category_15", "task_id": "ticket_category", "state": "包裹若被退回再联系我,目前页面白屏才是问题。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_15"}}
+{"id": "ticket_category_15_2", "group_id": "ticket_category_15", "task_id": "ticket_category", "state": "页面白屏需要修复,包裹退回是还没发生的假设。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_15"}}
+{"id": "ticket_category_16_1", "group_id": "ticket_category_16", "task_id": "ticket_category", "state": "不能说系统没有故障,每次保存都会报错。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_16", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_16"}}
+{"id": "ticket_category_16_2", "group_id": "ticket_category_16", "task_id": "ticket_category", "state": "系统保存每次报错,确实有故障。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_16", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_16"}}
+{"id": "ticket_category_17_1", "group_id": "ticket_category_17", "task_id": "ticket_category", "state": "物流显是签收,可我根本没拿到呀。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["typo", "option_order"], "relation": {"pair_id": "ticket_category_17", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_17"}}
+{"id": "ticket_category_17_2", "group_id": "ticket_category_17", "task_id": "ticket_category", "state": "物流显示已签收,我却没有收到。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["typo", "option_order"], "relation": {"pair_id": "ticket_category_17", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_17"}}
+{"id": "ticket_category_18_1", "group_id": "ticket_category_18", "task_id": "ticket_category", "state": "Tracking number 查不到,帮忙确认是否发货。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_18", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_18"}}
+{"id": "ticket_category_18_2", "group_id": "ticket_category_18", "task_id": "ticket_category", "state": "Tracking 正常,VAT invoice 少了一项费用。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_18", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_18"}}
+{"id": "ticket_category_19_1", "group_id": "ticket_category_19", "task_id": "ticket_category", "state": "我看了三篇教程也问过同事,仍想知道客服几点下班。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_19", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_19"}}
+{"id": "ticket_category_19_2", "group_id": "ticket_category_19", "task_id": "ticket_category", "state": "教程与同事都没告诉我,客服几点下班?", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_19", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_19"}}
+{"id": "ticket_category_20_1", "group_id": "ticket_category_20", "task_id": "ticket_category", "state": "不用改付款记录,也不用修程序,我是来提个配色建议。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_20"}}
+{"id": "ticket_category_20_2", "group_id": "ticket_category_20", "task_id": "ticket_category", "state": "配色挺好,不用改,我要反馈付款记录金额错误。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_20"}}
diff --git a/research/benchmarks/zh_reliability/massive_protocol.json b/research/benchmarks/zh_reliability/massive_protocol.json
new file mode 100644
index 00000000..43e65b09
--- /dev/null
+++ b/research/benchmarks/zh_reliability/massive_protocol.json
@@ -0,0 +1,74 @@
+{
+ "version": "massive-alarm-calibration-v1",
+ "model_revision": "e4e9ddf21a7b1903b7acffd8814ad4307bf63a67",
+ "seed": 20260924,
+ "max_len": 512,
+ "head_max_len": 192,
+ "dtype": "fp16",
+ "evaluation_device": "RTX 2080 Ti",
+ "calibration": {
+ "minimum_per_type": 30,
+ "n_per_type": 64,
+ "bounds": [
+ 0.5,
+ 5.0
+ ],
+ "objective": "calibration partition NLL only",
+ "smoke_only": false
+ },
+ "conditions": {
+ "E0": "original checkpoint with runtime-clamped original temperatures",
+ "E1": "original checkpoint + calibration-fitted temperatures"
+ },
+ "analysis": {
+ "primary_metrics": [
+ "nll",
+ "accuracy"
+ ],
+ "secondary_metrics": [
+ "macro_f1",
+ "brier",
+ "ece"
+ ],
+ "paired_group_bootstrap": {
+ "repeats": 1000,
+ "seed": 20260924,
+ "interval": "percentile 95%",
+ "min_groups": 20
+ },
+ "comparisons": [
+ "E1-E0"
+ ],
+ "interpretation": "Exploratory fixed-checkpoint estimates on 89 heuristic source groups; intervals condition on the fitted temperatures and exclude calibration uncertainty; descriptive intervals without multiplicity correction.",
+ "test_access": "Temperatures and evaluation settings locked before E0/E1 test evaluation; no settings selected using test."
+ },
+ "timing": {
+ "conditions": [
+ "E0"
+ ],
+ "warmup_batches": 10,
+ "measured_batches": 186,
+ "batch_decisions": 1,
+ "note": "all 186 test decisions once per timed stage; short utterances under max_len512, not 512-token stress. GPUs not exclusively reserved."
+ },
+ "limits": [
+ "public human-localized alarm subset, not naturally occurring customer-service logs",
+ "upstream intent-match evidence, no independent local human review",
+ "choice/noul share source text and group; not 942 independent utterances",
+ "base model exposure to MASSIVE/SLURP unknown",
+ "single seed, single task, no general Chinese improvement claim"
+ ],
+ "purpose": "Evaluation-only fixed settings. The runner records paths, hashes and time before predictions; train/dev partitions are retained for source split compatibility and are not consumed.",
+ "expected_split_counts": {
+ "train": 522,
+ "dev": 106,
+ "calibration": 128,
+ "test": 186
+ },
+ "expected_group_counts": {
+ "train": 247,
+ "dev": 53,
+ "calibration": 62,
+ "test": 89
+ }
+}
diff --git a/research/benchmarks/zh_reliability/metrics.py b/research/benchmarks/zh_reliability/metrics.py
new file mode 100644
index 00000000..e323b2f5
--- /dev/null
+++ b/research/benchmarks/zh_reliability/metrics.py
@@ -0,0 +1,126 @@
+"""Metrics use valid decision probabilities, never action/entropy confidence."""
+import math
+from collections import defaultdict
+import numpy as np
+
+
+def log_probs(logits, temperature=1.0, mask=None):
+ from laya.common import TEMP_MIN, TEMP_MAX
+ if not math.isfinite(temperature) or not TEMP_MIN <= temperature <= TEMP_MAX:
+ raise ValueError('temperature outside runtime range')
+ z = np.asarray(logits, dtype=np.float64)
+ if mask is not None:
+ z = z[np.asarray(mask, dtype=bool)]
+ if z.ndim != 1 or not len(z) or not np.isfinite(z).all():
+ raise ValueError('invalid decision logits')
+ z = z / temperature
+ z -= z.max()
+ return z - np.log(np.exp(z).sum())
+
+
+def probability_record(row, logits, temperature):
+ from .data import digest, options
+ order = options(row)
+ if len(logits) != len(order):
+ raise ValueError('logits/options mismatch')
+ lp = log_probs(logits, temperature)
+ p = np.exp(lp)
+ return {**row, 'sample_hash': digest(row), 'option_order': order, 'gold': row['label'], 'raw_logits': list(logits),
+ 'temperature': temperature, 'probabilities': p.tolist(), 'log_probabilities': lp.tolist(),
+ 'predicted': order[int(p.argmax())], 'confidence': float(p.max()),
+ 'p_true': float(p[1]) if row['question']['type'] == 'noul' else None}
+
+
+def quality(rows):
+ if not rows:
+ return {'n': 0, 'accuracy': None, 'macro_f1': None, 'nll': None, 'brier': None, 'ece': None,
+ 'reason': 'empty group'}
+ correct = np.array([r['predicted'] == r['gold'] for r in rows], dtype=float)
+ conf = np.array([max(r['probabilities']) for r in rows])
+ nll, brier = [], []
+ task_rows = defaultdict(list)
+ for r in rows:
+ task_rows[r['task_id']].append(r)
+ g = r['option_order'].index(r['gold'])
+ nll.append(-r['log_probabilities'][g])
+ brier.append(sum((p - (j == g)) ** 2 for j, p in enumerate(r['probabilities'])))
+ f1 = []
+ for task in task_rows.values():
+ labels = sorted({x for r in task for x in r['option_order']})
+ scores = []
+ for label in labels:
+ tp = sum(r['predicted'] == r['gold'] == label for r in task)
+ fp = sum(r['predicted'] == label and r['gold'] != label for r in task)
+ fn = sum(r['predicted'] != label and r['gold'] == label for r in task)
+ scores.append(2 * tp / (2 * tp + fp + fn) if 2 * tp + fp + fn else 0.0)
+ f1.append(float(np.mean(scores)))
+ bucket = np.minimum(14, (conf * 15).astype(int))
+ ece = sum(np.mean(bucket == b) * abs(correct[bucket == b].mean() - conf[bucket == b].mean())
+ for b in range(15) if (bucket == b).any())
+ # Ties broken lexically by stable ID, independently of labels/correctness.
+ ordered = sorted(rows, key=lambda r: (-max(r['probabilities']), r['id']))
+ risk = []
+ for c in (.25, .5, .75, .9, 1.):
+ k = max(1, math.ceil(c * len(rows)))
+ risk.append({'target_coverage': c, 'coverage': k / len(rows), 'n': k,
+ 'risk': sum(r['predicted'] != r['gold'] for r in ordered[:k]) / k})
+ return {'n': len(rows), 'accuracy': float(correct.mean()), 'macro_f1': float(np.mean(f1)),
+ 'nll': float(np.mean(nll)), 'brier': float(np.mean(brier)), 'ece': float(ece),
+ 'risk_coverage': risk}
+
+
+def summarize(rows):
+ result = {'overall': quality(rows), 'definitions': {
+ 'macro_f1': 'equal mean of task-local macro F1; all task options, zero for undefined F1',
+ 'brier': 'mean sum of squared errors over all valid classes (including both noul classes)',
+ 'ece': '15 equal bins [i/15,(i+1)/15), last bin includes 1; confidence=max(p)',
+ 'risk_ties': 'descending max(p), then lexical sample ID; ceil target count; diagnostic only'}}
+ for name, getter in [('task', lambda r: [r['task_id']]),
+ ('type', lambda r: [r['question']['type']]),
+ ('option_count', lambda r: [str(len(r['option_order']))]),
+ ('phenomenon', lambda r: r['tags']),
+ ('truncation', lambda r: [str(r.get('tokens', {}).get('truncated', False))])]:
+ groups = defaultdict(list)
+ for r in rows:
+ for key in getter(r):
+ groups[key].append(r)
+ if name == 'type':
+ for key in ('choice', 'noul'):
+ groups.setdefault(key, [])
+ result['by_' + name] = {k: quality(v) for k, v in sorted(groups.items())}
+ pairs = defaultdict(list)
+ for r in rows:
+ if r.get('relation'):
+ pairs[r['relation']['pair_id']].append(r)
+ result['relations'] = {}
+ for kind in ('preserve', 'flip'):
+ selected = [v for v in pairs.values() if len(v) == 2 and v[0]['relation']['kind'] == kind]
+ result['relations'][kind] = {
+ 'n_pairs': len(selected),
+ 'consistent_or_changed': (sum((v[0]['predicted'] == v[1]['predicted']) == (kind == 'preserve')
+ for v in selected) / len(selected)) if selected else None,
+ 'both_correct': (sum(all(r['predicted'] == r['gold'] for r in v) for v in selected)
+ / len(selected)) if selected else None}
+ return result
+
+
+def paired_bootstrap(before, after, repeats=1000, seed=20260924, min_groups=20):
+ a, b = {r['id']: r for r in before}, {r['id']: r for r in after}
+ if a.keys() != b.keys() or any(a[k]['group_id'] != b[k]['group_id'] or a[k]['gold'] != b[k]['gold'] for k in a):
+ raise ValueError('paired bootstrap requires identical samples/groups/labels')
+ groups = sorted({r['group_id'] for r in before})
+ if len(groups) < min_groups:
+ return {'status': 'SKIPPED', 'reason': 'too few independent groups', 'n_groups': len(groups)}
+ ids = {g: [k for k in a if a[k]['group_id'] == g] for g in groups}
+ rng = np.random.default_rng(seed)
+ keys = ('accuracy', 'macro_f1', 'nll', 'brier', 'ece')
+ deltas = {k: [] for k in keys}
+ for _ in range(repeats):
+ selected = [i for g in rng.choice(groups, len(groups), replace=True) for i in ids[g]]
+ qa, qb = quality([a[i] for i in selected]), quality([b[i] for i in selected])
+ for k in keys:
+ deltas[k].append(qb[k] - qa[k])
+ qa, qb = quality(before), quality(after)
+ return {'status': 'SUCCESS', 'n_groups': len(groups), 'repeats': repeats, 'seed': seed,
+ 'after_minus_before': {k: {'delta': qb[k] - qa[k], 'ci95': np.quantile(v, [.025, .975]).tolist()}
+ for k, v in deltas.items()}}
diff --git a/research/benchmarks/zh_reliability/prepare_massive.py b/research/benchmarks/zh_reliability/prepare_massive.py
new file mode 100644
index 00000000..e31c180e
--- /dev/null
+++ b/research/benchmarks/zh_reliability/prepare_massive.py
@@ -0,0 +1,357 @@
+"""Prepare a licensed, source-reviewed MASSIVE 1.1 Chinese alarm subset (no model calls)."""
+import argparse
+from collections import Counter, defaultdict
+from datetime import datetime, timezone
+import json
+from pathlib import Path
+import random
+import re
+import shutil
+import sys
+import tarfile
+import unicodedata
+import urllib.request
+
+from .data import atomic_json, digest, file_hash, partition, read_data
+
+PUBLISHER_REVISION = 'f966f21846043aabef9b0f974fa7970027f43738'
+ARCHIVE_URL = 'https://amazon-massive-nlu-dataset.s3.amazonaws.com/amazon-massive-dataset-1.1.tar.gz'
+EVIDENCE_URL = f'https://github.com/alexa/massive/blob/{PUBLISHER_REVISION}/README.md'
+RAW_DOCS = f'https://raw.githubusercontent.com/alexa/massive/{PUBLISHER_REVISION}/'
+PARQUET_REVISION = 'ed58ac423a2f4121720918bf5301577edce4ffd3'
+PARQUET_BASE = f'https://huggingface.co/datasets/AmazonScience/massive/resolve/{PARQUET_REVISION}/'
+PARQUET_FILES = tuple(f'{locale}-{split}.parquet' for locale in ('zh-CN', 'en-US')
+ for split in ('train', 'validation', 'test'))
+# The publisher NOTICE establishes that this license applies to MASSIVE.
+LICENSE_TEXT_URL = 'https://raw.githubusercontent.com/creativecommons/cc-legal-tools-data/b40ad648826d12d38207b111d66a708bfa6c5ce4/docs/licenses/by/4.0/legalcode.txt'
+INTENTS = {'alarm_set': '设置闹钟', 'alarm_query': '查询闹钟', 'alarm_remove': '删除或取消闹钟'}
+SOURCE_FILES = ('zh-CN.jsonl', 'en-US.jsonl', 'DATA_LICENSE', 'README.md', 'NOTICE.md', 'THIRD-PARTY.md', 'LICENSE.txt')
+
+
+def fetch_source(directory):
+ directory = Path(directory)
+ directory.mkdir()
+ archive = directory / 'massive-1.1.tar.gz'
+ with urllib.request.urlopen(ARCHIVE_URL, timeout=60) as source, archive.with_suffix('.part').open('wb') as out:
+ shutil.copyfileobj(source, out)
+ archive.with_suffix('.part').replace(archive)
+ with tarfile.open(archive, 'r:gz') as tar:
+ for member in tar:
+ name = member.name
+ target = None
+ if name.endswith('/data/zh-CN.jsonl'):
+ target = 'zh-CN.jsonl'
+ elif name.endswith('/data/en-US.jsonl'):
+ target = 'en-US.jsonl'
+ elif name.endswith('/LICENSE'):
+ target = 'DATA_LICENSE'
+ if target:
+ if not member.isfile() or (directory / target).exists():
+ raise ValueError('unexpected/duplicate archive member: ' + name)
+ # Read only exact members; never extract arbitrary archive paths/symlinks.
+ with tar.extractfile(member) as src, (directory / target).open('wb') as out:
+ shutil.copyfileobj(src, out)
+ for name in SOURCE_FILES[3:]:
+ with urllib.request.urlopen(RAW_DOCS + name, timeout=30) as src:
+ (directory / name).write_bytes(src.read())
+ metadata = {'archive_url': ARCHIVE_URL, 'archive_sha256': file_hash(archive),
+ 'publisher_revision': PUBLISHER_REVISION,
+ 'files': {name: file_hash(directory / name) for name in SOURCE_FILES}}
+ atomic_json(directory / 'source_manifest.json', metadata)
+ return directory
+
+
+def check_source(directory):
+ directory = Path(directory)
+ metadata = json.loads((directory / 'source_manifest.json').read_text())
+ if metadata['archive_url'] != ARCHIVE_URL or metadata['publisher_revision'] != PUBLISHER_REVISION:
+ raise ValueError('unexpected dataset version or documentation revision')
+ transport = metadata.get('transport', 'publisher_archive')
+ if transport not in ('publisher_archive', 'hf_parquet'):
+ raise ValueError('unexpected source transport')
+ if transport == 'hf_parquet' and (metadata.get('mirror_revision') != PARQUET_REVISION
+ or metadata.get('mirror_base_url') != PARQUET_BASE
+ or metadata.get('license_text_url') != LICENSE_TEXT_URL):
+ raise ValueError('unexpected parquet source revision or license URL')
+ source_files = SOURCE_FILES + (PARQUET_FILES if transport == 'hf_parquet' else ())
+ for name in source_files:
+ if metadata['files'].get(name) != file_hash(directory / name):
+ raise ValueError('source file hash mismatch: ' + name)
+ archive = directory / 'massive-1.1.tar.gz'
+ if archive.exists() and file_hash(archive) != metadata['archive_sha256']:
+ raise ValueError('source archive hash mismatch')
+ if 'CC BY 4.0' not in (directory / 'NOTICE.md').read_text():
+ raise ValueError('missing publisher dataset-license notice')
+ if 'Attribution 4.0 International' not in (directory / 'DATA_LICENSE').read_text():
+ raise ValueError('unexpected dataset license')
+ return metadata
+
+
+def decode_parquet_row(row, features, locale, split):
+ """Decode HF label indices and column-oriented sequences without changing source text."""
+ row = dict(row)
+ if row.get('locale') != locale or row.get('partition') != ('dev' if split == 'validation' else split):
+ raise ValueError('parquet locale/partition mismatch')
+ for field in ('scenario', 'intent'):
+ feature, value = features[field], row[field]
+ names = feature.get('names', [])
+ if feature.get('_type') != 'ClassLabel' or type(value) is not int or not 0 <= value < len(names):
+ raise ValueError('invalid parquet class label: ' + field)
+ row[field] = names[value]
+ for field in ('judgments', 'slot_method'):
+ value = row.get(field)
+ if not isinstance(value, dict) or not value or any(not isinstance(v, list) for v in value.values()):
+ raise ValueError('invalid parquet sequence: ' + field)
+ if len({len(v) for v in value.values()}) != 1:
+ raise ValueError('unequal parquet sequence lengths: ' + field)
+ row[field] = [dict(zip(value, values)) for values in zip(*value.values())]
+ return row
+
+
+def prepare_parquet_source(directory):
+ """Decode six pinned files and record artifact and decoded hashes; no network calls."""
+ import pyarrow
+ import pyarrow.parquet as pq
+ directory = Path(directory)
+ for locale in ('zh-CN', 'en-US'):
+ rows = []
+ for split in ('train', 'validation', 'test'):
+ table = pq.read_table(directory / f'{locale}-{split}.parquet')
+ features = json.loads(table.schema.metadata[b'huggingface'])['info']['features']
+ rows.extend(decode_parquet_row(row, features, locale, split) for row in table.to_pylist())
+ write_jsonl(directory / f'{locale}.jsonl', rows)
+ metadata = {'archive_url': ARCHIVE_URL, 'publisher_revision': PUBLISHER_REVISION,
+ 'transport': 'hf_parquet', 'mirror_revision': PARQUET_REVISION, 'mirror_base_url': PARQUET_BASE,
+ 'license_text_url': LICENSE_TEXT_URL, 'pyarrow_version': pyarrow.__version__,
+ 'decoding': 'ClassLabel indices use embedded feature names; column-oriented judgments/slot_method are transposed to lists. Row hashes cover decoded JSON, not original archive bytes.',
+ 'files': {name: file_hash(directory / name) for name in SOURCE_FILES + PARQUET_FILES}}
+ atomic_json(directory / 'source_manifest.json', metadata)
+ return directory
+
+
+def fetch_parquet_source(directory):
+ directory = Path(directory)
+ directory.mkdir()
+ downloads = {f'{locale}-{split}.parquet': f'{PARQUET_BASE}{locale}/{split}/0000.parquet'
+ for locale in ('zh-CN', 'en-US') for split in ('train', 'validation', 'test')}
+ downloads.update({name: RAW_DOCS + name for name in SOURCE_FILES[3:]})
+ downloads['DATA_LICENSE'] = LICENSE_TEXT_URL
+ for name, url in downloads.items():
+ target = directory / name
+ temporary = target.with_suffix(target.suffix + '.part')
+ with urllib.request.urlopen(url, timeout=60) as src, temporary.open('wb') as out:
+ shutil.copyfileobj(src, out)
+ temporary.replace(target)
+ return prepare_parquet_source(directory)
+
+
+def normalized(text):
+ text = unicodedata.normalize('NFKC', text).casefold()
+ return ''.join(c for c in text if not c.isspace() and not unicodedata.category(c).startswith('P'))
+
+
+def source_template(row):
+ # Slot values in English source annotations must not create independent groups.
+ return normalized(re.sub(r'\[\s*([^:\]]+?)\s*:\s*[^\]]*\]', r'[\1]', row['annot_utt']))
+
+
+def records(path):
+ rows = [json.loads(line) for line in Path(path).read_text(encoding='utf-8').splitlines() if line.strip()]
+ if len({r['id'] for r in rows}) != len(rows):
+ raise ValueError('duplicate source ID')
+ return rows
+
+
+def confirmed(row):
+ judgments = row.get('judgments', [])
+ return (isinstance(judgments, list) and len(judgments) >= 3
+ and all(isinstance(j, dict) and isinstance(j.get('worker_id'), str) and j['worker_id'].strip()
+ and type(j.get('intent_score')) is int and j['intent_score'] == 1 for j in judgments)
+ and len({j['worker_id'] for j in judgments}) == len(judgments))
+
+
+def convert(chinese, english, source, seed=20260924, calibration_fraction=.20):
+ if not 0 < calibration_fraction < 1:
+ raise ValueError('calibration fraction must be between zero and one')
+ en = {r['id']: r for r in english}
+ selected = [r for r in chinese if r.get('scenario') == 'alarm' and r.get('intent') in INTENTS]
+ if not selected or len({r['id'] for r in selected}) != len(selected):
+ raise ValueError('empty alarm selection or duplicate source IDs')
+ parent = {r['id']: r['id'] for r in selected}
+
+ def root(i):
+ while parent[i] != i:
+ parent[i] = parent[parent[i]]
+ i = parent[i]
+ return i
+
+ anchors, labels = {}, defaultdict(set)
+ for r in selected:
+ e = en.get(r['id'])
+ if (r.get('locale') != 'zh-CN' or r.get('partition') not in ('train', 'dev', 'test')
+ or not isinstance(r.get('utt'), str) or not normalized(r['utt']) or not e
+ or e.get('locale') != 'en-US' or e['partition'] != r['partition'] or e['intent'] != r['intent']):
+ raise ValueError('Chinese/English source mismatch: ' + str(r['id']))
+ chinese_key = normalized(r['utt'])
+ labels[chinese_key].add(r['intent'])
+ for key in (('zh', chinese_key), ('en_template', source_template(e))):
+ if key in anchors:
+ parent[root(r['id'])] = root(anchors[key])
+ else:
+ anchors[key] = r['id']
+ groups = defaultdict(list)
+ for r in selected:
+ groups[root(r['id'])].append(r)
+ decisions, diagnostic = [], []
+ priority = {'train': 0, 'dev': 1, 'test': 2}
+ for members in groups.values():
+ owner = max((r['partition'] for r in members), key=priority.get)
+ group_id = 'massive-alarm-' + digest(sorted(r['id'] for r in members))[:20]
+ conflict = any(len(labels[normalized(r['utt'])]) > 1 for r in members)
+ for r in members:
+ reasons = []
+ if r['partition'] != owner:
+ reasons.append('shares_source_template_or_text_with_' + owner)
+ if conflict:
+ reasons.append('conflicting_intents_for_identical_normalized_chinese')
+ if not confirmed(r):
+ reasons.append('upstream_intent_review_not_unanimous_or_missing')
+ if reasons:
+ diagnostic.append({'source_id': r['id'], 'group_id': group_id, 'reasons': reasons,
+ 'original_partition': r['partition'], 'source_row': r})
+ else:
+ decisions.append((r, group_id, owner))
+ train_groups = sorted({g for _, g, s in decisions if s == 'train'})
+ if len(train_groups) < 2:
+ raise ValueError('insufficient independent train groups after source-leakage quarantine')
+ random.Random(seed).shuffle(train_groups)
+ cal_groups = set(train_groups[:max(1, min(len(train_groups) - 1, round(len(train_groups) * calibration_fraction)))])
+ output = []
+ for r, group_id, original_split in sorted(decisions, key=lambda x: x[0]['id']):
+ split = 'calibration' if group_id in cal_groups else original_split
+ provenance = {'kind': 'public_human_annotated', 'human_verified': False,
+ 'source': ARCHIVE_URL, 'license': 'CC-BY-4.0', 'source_intent': r['intent'],
+ 'annotation_status': 'Unanimous upstream human intent-match judgments; derived mapping has no independent local human review',
+ 'local_human_review': False, 'template_family': group_id,
+ 'source_partition': original_split, 'source_manifest_hash': digest(source),
+ 'source_transport': source.get('transport', 'publisher_archive'),
+ 'upstream_annotation': {'dataset': 'MASSIVE', 'version': '1.1', 'source_id': r['id'],
+ 'intent': r['intent'], 'source_row_hash': digest(r), 'evidence_url': EVIDENCE_URL,
+ 'judgments': r['judgments']}}
+ if 'archive_sha256' in source:
+ provenance['source_archive_sha256'] = source['archive_sha256']
+ if source.get('transport') == 'hf_parquet':
+ provenance['source_mirror_revision'] = source['mirror_revision']
+ common = {'group_id': group_id, 'state': r['utt'], 'split': split,
+ 'tags': ['public_localized', 'alarm'], 'provenance': provenance}
+ output.append({**common, 'id': 'massive-' + r['id'] + '-choice', 'task_id': 'massive_alarm_choice_v1',
+ 'question': {'type': 'choice', 'instructions': '这句话的主要意图是哪一种闹钟操作?', 'criteria': dict(INTENTS)},
+ 'label': r['intent']})
+ output.append({**common, 'id': 'massive-' + r['id'] + '-noul', 'task_id': 'massive_alarm_set_noul_v1',
+ 'question': {'type': 'noul', 'instructions': '这句话的主要意图是否是设置闹钟?',
+ 'criteria': {'false': '意图是查询、删除或取消闹钟', 'true': '意图是设置闹钟'}},
+ 'label': 'true' if r['intent'] == 'alarm_set' else 'false'})
+ summary = {'source_zh_rows': len(chinese), 'selected_alarm_source_rows': len(selected),
+ 'retained_source_rows': len(decisions), 'diagnostic_source_rows': len(diagnostic),
+ 'source_groups_before_quarantine': len(groups), 'derived_decisions': len(output),
+ 'source_partition_counts': dict(Counter(r['partition'] for r in selected)),
+ 'excluded_reasons': dict(Counter(reason for r in diagnostic for reason in r['reasons'])),
+ 'seed': seed, 'calibration_fraction_of_remaining_train_groups': calibration_fraction,
+ 'local_human_review': False, 'scope': 'MASSIVE-derived source-reviewed alarm subset; not official full benchmark or natural customer-service logs',
+ 'grouping_limit': 'source IDs, exact normalized Chinese and delexicalized English templates; undisclosed semantic/template relations may remain',
+ 'base_model_training_overlap': 'unknown; no unseen-data claim',
+ 'by_split': {s: {'decisions': sum(r['split'] == s for r in output),
+ 'source_rows': sum(('calibration' if g in cal_groups else t) == s for _, g, t in decisions),
+ 'groups': len({r['group_id'] for r in output if r['split'] == s}),
+ 'source_intents': dict(Counter(r['label'] for r in output if r['split'] == s and r['question']['type'] == 'choice'))}
+ for s in ('train', 'dev', 'calibration', 'test')}}
+ return output, diagnostic, summary
+
+
+def write_jsonl(path, rows):
+ with Path(path).open('w', encoding='utf-8') as f:
+ for row in rows:
+ f.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + '\n')
+
+
+def main(argv=None):
+ p = argparse.ArgumentParser(description=__doc__)
+ p.add_argument('--source-dir', type=Path, help='existing source with source_manifest.json; default offline')
+ p.add_argument('--download', action='store_true', help='download publisher release into this new run')
+ p.add_argument('--download-parquet', action='store_true', help='download only zh-CN/en-US from pinned publisher HF mirror; requires pyarrow')
+ p.add_argument('--run-dir', type=Path, required=True, help='new output directory')
+ p.add_argument('--seed', type=int, default=20260924)
+ p.add_argument('--calibration-fraction', type=float, default=.20)
+ args = p.parse_args(argv)
+ if sum((bool(args.source_dir), args.download, args.download_parquet)) != 1:
+ p.error('choose exactly one of --source-dir, --download or --download-parquet')
+ run = args.run_dir
+ if run.exists():
+ print('refusing existing output directory: ' + str(run), file=sys.stderr)
+ return 2
+ run.mkdir(parents=True)
+ manifest = {'status': 'RUNNING', 'command': 'prepare-massive', 'gpu_experiment': 'NOT_RUN',
+ 'created_utc': datetime.now(timezone.utc).isoformat(), 'completed': [],
+ 'converter_sha256': file_hash(__file__),
+ 'validation_sha256': file_hash(Path(__file__).with_name('data.py')),
+ 'config': {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()}}
+ atomic_json(run / 'manifest.json', manifest)
+ stage = 'source'
+ try:
+ if args.download:
+ source_dir = fetch_source(run / 'source')
+ elif args.download_parquet:
+ source_dir = fetch_parquet_source(run / 'source')
+ else:
+ source_dir = args.source_dir
+ source = check_source(source_dir)
+ atomic_json(run / 'source_manifest.json', source)
+ license_dir = run / 'attribution'
+ license_dir.mkdir()
+ for name in SOURCE_FILES[2:]:
+ shutil.copyfile(source_dir / name, license_dir / name)
+ (license_dir / 'DERIVATION.md').write_text(
+ '# Derived MASSIVE alarm subset\n\n'
+ 'MASSIVE Copyright Amazon.com, Inc. or its affiliates. Source: ' + ARCHIVE_URL + '\n\n'
+ 'Dataset and underlying SLURP text: CC BY 4.0; see NOTICE.md and DATA_LICENSE.\n\n'
+ 'Changes: select three alarm intents with unanimous upstream intent-match reviews; '
+ 'quarantine template overlaps and conflicting normalized utterances; derive choice and noul '
+ 'questions; split calibration groups from original train. Original Chinese text and '
+ 'intent labels remain unchanged. No local human review was performed.\n\n'
+ 'Transport and decoding: see ../source_manifest.json. For Parquet, DATA_LICENSE is '
+ 'the CC legal text fetched from its recorded official URL, not archive/LICENSE bytes.\n',
+ encoding='utf-8')
+ manifest['completed'].append('source hashes and publisher attribution')
+ stage = 'conversion'
+ rows, diagnostics, summary = convert(records(source_dir / 'zh-CN.jsonl'), records(source_dir / 'en-US.jsonl'),
+ source, args.seed, args.calibration_fraction)
+ write_jsonl(run / 'data.jsonl', rows)
+ write_jsonl(run / 'diagnostic.jsonl', diagnostics)
+ # Full normal importer validation is a gate, not just converter-specific assertions.
+ validated = read_data(run / 'data.jsonl', formal=True)
+ split = partition(validated, run / 'split_manifest.json', args.seed, formal=True)
+ summary.update(data_hash=split['data_hash'], split_hash=split['split_hash'])
+ atomic_json(run / 'metrics.json', summary)
+ manifest.update(status='SUCCESS', data_sha256=file_hash(run / 'data.jsonl'),
+ diagnostic_sha256=file_hash(run / 'diagnostic.jsonl'))
+ manifest['completed'].append('source-reviewed conversion and fixed group partition validation')
+ atomic_json(run / 'manifest.json', manifest)
+ (run / 'report.md').write_text('# MASSIVE Chinese alarm data preparation\n\n'
+ 'Public human-localized data; no local human review or model evaluation has been performed.\n\n'
+ 'The original Chinese utterances and publisher intent labels are preserved. '
+ 'Choice and noul are deterministic views of the same source and share a group. '
+ 'Official test/dev membership is retained for eligible rows; lower-partition template overlaps '
+ 'and contested/missing human reviews remain in diagnostic.jsonl. This is a filtered subset, '
+ 'not the full MASSIVE benchmark.\n\n```json\n' + json.dumps(summary, ensure_ascii=False, indent=2)
+ + '\n```\n\nAttribution: Amazon MASSIVE and SLURP, CC BY 4.0; see attribution/.\n', encoding='utf-8')
+ print(str(run.resolve()))
+ return 0
+ except Exception as e:
+ manifest.update(status='FAILED', error={'stage': stage, 'type': type(e).__name__, 'reason': str(e)})
+ atomic_json(run / 'manifest.json', manifest)
+ print(f'{stage}: {e}; see {run / "manifest.json"}', file=sys.stderr)
+ return 1
+
+
+if __name__ == '__main__':
+ raise SystemExit(main())
diff --git a/research/benchmarks/zh_reliability/reproduce_massive.sh b/research/benchmarks/zh_reliability/reproduce_massive.sh
new file mode 100644
index 00000000..dae116a1
--- /dev/null
+++ b/research/benchmarks/zh_reliability/reproduce_massive.sh
@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+# Relative paths are interpreted from the repository root. Outputs must be new.
+BENCH_REPO=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../../.." && pwd)
+cd "$BENCH_REPO"
+EXP_DIR=${1:?Usage: bash research/benchmarks/zh_reliability/reproduce_massive.sh OUTPUT_DIR [MODEL_DIR [DATA_JSONL [SPLIT_JSON]]]}
+BENCH_PY=${LAYA_ZH_PYTHON:-.venv/bin/python}
+BENCH_BASE=${2:-${LAYA_ZH_MODEL:-runs/zh-download/e4e9ddf21a7b1903b7acffd8814ad4307bf63a67}}
+BENCH_DATA=${3:-${LAYA_ZH_DATA:-runs/massive-alarm-data/data.jsonl}}
+BENCH_SPLIT=${4:-${LAYA_ZH_SPLIT:-runs/massive-alarm-data/split_manifest.json}}
+BENCH_REVISION=e4e9ddf21a7b1903b7acffd8814ad4307bf63a67
+BENCH_DEVICE=${LAYA_ZH_DEVICE:-cuda:0}
+mkdir -- "$EXP_DIR"
+BENCH_STAGE=initialization
+mark_failed() {
+ BENCH_EXIT_CODE=$?
+ "$BENCH_PY" - "$EXP_DIR" "$BENCH_STAGE" "$BENCH_EXIT_CODE" <<'PY_FAIL'
+import json
+from pathlib import Path
+import sys
+from research.benchmarks.zh_reliability.data import atomic_json
+root, stage, code = Path(sys.argv[1]), sys.argv[2], int(sys.argv[3])
+path = root / 'manifest.json'
+manifest = json.loads(path.read_text()) if path.exists() else {}
+manifest.update(status='FAILED', gpu_experiment='FAILED', active_stage=stage,
+ error={'stage': stage, 'exit_code': code})
+atomic_json(path, manifest)
+PY_FAIL
+ return "$BENCH_EXIT_CODE"
+}
+trap mark_failed ERR
+"$BENCH_PY" - "$EXP_DIR" "$BENCH_BASE" "$BENCH_DATA" "$BENCH_SPLIT" "$BENCH_DEVICE" <<'PY_INIT'
+from datetime import datetime, timezone
+import json
+from pathlib import Path
+import sys
+from research.benchmarks.zh_reliability.data import atomic_json, file_hash, partition, read_data
+
+root, model, data, split_path = (Path(value).resolve() for value in sys.argv[1:5])
+template = Path('research/benchmarks/zh_reliability/massive_protocol.json')
+protocol = json.loads(template.read_text(encoding='utf-8'))
+rows = read_data(data, formal=True)
+split = partition(rows, split_path, protocol['seed'], formal=True)
+if split['counts'] != protocol['expected_split_counts'] or split['group_counts'] != protocol['expected_group_counts']:
+ raise ValueError('data partitions do not match the fixed MASSIVE alarm protocol')
+lineage = json.loads((model / 'zh_lineage.json').read_text(encoding='utf-8'))
+if (lineage.get('model_id') != 'convaiinnovations/laya-multilingual'
+ or lineage.get('base_revision') != protocol['model_revision'] or lineage.get('trained') is not False):
+ raise ValueError('use the pinned original multilingual checkpoint with its zh_lineage.json')
+protocol['evaluation_device'] = sys.argv[5]
+protocol.update(created_utc=datetime.now(timezone.utc).isoformat(), status='FIXED_BEFORE_MODEL_PREDICTIONS',
+ data=str(data), data_file_sha256=file_hash(data), split_manifest=str(split_path),
+ split_file_sha256=file_hash(split_path), base_model=str(model),
+ base_model_weights_sha256=file_hash(model / 'model.safetensors'),
+ base_model_lineage_sha256=file_hash(model / 'zh_lineage.json'),
+ template_sha256=file_hash(template))
+atomic_json(root / 'protocol.json', protocol)
+atomic_json(root / 'manifest.json', {
+ 'status': 'RUNNING', 'gpu_experiment': 'RUNNING', 'command': 'formal-massive-calibration-replay',
+ 'completed': [], 'protocol_sha256': file_hash(root / 'protocol.json')})
+PY_INIT
+run_benchmark() {
+ "$BENCH_PY" -m research.benchmarks.zh_reliability "$@" --data "$BENCH_DATA" --formal \
+ --split-manifest "$BENCH_SPLIT" --max-len 512 --head-max-len 192 --dtype fp16 \
+ --seed 20260924 --revision "$BENCH_REVISION"
+}
+BENCH_STAGE=base-calibration
+run_benchmark calibrate --model "$BENCH_BASE" --device "$BENCH_DEVICE" --minimum 30 \
+ --run-dir "$EXP_DIR/base-calibration"
+BENCH_STAGE=E0
+run_benchmark eval --model "$BENCH_BASE" --device "$BENCH_DEVICE" --split test \
+ --measure --warmup 10 --batches 186 --run-dir "$EXP_DIR/E0"
+BENCH_STAGE=E1
+run_benchmark eval --model "$BENCH_BASE" --device "$BENCH_DEVICE" --split test \
+ --calibration "$EXP_DIR/base-calibration/calibration.json" --run-dir "$EXP_DIR/E1"
+BENCH_STAGE=aggregation
+"$BENCH_PY" -m research.benchmarks.zh_reliability.aggregate_massive "$EXP_DIR"
diff --git a/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json
new file mode 100644
index 00000000..35377d4b
--- /dev/null
+++ b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json
@@ -0,0 +1,776 @@
+{
+ "schema": "massive-alarm-calibration-evidence-v1",
+ "provenance": {
+ "measurement_date": "2026-09-24",
+ "execution_base_commit": "23a17522aa4942da6cce53a995a275760320b691",
+ "contribution_commit": "37cc58465da5cdabddf1ff543234397d666ad546",
+ "version_note": "Historical inference was executed on 2026-09-24 from base commit 23a17522aa4942da6cce53a995a275760320b691 with a dirty contribution worktree subsequently recorded in 37cc58465da5cdabddf1ff543234397d666ad546. The recorded execution source-file hashes are authoritative for the measured code. These measurements were not rerun on the later upstream rebase or evaluation-only refactor.",
+ "execution_worktree_dirty": true,
+ "execution_source_hashes": {
+ "__main__.py": "2db327517919084fa3257659a168442024952fd36c649a37869cd460275cf692",
+ "calibration.py": "0bd68229d859cf55b0826a43f1a43789edc9cb9c07ee849e2144e6ad747263c7",
+ "data.py": "baf8061a097c0993fe31d8c7d69db8ccddd6d714075f3fc717b8ddf12ce11c66",
+ "metrics.py": "8c0be190b7c9e1d55ce5ebe7a5b8aff6ab8c8ae24183ecb59f7838cf7867925f",
+ "prepare_massive.py": "a83c5f74285b6ccc03851efebbbc9a77bf220d891d78ff8e4faddf9a22944e95",
+ "runtime.py": "8c28e4d8113d3fc25ad1ca8f06d1d8254a4334fbeaff23e93c8098db0b25cfa9"
+ },
+ "original_file_sha256": {
+ "base-calibration/manifest.json": "106c40af31bdfbc436ce68c6482ec36cef747566ff6d428f5335cdc2061da0ac",
+ "base-calibration/predictions.jsonl": "c763309e5f81a8cc97656c89d8a7ee945914ca601a9c9cc6402ce77235315e35",
+ "base-calibration/metrics.json": "07d3ea93222c202d1bc95757fee99c064c9448bc324eaaa759f5d06429426384",
+ "base-calibration/split_manifest.json": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",
+ "base-calibration/model_identity.json": "144087e7a313e8b144e2dc5296d502750c6f1d2223d45910038ccdddcfcabf19",
+ "base-calibration/environment.json": "0b8b2a2d2cc8b2fdeef997c608b92309fce3fe8bdb2fde2b2fdaa1a29dd362b2",
+ "E0/manifest.json": "e3c9a4791c3c2f152944d2245e6cc05a8b78e881de1283f90fe50f438722fb78",
+ "E0/predictions.jsonl": "6524bdd74d10b513e6b6cb468f1993a8ff1d74abfda69eb767efdd41213c6f96",
+ "E0/metrics.json": "b5bd3a28d867745e4f44c25bc1fdf4e395cf328b95b384f7610567f9635bd933",
+ "E0/split_manifest.json": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",
+ "E0/model_identity.json": "144087e7a313e8b144e2dc5296d502750c6f1d2223d45910038ccdddcfcabf19",
+ "E0/environment.json": "0b8b2a2d2cc8b2fdeef997c608b92309fce3fe8bdb2fde2b2fdaa1a29dd362b2",
+ "E1/manifest.json": "b60a71273d9f3a5d48debe5040107a36af045d8ed7447612e3b576242bf6f284",
+ "E1/predictions.jsonl": "393009f41ba9e9d44f0ad88926b85f95896f40c922d1bfbf24759669a162b763",
+ "E1/metrics.json": "b6d6cc8bfe1a3376648b17d79ac25b7706a0c98ca586a2c3b6e6d5a891cf7451",
+ "E1/split_manifest.json": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",
+ "E1/model_identity.json": "144087e7a313e8b144e2dc5296d502750c6f1d2223d45910038ccdddcfcabf19",
+ "E1/environment.json": "0b8b2a2d2cc8b2fdeef997c608b92309fce3fe8bdb2fde2b2fdaa1a29dd362b2",
+ "base-calibration/calibration.json": "b5e8b409cb5fb340a4911109327f166c85437307a5684235509ca0c1ada92ae2"
+ },
+ "dataset": "MASSIVE 1.1 zh-CN alarm subset",
+ "dataset_license": "CC-BY-4.0",
+ "publisher_notice": "https://github.com/alexa/massive/blob/f966f21846043aabef9b0f974fa7970027f43738/NOTICE.md",
+ "data_hash": "dd800e8bc62366cbcf06907e63a1cf225c9aa1add35aa9892fe567e07d470b16",
+ "split_hash": "9abde4aa5ecbc90de2929617b72f380ab18f51b5bac38659f7c5641c7739c707",
+ "data_file_sha256": "e97c52a226d202a6a3a654d6645b7caff5b3a48952258e8d67149778faa4939a",
+ "split_file_sha256": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",
+ "model": {
+ "model_id": "convaiinnovations/laya-multilingual",
+ "revision": "e4e9ddf21a7b1903b7acffd8814ad4307bf63a67",
+ "content_hash": "27d6023d6a2033c2c8e504d88ede242c0f7faf7ffd85781540d5fb935bbde855",
+ "files": {
+ "model.safetensors": "9d628fd971b700382ac6f65920a86f149777b2e748e0c955fb3b19695aa8f204",
+ "rl_agent_config.json": "25061739243b617ad88d1219ba6f8a9c86c5881ca28df024fa2d9b3b2fcc30c6",
+ "encoder/config.json": "83f6916d13ef0f556ac461f28308dc2bffa7ebeadee8ec9e2db5812020ea5bb4",
+ "tokenizer/tokenizer.json": "609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f",
+ "tokenizer/tokenizer_config.json": "424b69444bf7b5809dc2cd2e36d0bd71b8055124dd24274d6db3c655d38205e7"
+ },
+ "trained": false
+ },
+ "environment": {
+ "python": "3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]",
+ "os": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.39",
+ "dependencies": {
+ "torch": "2.7.1+cu126",
+ "transformers": "4.57.6",
+ "numpy": "2.5.3",
+ "safetensors": "0.8.0",
+ "huggingface_hub": "0.36.2",
+ "tokenizers": "0.22.2"
+ },
+ "cuda": "12.6",
+ "cudnn": 90501
+ },
+ "evaluation_config": {
+ "seed": 20260924,
+ "max_len": 512,
+ "head_max_len": 192,
+ "dtype": "fp16",
+ "device": "RTX 2080 Ti"
+ }
+ },
+ "analysis": {
+ "paired_group_bootstrap": {
+ "repeats": 1000,
+ "seed": 20260924,
+ "min_groups": 20
+ }
+ },
+ "calibration": {
+ "minimum_per_type": 30,
+ "bounds": [
+ 0.5,
+ 5.0
+ ],
+ "fitted": {
+ "choice": {
+ "status": "SUCCESS",
+ "n": 64,
+ "temperature": 3.1355035588563744,
+ "boundary_hit": false,
+ "nll_t1": 1.21606373270805,
+ "nll_fitted": 0.771914452059509
+ },
+ "noul": {
+ "status": "SUCCESS",
+ "n": 64,
+ "temperature": 5.0,
+ "boundary_hit": true,
+ "nll_t1": 1.5203078336357765,
+ "nll_fitted": 0.7411119749553794
+ }
+ },
+ "original": {
+ "temperature_raw": [
+ 1.0,
+ 1.0,
+ 1.0
+ ],
+ "temperature_by_options_raw": {},
+ "temperature": [
+ 1.0,
+ 1.0,
+ 1.0
+ ],
+ "temperature_by_options": {},
+ "lang_temperatures": {}
+ }
+ },
+ "records_sha256": "570f24fe7110855ac974d12903e45579c496d853625a45e916e20f2381e28852",
+ "summary": {
+ "E0": {
+ "overall": {
+ "n": 186,
+ "accuracy": 0.532258064516129,
+ "macro_f1": 0.4408400882085093,
+ "nll": 1.3837465414314734,
+ "brier": 0.7300712607122152,
+ "ece": 0.36648789161804096,
+ "risk_coverage": [
+ {
+ "target_coverage": 0.25,
+ "coverage": 0.25268817204301075,
+ "n": 47,
+ "risk": 0.1702127659574468
+ },
+ {
+ "target_coverage": 0.5,
+ "coverage": 0.5,
+ "n": 93,
+ "risk": 0.27956989247311825
+ },
+ {
+ "target_coverage": 0.75,
+ "coverage": 0.7526881720430108,
+ "n": 140,
+ "risk": 0.4
+ },
+ {
+ "target_coverage": 0.9,
+ "coverage": 0.9032258064516129,
+ "n": 168,
+ "risk": 0.43452380952380953
+ },
+ {
+ "target_coverage": 1.0,
+ "coverage": 1.0,
+ "n": 186,
+ "risk": 0.46774193548387094
+ }
+ ]
+ },
+ "by_type": {
+ "choice": {
+ "n": 93,
+ "accuracy": 0.6344086021505376,
+ "macro_f1": 0.5809282967177705,
+ "nll": 1.2960431695747896,
+ "brier": 0.5208253347319968,
+ "ece": 0.2534260594442905,
+ "risk_coverage": [
+ {
+ "target_coverage": 0.25,
+ "coverage": 0.25806451612903225,
+ "n": 24,
+ "risk": 0.2916666666666667
+ },
+ {
+ "target_coverage": 0.5,
+ "coverage": 0.5053763440860215,
+ "n": 47,
+ "risk": 0.19148936170212766
+ },
+ {
+ "target_coverage": 0.75,
+ "coverage": 0.7526881720430108,
+ "n": 70,
+ "risk": 0.2571428571428571
+ },
+ {
+ "target_coverage": 0.9,
+ "coverage": 0.9032258064516129,
+ "n": 84,
+ "risk": 0.3333333333333333
+ },
+ {
+ "target_coverage": 1.0,
+ "coverage": 1.0,
+ "n": 93,
+ "risk": 0.3655913978494624
+ }
+ ]
+ },
+ "noul": {
+ "n": 93,
+ "accuracy": 0.43010752688172044,
+ "macro_f1": 0.3007518796992481,
+ "nll": 1.4714499132881573,
+ "brier": 0.9393171866924332,
+ "ece": 0.49461061636621556,
+ "risk_coverage": [
+ {
+ "target_coverage": 0.25,
+ "coverage": 0.25806451612903225,
+ "n": 24,
+ "risk": 0.20833333333333334
+ },
+ {
+ "target_coverage": 0.5,
+ "coverage": 0.5053763440860215,
+ "n": 47,
+ "risk": 0.3829787234042553
+ },
+ {
+ "target_coverage": 0.75,
+ "coverage": 0.7526881720430108,
+ "n": 70,
+ "risk": 0.4857142857142857
+ },
+ {
+ "target_coverage": 0.9,
+ "coverage": 0.9032258064516129,
+ "n": 84,
+ "risk": 0.5357142857142857
+ },
+ {
+ "target_coverage": 1.0,
+ "coverage": 1.0,
+ "n": 93,
+ "risk": 0.5698924731182796
+ }
+ ]
+ }
+ },
+ "definitions": {
+ "macro_f1": "equal mean of task-local macro F1; all task options, zero for undefined F1",
+ "brier": "mean sum of squared errors over all valid classes (including both noul classes)",
+ "ece": "15 equal bins [i/15,(i+1)/15), last bin includes 1; confidence=max(p)",
+ "risk_ties": "descending max(p), then lexical sample ID; ceil target count; diagnostic only"
+ }
+ },
+ "E1": {
+ "overall": {
+ "n": 186,
+ "accuracy": 0.532258064516129,
+ "macro_f1": 0.4408400882085093,
+ "nll": 0.7850334112663782,
+ "brier": 0.5068983072913132,
+ "ece": 0.1975294208240641,
+ "risk_coverage": [
+ {
+ "target_coverage": 0.25,
+ "coverage": 0.25268817204301075,
+ "n": 47,
+ "risk": 0.19148936170212766
+ },
+ {
+ "target_coverage": 0.5,
+ "coverage": 0.5,
+ "n": 93,
+ "risk": 0.2903225806451613
+ },
+ {
+ "target_coverage": 0.75,
+ "coverage": 0.7526881720430108,
+ "n": 140,
+ "risk": 0.40714285714285714
+ },
+ {
+ "target_coverage": 0.9,
+ "coverage": 0.9032258064516129,
+ "n": 168,
+ "risk": 0.43452380952380953
+ },
+ {
+ "target_coverage": 1.0,
+ "coverage": 1.0,
+ "n": 186,
+ "risk": 0.46774193548387094
+ }
+ ]
+ },
+ "by_type": {
+ "choice": {
+ "n": 93,
+ "accuracy": 0.6344086021505376,
+ "macro_f1": 0.5809282967177705,
+ "nll": 0.8372707851828709,
+ "brier": 0.4740320871109113,
+ "ece": 0.16990508653316772,
+ "risk_coverage": [
+ {
+ "target_coverage": 0.25,
+ "coverage": 0.25806451612903225,
+ "n": 24,
+ "risk": 0.2916666666666667
+ },
+ {
+ "target_coverage": 0.5,
+ "coverage": 0.5053763440860215,
+ "n": 47,
+ "risk": 0.19148936170212766
+ },
+ {
+ "target_coverage": 0.75,
+ "coverage": 0.7526881720430108,
+ "n": 70,
+ "risk": 0.24285714285714285
+ },
+ {
+ "target_coverage": 0.9,
+ "coverage": 0.9032258064516129,
+ "n": 84,
+ "risk": 0.32142857142857145
+ },
+ {
+ "target_coverage": 1.0,
+ "coverage": 1.0,
+ "n": 93,
+ "risk": 0.3655913978494624
+ }
+ ]
+ },
+ "noul": {
+ "n": 93,
+ "accuracy": 0.43010752688172044,
+ "macro_f1": 0.3007518796992481,
+ "nll": 0.7327960373498855,
+ "brier": 0.5397645274717153,
+ "ece": 0.2896348457994376,
+ "risk_coverage": [
+ {
+ "target_coverage": 0.25,
+ "coverage": 0.25806451612903225,
+ "n": 24,
+ "risk": 0.20833333333333334
+ },
+ {
+ "target_coverage": 0.5,
+ "coverage": 0.5053763440860215,
+ "n": 47,
+ "risk": 0.3829787234042553
+ },
+ {
+ "target_coverage": 0.75,
+ "coverage": 0.7526881720430108,
+ "n": 70,
+ "risk": 0.4857142857142857
+ },
+ {
+ "target_coverage": 0.9,
+ "coverage": 0.9032258064516129,
+ "n": 84,
+ "risk": 0.5357142857142857
+ },
+ {
+ "target_coverage": 1.0,
+ "coverage": 1.0,
+ "n": 93,
+ "risk": 0.5698924731182796
+ }
+ ]
+ }
+ },
+ "definitions": {
+ "macro_f1": "equal mean of task-local macro F1; all task options, zero for undefined F1",
+ "brier": "mean sum of squared errors over all valid classes (including both noul classes)",
+ "ece": "15 equal bins [i/15,(i+1)/15), last bin includes 1; confidence=max(p)",
+ "risk_ties": "descending max(p), then lexical sample ID; ceil target count; diagnostic only"
+ }
+ },
+ "fitted": {
+ "choice": {
+ "status": "SUCCESS",
+ "n": 64,
+ "temperature": 3.1355035588563744,
+ "boundary_hit": false,
+ "nll_t1": 1.21606373270805,
+ "nll_fitted": 0.771914452059509
+ },
+ "noul": {
+ "status": "SUCCESS",
+ "n": 64,
+ "temperature": 5.0,
+ "boundary_hit": true,
+ "nll_t1": 1.5203078336357765,
+ "nll_fitted": 0.7411119749553794
+ }
+ },
+ "counts": {
+ "calibration_decisions": 128,
+ "calibration_groups": 62,
+ "test_decisions": 186,
+ "test_groups": 89,
+ "test_source_utterances": 93
+ },
+ "paired": {
+ "E1-E0": {
+ "status": "SUCCESS",
+ "n_groups": 89,
+ "repeats": 1000,
+ "seed": 20260924,
+ "after_minus_before": {
+ "accuracy": {
+ "delta": 0.0,
+ "ci95": [
+ 0.0,
+ 0.0
+ ]
+ },
+ "macro_f1": {
+ "delta": 0.0,
+ "ci95": [
+ 0.0,
+ 0.0
+ ]
+ },
+ "nll": {
+ "delta": -0.5987131301650952,
+ "ci95": [
+ -0.8657735567616283,
+ -0.3690991941162156
+ ]
+ },
+ "brier": {
+ "delta": -0.22317295342090193,
+ "ci95": [
+ -0.2988262606403422,
+ -0.15288959035634678
+ ]
+ },
+ "ece": {
+ "delta": -0.16895847079397686,
+ "ci95": [
+ -0.22499543687347137,
+ -0.09854721867871374
+ ]
+ }
+ }
+ }
+ }
+ },
+ "records": [
+ {"id":"massive-1003-choice","group_id":"massive-alarm-9c31991bfab87db64771","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"e6fa805478662b9dbcc6c13d66010516a205998b1e55b8fa88718b9a5ca94f46","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.212890625,-3.9296875,1.6923828125],"temperature":1.0,"type":"choice","source_id":"1003","source_row_hash":"c139b76204b21a35e4e183738cdcebd381ba62ffd54b666f3265e945432e67a4","input_tokens":52,"truncated":false},
+ {"id":"massive-1003-noul","group_id":"massive-alarm-9c31991bfab87db64771","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"6f5566a71b373d478175fad317696441493eb97aed759b090d2a2c48545ee861","option_order":["false","true"],"gold":"false","raw_logits":[-1.9501953125,0.350830078125],"temperature":1.0,"type":"noul","source_id":"1003","source_row_hash":"c139b76204b21a35e4e183738cdcebd381ba62ffd54b666f3265e945432e67a4","input_tokens":47,"truncated":false},
+ {"id":"massive-1037-choice","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"bb5e6d977cbf55ef5e40bb73ea9baf10960578c334f3d536be4306644facb148","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.31640625,-1.30078125,-2.2265625],"temperature":1.0,"type":"choice","source_id":"1037","source_row_hash":"cc358f7ff6234a620435ded52f9c6458c9725655acb4e04af4631fd08e5a5204","input_tokens":54,"truncated":false},
+ {"id":"massive-1037-noul","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"3e21af49ed7e7d8892eab2ba9ab022e2e3588c8b688c2341ba43c4b28c8864f9","option_order":["false","true"],"gold":"false","raw_logits":[-2.087890625,0.63623046875],"temperature":1.0,"type":"noul","source_id":"1037","source_row_hash":"cc358f7ff6234a620435ded52f9c6458c9725655acb4e04af4631fd08e5a5204","input_tokens":49,"truncated":false},
+ {"id":"massive-1263-choice","group_id":"massive-alarm-63e247661f9a8287cd19","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"08d4385e1c618bb738acb9a5a596d85083a795736d25f99d24a4109c26bce6e1","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.66015625,1.666015625,-2.01171875],"temperature":1.0,"type":"choice","source_id":"1263","source_row_hash":"4c56c9a24f86097f01329c4a62df6b539d96202d275ca101ee455c86e1997ca1","input_tokens":51,"truncated":false},
+ {"id":"massive-1263-noul","group_id":"massive-alarm-63e247661f9a8287cd19","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"6754f8f7870ad226a15315003e5c8c2d81894182d894561530bf9df030f82f52","option_order":["false","true"],"gold":"false","raw_logits":[-1.7666015625,0.31396484375],"temperature":1.0,"type":"noul","source_id":"1263","source_row_hash":"4c56c9a24f86097f01329c4a62df6b539d96202d275ca101ee455c86e1997ca1","input_tokens":46,"truncated":false},
+ {"id":"massive-1436-choice","group_id":"massive-alarm-c583aafff75bc3841fef","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3424e9c88433013f65faaf5bd116ec0d597ba22d08776672379224d576f843f6","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[5.23828125,-2.333984375,-2.853515625],"temperature":1.0,"type":"choice","source_id":"1436","source_row_hash":"94eb7e687527f8f71b3feca63c7dbbd70e8a983b83278b2da3c6c151e6c788d7","input_tokens":52,"truncated":false},
+ {"id":"massive-1436-noul","group_id":"massive-alarm-c583aafff75bc3841fef","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"3a9863e2c88dffd907da9f6044347c9f61de6505d1a9dce9417f406054ad0b2e","option_order":["false","true"],"gold":"false","raw_logits":[-1.951171875,1.1201171875],"temperature":1.0,"type":"noul","source_id":"1436","source_row_hash":"94eb7e687527f8f71b3feca63c7dbbd70e8a983b83278b2da3c6c151e6c788d7","input_tokens":47,"truncated":false},
+ {"id":"massive-1604-choice","group_id":"massive-alarm-7aa04ea17df3cdddb3fd","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"c544d289a669972f9dd9ffc0abfbabbd6242e07c0777812794fa4cd915f376dd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.869140625,0.30810546875,-0.5908203125],"temperature":1.0,"type":"choice","source_id":"1604","source_row_hash":"11a0e6002c5740f2e54e2854cfc01f4b30c2d0d7e65eccfa97155aa39c654f4b","input_tokens":52,"truncated":false},
+ {"id":"massive-1604-noul","group_id":"massive-alarm-7aa04ea17df3cdddb3fd","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"07d97cb090365c59b005f4f668e2e3fff1c297a02363218def7e349191debaf7","option_order":["false","true"],"gold":"false","raw_logits":[-1.8564453125,0.95654296875],"temperature":1.0,"type":"noul","source_id":"1604","source_row_hash":"11a0e6002c5740f2e54e2854cfc01f4b30c2d0d7e65eccfa97155aa39c654f4b","input_tokens":47,"truncated":false},
+ {"id":"massive-1657-choice","group_id":"massive-alarm-8bd33166b8b9b2f13b07","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d7486a4cb8953ccc3bb3ddd9179299768df6d34c1e2e880e766e75efedfd7be5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.91015625,-1.087890625,-3.57421875],"temperature":1.0,"type":"choice","source_id":"1657","source_row_hash":"ef2da490978458ee2ef0ee0a0d3b0133c56588c1e26696bccf3e2aa4c8259058","input_tokens":53,"truncated":false},
+ {"id":"massive-1657-noul","group_id":"massive-alarm-8bd33166b8b9b2f13b07","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"a560e880161b61f771ac3a6338ff7e841df824be6b274d90320b5785ce6d72b3","option_order":["false","true"],"gold":"false","raw_logits":[-1.30859375,1.0390625],"temperature":1.0,"type":"noul","source_id":"1657","source_row_hash":"ef2da490978458ee2ef0ee0a0d3b0133c56588c1e26696bccf3e2aa4c8259058","input_tokens":48,"truncated":false},
+ {"id":"massive-1733-choice","group_id":"massive-alarm-57e77352e57177a49115","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a7c5c35f649bc0b1446ed1c39de844fbbe2bfedb8762971db11bff47397c568d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.76171875,-1.87890625,-0.54833984375],"temperature":1.0,"type":"choice","source_id":"1733","source_row_hash":"a1f5a6d541d3e0f93903c8eb255a58ef9734fe6e55a7407e4015d4ebe61f872a","input_tokens":50,"truncated":false},
+ {"id":"massive-1733-noul","group_id":"massive-alarm-57e77352e57177a49115","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"444a0c395aed016e36c3ee3c22e2f4668c4d0012de718db84987c75b48deba4a","option_order":["false","true"],"gold":"false","raw_logits":[-0.98388671875,0.47119140625],"temperature":1.0,"type":"noul","source_id":"1733","source_row_hash":"a1f5a6d541d3e0f93903c8eb255a58ef9734fe6e55a7407e4015d4ebe61f872a","input_tokens":45,"truncated":false},
+ {"id":"massive-233-choice","group_id":"massive-alarm-a76f37cef918fa476dce","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1edd77d48a208990431178b3f9ff0e81bba90fc424f207edfe4901049150c5c6","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.35546875,-3.86328125,-3.658203125],"temperature":1.0,"type":"choice","source_id":"233","source_row_hash":"d4a1830096b5b362cb2c8979d9bee17ce35e11662498301fcf7d424aba32f005","input_tokens":53,"truncated":false},
+ {"id":"massive-233-noul","group_id":"massive-alarm-a76f37cef918fa476dce","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"bbac79120c3ff2705b203e60febc97ce48a4593d3ed949b4e48d407694ac163a","option_order":["false","true"],"gold":"true","raw_logits":[-2.30078125,1.5107421875],"temperature":1.0,"type":"noul","source_id":"233","source_row_hash":"d4a1830096b5b362cb2c8979d9bee17ce35e11662498301fcf7d424aba32f005","input_tokens":48,"truncated":false},
+ {"id":"massive-2335-choice","group_id":"massive-alarm-2e28becb2c156092a49a","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1b380ecf765e790d39243a04e05a6a1a4e89dc1dcd3153c1351181a4f7f04dea","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.982421875,-4.1015625,-3.669921875],"temperature":1.0,"type":"choice","source_id":"2335","source_row_hash":"714a2094c1a038fba50aa33f971e26771226ce7e9e04ebb9d2c7efcec38d0310","input_tokens":56,"truncated":false},
+ {"id":"massive-2335-noul","group_id":"massive-alarm-2e28becb2c156092a49a","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"2ab3d560c5bc5dafe905f62e711b23fa7294c813fbb83752e5d86531b0739ce4","option_order":["false","true"],"gold":"true","raw_logits":[-2.322265625,1.3525390625],"temperature":1.0,"type":"noul","source_id":"2335","source_row_hash":"714a2094c1a038fba50aa33f971e26771226ce7e9e04ebb9d2c7efcec38d0310","input_tokens":51,"truncated":false},
+ {"id":"massive-238-choice","group_id":"massive-alarm-d0053508597b2d9fa47e","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d7642d01f72c4bf4414827571c80447d99b745af71a1fbb58350a576c27de378","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.7109375,-1.130859375,-1.6572265625],"temperature":1.0,"type":"choice","source_id":"238","source_row_hash":"b05a2c56edee6e6a1f6ad98e6e4db9b0ac80e73346236fb818961f2a8aad75eb","input_tokens":50,"truncated":false},
+ {"id":"massive-238-noul","group_id":"massive-alarm-d0053508597b2d9fa47e","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"8901ba86edfac63a84581592485d396272294c4567e25620eecaf2ab26a0efeb","option_order":["false","true"],"gold":"false","raw_logits":[-1.6650390625,1.33203125],"temperature":1.0,"type":"noul","source_id":"238","source_row_hash":"b05a2c56edee6e6a1f6ad98e6e4db9b0ac80e73346236fb818961f2a8aad75eb","input_tokens":45,"truncated":false},
+ {"id":"massive-2872-choice","group_id":"massive-alarm-28ce6af777b18d848cc4","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3266abead6625894002caf1ffbd1b89d1302d2cfda49d08401fd6188c04d450e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[5.796875,-4.578125,-4.578125],"temperature":1.0,"type":"choice","source_id":"2872","source_row_hash":"4702426e7fb1d8fa48b86077ff3fb58fa8999c52a880e860661019a7aff3a2ff","input_tokens":50,"truncated":false},
+ {"id":"massive-2872-noul","group_id":"massive-alarm-28ce6af777b18d848cc4","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"a90d58891fd2025284bd565a1f8fa39b735090a012bd696054f66d8d81022dfb","option_order":["false","true"],"gold":"false","raw_logits":[-1.7099609375,1.833984375],"temperature":1.0,"type":"noul","source_id":"2872","source_row_hash":"4702426e7fb1d8fa48b86077ff3fb58fa8999c52a880e860661019a7aff3a2ff","input_tokens":45,"truncated":false},
+ {"id":"massive-3150-choice","group_id":"massive-alarm-97d6527cc37656846b02","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"090b673551073f1ebfdf857a5bb68d8e62b08acfc82426883ea71e1709f3797e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.951171875,-2.08203125,-0.1639404296875],"temperature":1.0,"type":"choice","source_id":"3150","source_row_hash":"4d419b74038f055d4737cbf88b4723ee2a78b9ce112e8bf29e34e37d457b8022","input_tokens":53,"truncated":false},
+ {"id":"massive-3150-noul","group_id":"massive-alarm-97d6527cc37656846b02","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c90f89f70a86da01043d5a5439c184efa63669f3f9a36adace7492118154db62","option_order":["false","true"],"gold":"false","raw_logits":[-2.302734375,1.0859375],"temperature":1.0,"type":"noul","source_id":"3150","source_row_hash":"4d419b74038f055d4737cbf88b4723ee2a78b9ce112e8bf29e34e37d457b8022","input_tokens":48,"truncated":false},
+ {"id":"massive-3155-choice","group_id":"massive-alarm-1a05a193b7245896522f","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"bff7c384e9000bf1ca3d85672a1230b2e6b175b679b063b0be917adc3a0b6ae9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.4921875,-2.19140625,-2.087890625],"temperature":1.0,"type":"choice","source_id":"3155","source_row_hash":"ad03657b56afbfa502e69cf2bd0e29a98c83996c41222f4931b15bd39ae5c116","input_tokens":61,"truncated":false},
+ {"id":"massive-3155-noul","group_id":"massive-alarm-1a05a193b7245896522f","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"761825a90cafa87fe6dc6a599248c59f4ac2296feb91e87a0b7ba56befc4453a","option_order":["false","true"],"gold":"true","raw_logits":[-2.064453125,1.5185546875],"temperature":1.0,"type":"noul","source_id":"3155","source_row_hash":"ad03657b56afbfa502e69cf2bd0e29a98c83996c41222f4931b15bd39ae5c116","input_tokens":56,"truncated":false},
+ {"id":"massive-3180-choice","group_id":"massive-alarm-e4f7f3a520230de4d198","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1be584752d9da34d19148ed46c921d36f618e808f753b1b66487bab510798a0c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.625,-2.34375,-1.4287109375],"temperature":1.0,"type":"choice","source_id":"3180","source_row_hash":"48769eebc7610c81b81ad51c5266ea0166abeb999712ec3304969fda5c1136c6","input_tokens":55,"truncated":false},
+ {"id":"massive-3180-noul","group_id":"massive-alarm-e4f7f3a520230de4d198","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"74856b6b33b3577a5dd45f8b2bf787504276b37df6b707eeffe9fae3efca17d7","option_order":["false","true"],"gold":"true","raw_logits":[-0.70849609375,0.75732421875],"temperature":1.0,"type":"noul","source_id":"3180","source_row_hash":"48769eebc7610c81b81ad51c5266ea0166abeb999712ec3304969fda5c1136c6","input_tokens":50,"truncated":false},
+ {"id":"massive-3181-choice","group_id":"massive-alarm-513568a33433630f98d8","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"830d07369c5b2e5c39ee664f2fdcbf69533affad202f7d9a653a5494cf4b489e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.8046875,-4.19140625,-3.318359375],"temperature":1.0,"type":"choice","source_id":"3181","source_row_hash":"ac7f0b47957f7510837d1547341e088f0a4d09ed3a9701a61b95a7854d695e0e","input_tokens":55,"truncated":false},
+ {"id":"massive-3181-noul","group_id":"massive-alarm-513568a33433630f98d8","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"3defc8360fec5e2ca53cce81386406fafd3a3537186fb44d3bd1e8dc5f2d474b","option_order":["false","true"],"gold":"true","raw_logits":[-1.85546875,1.07421875],"temperature":1.0,"type":"noul","source_id":"3181","source_row_hash":"ac7f0b47957f7510837d1547341e088f0a4d09ed3a9701a61b95a7854d695e0e","input_tokens":50,"truncated":false},
+ {"id":"massive-3241-choice","group_id":"massive-alarm-b22b203130c5192a6423","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"f524b17f665cda002061c754e3fd41fd0e740c3018b24ba2d98ae591ceedb7c4","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.1953125,-3.646484375,-3.552734375],"temperature":1.0,"type":"choice","source_id":"3241","source_row_hash":"16f6792d8e8ccd3989e04e04b8148d3be61199c7e43d402bc6eeb08cbe3fd917","input_tokens":53,"truncated":false},
+ {"id":"massive-3241-noul","group_id":"massive-alarm-b22b203130c5192a6423","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"fc6c5b8d2f44d273695f46d9a43719cadeb39c36eacba4524af59eb3c7f4767b","option_order":["false","true"],"gold":"true","raw_logits":[-2.30078125,1.1953125],"temperature":1.0,"type":"noul","source_id":"3241","source_row_hash":"16f6792d8e8ccd3989e04e04b8148d3be61199c7e43d402bc6eeb08cbe3fd917","input_tokens":48,"truncated":false},
+ {"id":"massive-3271-choice","group_id":"massive-alarm-cc4399c7e1fe9bedaecc","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"b2fb2fd52dd7d6293ffb2ce75dfa9d145afb3da928c3f35cb90089a75a41290c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.455078125,-4.16015625,1.923828125],"temperature":1.0,"type":"choice","source_id":"3271","source_row_hash":"3d4b1654d3459ec3a62cc33209806f071eb0d809cc71d0fb51be19014a8b13d9","input_tokens":51,"truncated":false},
+ {"id":"massive-3271-noul","group_id":"massive-alarm-cc4399c7e1fe9bedaecc","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"304b596915cb7afe22d5674667661a63184a832a5d2bd367bc730d199d2cd0ba","option_order":["false","true"],"gold":"false","raw_logits":[-2.5078125,-0.36181640625],"temperature":1.0,"type":"noul","source_id":"3271","source_row_hash":"3d4b1654d3459ec3a62cc33209806f071eb0d809cc71d0fb51be19014a8b13d9","input_tokens":46,"truncated":false},
+ {"id":"massive-3313-choice","group_id":"massive-alarm-9624abc57fb5f9491c3c","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a63a16cad37a46fa428b72311b35fdecf3557e049597fe79ffbc0e23619e92ac","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.78125,-4.37890625,-4.58203125],"temperature":1.0,"type":"choice","source_id":"3313","source_row_hash":"25d5d82e30b677564c58a29305e0946447be07c0985598b902c051ef1fb1ea80","input_tokens":60,"truncated":false},
+ {"id":"massive-3313-noul","group_id":"massive-alarm-9624abc57fb5f9491c3c","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"62a66e214884f42fa03ddc5d6e062bb3657c1418478a8e7b625c87853a0ee453","option_order":["false","true"],"gold":"true","raw_logits":[-2.55078125,1.4404296875],"temperature":1.0,"type":"noul","source_id":"3313","source_row_hash":"25d5d82e30b677564c58a29305e0946447be07c0985598b902c051ef1fb1ea80","input_tokens":55,"truncated":false},
+ {"id":"massive-3344-choice","group_id":"massive-alarm-2124aad0708d0ec9337e","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"9748505b1055a8e4b9b580e4f065f1fb025a351d0316f2664fe9fb467aa20b01","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.60205078125,-1.8876953125,-0.740234375],"temperature":1.0,"type":"choice","source_id":"3344","source_row_hash":"503f2378950bb6b7542b8fd34d1781fa9a81f3da8e533bc3e5474ee2e90182a6","input_tokens":51,"truncated":false},
+ {"id":"massive-3344-noul","group_id":"massive-alarm-2124aad0708d0ec9337e","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"40547d94b9391018d836f7c33cb83a9888d4a349795a9961a9c4d63bd0f9b699","option_order":["false","true"],"gold":"false","raw_logits":[-1.6083984375,0.9453125],"temperature":1.0,"type":"noul","source_id":"3344","source_row_hash":"503f2378950bb6b7542b8fd34d1781fa9a81f3da8e533bc3e5474ee2e90182a6","input_tokens":46,"truncated":false},
+ {"id":"massive-3461-choice","group_id":"massive-alarm-ce0cb4819c1987861556","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"73431ea9804730e590ddc48be8d2c8035bb0dbeb45e6c1cfa2b74cddf9289334","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.64453125,-3.767578125,-3.158203125],"temperature":1.0,"type":"choice","source_id":"3461","source_row_hash":"0e09b286b7b15ee40d39a9dc76bc5d58004ddbae3898a25e3673897d633f6fd7","input_tokens":54,"truncated":false},
+ {"id":"massive-3461-noul","group_id":"massive-alarm-ce0cb4819c1987861556","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"9c083d9ba471c24cd29e87a06ee702b6f461f167c9e5f0040a36737925acfc5b","option_order":["false","true"],"gold":"true","raw_logits":[-2.619140625,1.3740234375],"temperature":1.0,"type":"noul","source_id":"3461","source_row_hash":"0e09b286b7b15ee40d39a9dc76bc5d58004ddbae3898a25e3673897d633f6fd7","input_tokens":49,"truncated":false},
+ {"id":"massive-3537-choice","group_id":"massive-alarm-1b97665270a474c9189b","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"58b2a286faf22ace00cb5b5322c9207f9304330fc3153fe79b6e308492cccfb8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.4990234375,-3.5234375,2.623046875],"temperature":1.0,"type":"choice","source_id":"3537","source_row_hash":"3bb0e1d5d6780212fc57f7c5bdc635e4f3b8af60e8f05a8864c0e0801b159e28","input_tokens":50,"truncated":false},
+ {"id":"massive-3537-noul","group_id":"massive-alarm-1b97665270a474c9189b","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"27a1f6fa962a3cd311f0a1ded2227f879f7d3f63afd038278229abc437872012","option_order":["false","true"],"gold":"false","raw_logits":[-2.248046875,0.375244140625],"temperature":1.0,"type":"noul","source_id":"3537","source_row_hash":"3bb0e1d5d6780212fc57f7c5bdc635e4f3b8af60e8f05a8864c0e0801b159e28","input_tokens":45,"truncated":false},
+ {"id":"massive-3696-choice","group_id":"massive-alarm-66bb02dcd4dfec9a31da","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"5c799de6dc9b83c02640969e26d2a2b4abd09017c1da008663ae386f27a420dc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.1328125,-3.171875,0.63134765625],"temperature":1.0,"type":"choice","source_id":"3696","source_row_hash":"50b44cd61dd0112bec73547e36d51b18ec64f7f88e01642a88fe65e66aaf8019","input_tokens":57,"truncated":false},
+ {"id":"massive-3696-noul","group_id":"massive-alarm-66bb02dcd4dfec9a31da","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"83b6f74e7c01aed19a8a603c7b23d4c58da8a0282bc12b481546ed07d0dae1ca","option_order":["false","true"],"gold":"true","raw_logits":[-2.15625,1.0517578125],"temperature":1.0,"type":"noul","source_id":"3696","source_row_hash":"50b44cd61dd0112bec73547e36d51b18ec64f7f88e01642a88fe65e66aaf8019","input_tokens":52,"truncated":false},
+ {"id":"massive-3760-choice","group_id":"massive-alarm-7f77ea61d28ffa0c4760","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a226efece8e4ca552b5c604d2c0e77f1e7ad8ef574882ccdf2abc1353914a530","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.5634765625,-1.9404296875,1.23046875],"temperature":1.0,"type":"choice","source_id":"3760","source_row_hash":"c58ac469fe7402eeee9373c32aed43cde55b7f6a576c78cf3e6cdbdcad81b62f","input_tokens":54,"truncated":false},
+ {"id":"massive-3760-noul","group_id":"massive-alarm-7f77ea61d28ffa0c4760","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"a17c30741ef64b9a54cd5fa80638f0d977ace713acc174493612ac98af69da35","option_order":["false","true"],"gold":"true","raw_logits":[-2.052734375,1.4296875],"temperature":1.0,"type":"noul","source_id":"3760","source_row_hash":"c58ac469fe7402eeee9373c32aed43cde55b7f6a576c78cf3e6cdbdcad81b62f","input_tokens":49,"truncated":false},
+ {"id":"massive-3948-choice","group_id":"massive-alarm-74196bc28bd235b5ebd4","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2244c648b7fdc8555facdb170e46c5ab59d024a693910e18c7b28b94ac0edb53","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.94140625,-4.47265625,-4.48828125],"temperature":1.0,"type":"choice","source_id":"3948","source_row_hash":"efdef6a7ca0afe5619c4c98377ee543a2c63c2f4a10c19d98cf9308324489779","input_tokens":52,"truncated":false},
+ {"id":"massive-3948-noul","group_id":"massive-alarm-74196bc28bd235b5ebd4","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"72dd631c4fbc49146d876163d1ab74c4cb6eeb7076498c11a5a138878ca345bd","option_order":["false","true"],"gold":"true","raw_logits":[-2.017578125,1.310546875],"temperature":1.0,"type":"noul","source_id":"3948","source_row_hash":"efdef6a7ca0afe5619c4c98377ee543a2c63c2f4a10c19d98cf9308324489779","input_tokens":47,"truncated":false},
+ {"id":"massive-3959-choice","group_id":"massive-alarm-1e2d1e22e2a448d0d4be","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3d05bb7983f196a0d3e77f5d1eda7f2de7d1eb9f65c25e6a5aa4985decca2ffc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.8291015625,-4.109375,-0.41552734375],"temperature":1.0,"type":"choice","source_id":"3959","source_row_hash":"7539f118e45280ccaa989666ca00e8b1765628ce7a944ffa526b60c3c58b91ab","input_tokens":52,"truncated":false},
+ {"id":"massive-3959-noul","group_id":"massive-alarm-1e2d1e22e2a448d0d4be","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"0ca03992c0d8713a6590531ef8364a455a9ee6f7d7116d6a6262e593a496f42e","option_order":["false","true"],"gold":"true","raw_logits":[-1.388671875,1.0908203125],"temperature":1.0,"type":"noul","source_id":"3959","source_row_hash":"7539f118e45280ccaa989666ca00e8b1765628ce7a944ffa526b60c3c58b91ab","input_tokens":47,"truncated":false},
+ {"id":"massive-3975-choice","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"e9c4661dc32b87b3f0762cd7a7254309cd7852b13a77514c93a4e4003111fe2f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.0711669921875,-2.5546875,-0.60888671875],"temperature":1.0,"type":"choice","source_id":"3975","source_row_hash":"c8c3d4b20f0116b687d7d684dba6fe8070f18bad470dc44ddbb7000c8c4ebaee","input_tokens":49,"truncated":false},
+ {"id":"massive-3975-noul","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"df1f03cd870145a75b4e1403f5ec6464d4217ddf98f0e60c79cebc6dcf18f34c","option_order":["false","true"],"gold":"true","raw_logits":[-2.453125,1.7314453125],"temperature":1.0,"type":"noul","source_id":"3975","source_row_hash":"c8c3d4b20f0116b687d7d684dba6fe8070f18bad470dc44ddbb7000c8c4ebaee","input_tokens":44,"truncated":false},
+ {"id":"massive-3985-choice","group_id":"massive-alarm-0eab4c88f4262e142df9","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a0ae587eb80ef7ddb6b9eb42df39de2690cd1df443b346b429870aa844881111","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.611328125,-4.12890625,-4.41015625],"temperature":1.0,"type":"choice","source_id":"3985","source_row_hash":"405084aa291615ce23004c5c83303d8418be167a30ba30ed939eb50f7b2e6b87","input_tokens":53,"truncated":false},
+ {"id":"massive-3985-noul","group_id":"massive-alarm-0eab4c88f4262e142df9","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"ea1cfc0c358999cca571b97cb637ab166d0ca3225c812cf9c03f24cc95d8d626","option_order":["false","true"],"gold":"true","raw_logits":[-2.0390625,1.4208984375],"temperature":1.0,"type":"noul","source_id":"3985","source_row_hash":"405084aa291615ce23004c5c83303d8418be167a30ba30ed939eb50f7b2e6b87","input_tokens":48,"truncated":false},
+ {"id":"massive-4016-choice","group_id":"massive-alarm-4be456711aa1993110b2","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"505ee765de12c09f3f828b85ca69cd4953e2191d4d8c4ee6a25189a83a760043","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.306640625,-4.37890625,4.83203125],"temperature":1.0,"type":"choice","source_id":"4016","source_row_hash":"eb5665c8fa5ae7c7dc5804205acfba089cd6c2936b540d0c86da0c3ee73ec92c","input_tokens":53,"truncated":false},
+ {"id":"massive-4016-noul","group_id":"massive-alarm-4be456711aa1993110b2","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"d140649f39079e246dadf8169a7a97f6b2489328b4734f0c178f3de16660dba5","option_order":["false","true"],"gold":"false","raw_logits":[-1.740234375,0.7607421875],"temperature":1.0,"type":"noul","source_id":"4016","source_row_hash":"eb5665c8fa5ae7c7dc5804205acfba089cd6c2936b540d0c86da0c3ee73ec92c","input_tokens":48,"truncated":false},
+ {"id":"massive-4108-choice","group_id":"massive-alarm-ba768fc0bd4d5e4b7d89","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"98479093baf492b301adaa3cc8ba52c0481115d2eca82b13c0b146b35bb9a417","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.076171875,1.240234375,-1.5771484375],"temperature":1.0,"type":"choice","source_id":"4108","source_row_hash":"442cf4064902074bab424f8a13b5da971239324a1ddb14ac9c1461459605890d","input_tokens":51,"truncated":false},
+ {"id":"massive-4108-noul","group_id":"massive-alarm-ba768fc0bd4d5e4b7d89","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"85af1497981b267f36416626603b0919ffe18af189c37a8b46e7da2c551c5255","option_order":["false","true"],"gold":"false","raw_logits":[-1.4697265625,1.169921875],"temperature":1.0,"type":"noul","source_id":"4108","source_row_hash":"442cf4064902074bab424f8a13b5da971239324a1ddb14ac9c1461459605890d","input_tokens":46,"truncated":false},
+ {"id":"massive-4122-choice","group_id":"massive-alarm-7b84a2782f58c91b2bae","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"acc9bcb43f252751dd15370f7a1f9f35ad22a6b1c088fb3e9b36ede8d965b270","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.61328125,-0.46728515625,-1.58203125],"temperature":1.0,"type":"choice","source_id":"4122","source_row_hash":"844fa5f915442e44ef3076cd0ce2150f380ac58e2b067d29b96f08c100a1d939","input_tokens":50,"truncated":false},
+ {"id":"massive-4122-noul","group_id":"massive-alarm-7b84a2782f58c91b2bae","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"907aa9c5faa26b58d904e64f202bcce86d6c4f242c23186d9ee265ae4166ee9d","option_order":["false","true"],"gold":"false","raw_logits":[-1.4423828125,0.82763671875],"temperature":1.0,"type":"noul","source_id":"4122","source_row_hash":"844fa5f915442e44ef3076cd0ce2150f380ac58e2b067d29b96f08c100a1d939","input_tokens":45,"truncated":false},
+ {"id":"massive-4144-choice","group_id":"massive-alarm-95d77d103fd5fab659a2","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3f8abfe30982bff102e8b189574ee14bbf601371a4c033c0ebc000c980e801f7","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[1.3994140625,-2.697265625,-0.350830078125],"temperature":1.0,"type":"choice","source_id":"4144","source_row_hash":"0fd0739e6946a24cc24d8339f712773ba73cdcb18125f66f4b1af0f6c56cb409","input_tokens":54,"truncated":false},
+ {"id":"massive-4144-noul","group_id":"massive-alarm-95d77d103fd5fab659a2","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"bb33b9b4e1da7c9029b695525c094b5019ad962957ae1e33e14fe4d50ffc5925","option_order":["false","true"],"gold":"false","raw_logits":[-1.7705078125,0.91015625],"temperature":1.0,"type":"noul","source_id":"4144","source_row_hash":"0fd0739e6946a24cc24d8339f712773ba73cdcb18125f66f4b1af0f6c56cb409","input_tokens":49,"truncated":false},
+ {"id":"massive-4200-choice","group_id":"massive-alarm-a56e43f47ec9a9386d07","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"6774a6320d42c290b80ddb390be36ac859ee6f15374b0bd6ff3fefd461823904","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.27734375,-4.15234375,1.6220703125],"temperature":1.0,"type":"choice","source_id":"4200","source_row_hash":"c355f735101213734ce53521f480d2713ee60fde1860e82542bfa9d357596c71","input_tokens":52,"truncated":false},
+ {"id":"massive-4200-noul","group_id":"massive-alarm-a56e43f47ec9a9386d07","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c2d79b9b8cf8fe2e19bf991be938ccd39a5638a21c2c00469f612d6d93b32d30","option_order":["false","true"],"gold":"false","raw_logits":[-3.84375,0.4052734375],"temperature":1.0,"type":"noul","source_id":"4200","source_row_hash":"c355f735101213734ce53521f480d2713ee60fde1860e82542bfa9d357596c71","input_tokens":47,"truncated":false},
+ {"id":"massive-4509-choice","group_id":"massive-alarm-d4f2a8b9df951f72e77a","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"31ad31c567248f3781353d8e1cc63b681d0d3e9c5bdda2d3fab14afc0a04b41a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.2138671875,-3.05078125,1.6796875],"temperature":1.0,"type":"choice","source_id":"4509","source_row_hash":"bf224c141975e01266dcf5ab02b2f22e5e961fe3acb83f2ff99c3f8f960284b2","input_tokens":51,"truncated":false},
+ {"id":"massive-4509-noul","group_id":"massive-alarm-d4f2a8b9df951f72e77a","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"dbe036ea36070f090a10050a3982caa2a9e3a00a5640b7586cef8afc7276731c","option_order":["false","true"],"gold":"false","raw_logits":[-2.4609375,1.2763671875],"temperature":1.0,"type":"noul","source_id":"4509","source_row_hash":"bf224c141975e01266dcf5ab02b2f22e5e961fe3acb83f2ff99c3f8f960284b2","input_tokens":46,"truncated":false},
+ {"id":"massive-4530-choice","group_id":"massive-alarm-c0492ef78107c7e23f86","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"4bcb70abe01980ab58b9e301d26da030dae938a5e9ae12f907c932fb5ffa09a8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.2783203125,-0.74560546875,-0.94482421875],"temperature":1.0,"type":"choice","source_id":"4530","source_row_hash":"3dc1bd3fdc52e2c231408ad4a522b21e4518cff1813ef183884a10d582345cee","input_tokens":55,"truncated":false},
+ {"id":"massive-4530-noul","group_id":"massive-alarm-c0492ef78107c7e23f86","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"15ca7806e7ca42d6ce4a143c2712cc799fa9ccd81611e366e35693d2927483cb","option_order":["false","true"],"gold":"false","raw_logits":[-1.53125,0.697265625],"temperature":1.0,"type":"noul","source_id":"4530","source_row_hash":"3dc1bd3fdc52e2c231408ad4a522b21e4518cff1813ef183884a10d582345cee","input_tokens":50,"truncated":false},
+ {"id":"massive-4585-choice","group_id":"massive-alarm-57ee6f9659a3c6835215","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"8e4c5a399d92bc0fee79a0b7c274ef67eb3dadc5c492cccc34f148aa6184c53d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.625,-3.06640625,-2.103515625],"temperature":1.0,"type":"choice","source_id":"4585","source_row_hash":"29cb3192dcfaf8dd3c91f8560e6956e6d62314b02d2c3a19b339b44e7f94808e","input_tokens":61,"truncated":false},
+ {"id":"massive-4585-noul","group_id":"massive-alarm-57ee6f9659a3c6835215","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"5d41960f1abd888d11d48ec0e983393f7efd4ffdad5f9941815404c8ba47ba75","option_order":["false","true"],"gold":"false","raw_logits":[-1.8154296875,0.28125],"temperature":1.0,"type":"noul","source_id":"4585","source_row_hash":"29cb3192dcfaf8dd3c91f8560e6956e6d62314b02d2c3a19b339b44e7f94808e","input_tokens":56,"truncated":false},
+ {"id":"massive-4593-choice","group_id":"massive-alarm-70c776b38b2c5f805dcd","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2b329bec09cbc6dbd489dba7bf5ba03c208c7c760c3b11dcae5beab03e9de1d9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.73828125,1.751953125,-0.728515625],"temperature":1.0,"type":"choice","source_id":"4593","source_row_hash":"573a25fd5fa0ccdcbdf4ebee81ca2c4bbaab53ba3cb1563b483549e39c6e6a7a","input_tokens":53,"truncated":false},
+ {"id":"massive-4593-noul","group_id":"massive-alarm-70c776b38b2c5f805dcd","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c9d1252c93abe919faf36506ceee2a4d572863bd74e43eb30c6e82ad87f3f29e","option_order":["false","true"],"gold":"false","raw_logits":[-1.8310546875,0.95751953125],"temperature":1.0,"type":"noul","source_id":"4593","source_row_hash":"573a25fd5fa0ccdcbdf4ebee81ca2c4bbaab53ba3cb1563b483549e39c6e6a7a","input_tokens":48,"truncated":false},
+ {"id":"massive-4744-choice","group_id":"massive-alarm-4328c171395268c31624","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d0d46e21b89af922fa963c14d2cecf2e2b20b70099999c4ad40900017de92075","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.828125,3.25,0.44384765625],"temperature":1.0,"type":"choice","source_id":"4744","source_row_hash":"a4566885577d6aab551b8f32aa9432224700f661cf64bebd0176e38f03274c83","input_tokens":54,"truncated":false},
+ {"id":"massive-4744-noul","group_id":"massive-alarm-4328c171395268c31624","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"afae574d52cbd08a229a7819423ec0cbd438a7d7268bf06023f2f5e18648c7b9","option_order":["false","true"],"gold":"false","raw_logits":[-1.40234375,0.57763671875],"temperature":1.0,"type":"noul","source_id":"4744","source_row_hash":"a4566885577d6aab551b8f32aa9432224700f661cf64bebd0176e38f03274c83","input_tokens":49,"truncated":false},
+ {"id":"massive-4929-choice","group_id":"massive-alarm-74a1feb1e119db1fb400","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"bc3b693c29ce78f19435be8dd99e9451443befc07587185de8be06a0574902de","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.08660888671875,-2.994140625,1.0556640625],"temperature":1.0,"type":"choice","source_id":"4929","source_row_hash":"53f4b87a6a89da6de1f9302d72950cd98c6c9ba1bc9d35a3b33b0a2e2c4769a1","input_tokens":53,"truncated":false},
+ {"id":"massive-4929-noul","group_id":"massive-alarm-74a1feb1e119db1fb400","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"59f48bdeda5d59c536e5c4364cba1a673b4afed9d9b79f6d8eab2d3e95c2e25f","option_order":["false","true"],"gold":"true","raw_logits":[-2.052734375,1.2275390625],"temperature":1.0,"type":"noul","source_id":"4929","source_row_hash":"53f4b87a6a89da6de1f9302d72950cd98c6c9ba1bc9d35a3b33b0a2e2c4769a1","input_tokens":48,"truncated":false},
+ {"id":"massive-5114-choice","group_id":"massive-alarm-7fdbaa1c48c1e3afefe1","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"43567ef7d5ff87f4c3710c3424ab5f2cb19756e69c4d0e44ed470313ce41347b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.53076171875,-0.1943359375,-0.9033203125],"temperature":1.0,"type":"choice","source_id":"5114","source_row_hash":"0429eec68e26dfecd519ba81f0c92d85793aad1c8d202737bb439666a411b485","input_tokens":54,"truncated":false},
+ {"id":"massive-5114-noul","group_id":"massive-alarm-7fdbaa1c48c1e3afefe1","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"7e3f35cef593675bcafc126f93ea622d79fe105f82c4e8c9ab80d884ac2201e5","option_order":["false","true"],"gold":"false","raw_logits":[-1.6279296875,1.0166015625],"temperature":1.0,"type":"noul","source_id":"5114","source_row_hash":"0429eec68e26dfecd519ba81f0c92d85793aad1c8d202737bb439666a411b485","input_tokens":49,"truncated":false},
+ {"id":"massive-5172-choice","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"4f6d1a6703c39c88cebaff032394052408cba7c60cc58c80bd70f6bfd4112ba9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.008392333984375,-2.896484375,-0.76806640625],"temperature":1.0,"type":"choice","source_id":"5172","source_row_hash":"25e5a31c20bb4859195c396f45f721767e4d6c8bb6eceaaf2c63dcfc20adeb64","input_tokens":52,"truncated":false},
+ {"id":"massive-5172-noul","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"69e8607aaeebf1a1350d73f7ea71224817d665a684ac4f3cdfe8ec464ace63b8","option_order":["false","true"],"gold":"true","raw_logits":[-2.07421875,1.3017578125],"temperature":1.0,"type":"noul","source_id":"5172","source_row_hash":"25e5a31c20bb4859195c396f45f721767e4d6c8bb6eceaaf2c63dcfc20adeb64","input_tokens":47,"truncated":false},
+ {"id":"massive-5210-choice","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"c668b88dd8e85f1b4e78bb29f8f80248aa67aab044cfc54d576c102766632bfa","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.8671875,-3.4296875,-3.0703125],"temperature":1.0,"type":"choice","source_id":"5210","source_row_hash":"fa1ecb16b6a4385a45785dd19132a817f21fff52934ee5a626a97c780f4ab078","input_tokens":53,"truncated":false},
+ {"id":"massive-5210-noul","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"067fdd02a518f2dd87f537b24dabb02f1d8133c6bab8803ba201115d10f6d82d","option_order":["false","true"],"gold":"false","raw_logits":[-2.16015625,0.9296875],"temperature":1.0,"type":"noul","source_id":"5210","source_row_hash":"fa1ecb16b6a4385a45785dd19132a817f21fff52934ee5a626a97c780f4ab078","input_tokens":48,"truncated":false},
+ {"id":"massive-5319-choice","group_id":"massive-alarm-322a8df31b5ea91267e0","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"720d5cc7c24bb6954e0c41cc16a40fb2b5124e4bb102f2aed64b7acc5df81817","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.56640625,-1.7998046875,-1.109375],"temperature":1.0,"type":"choice","source_id":"5319","source_row_hash":"2d07404bc4a0e9de04cd7b8305b4d7fb4db95c545a145cc3f8fb6ab7b5934529","input_tokens":51,"truncated":false},
+ {"id":"massive-5319-noul","group_id":"massive-alarm-322a8df31b5ea91267e0","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"334afe5034e053d889e7e832d8fe0b2fd80448a8f579445964b27a15650a3933","option_order":["false","true"],"gold":"true","raw_logits":[-2.2421875,1.4140625],"temperature":1.0,"type":"noul","source_id":"5319","source_row_hash":"2d07404bc4a0e9de04cd7b8305b4d7fb4db95c545a145cc3f8fb6ab7b5934529","input_tokens":46,"truncated":false},
+ {"id":"massive-535-choice","group_id":"massive-alarm-669fb7803ab072ceb0b5","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"10b4de6d460909b2303c9dab7a9e033a29d50076d3e4bb27e4a936a36786ce62","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.300048828125,-1.794921875,-0.435546875],"temperature":1.0,"type":"choice","source_id":"535","source_row_hash":"0e3fcc2ba0c2c5c5f577632298fa42814922bfd1141206f7474fd190dc965c7e","input_tokens":54,"truncated":false},
+ {"id":"massive-535-noul","group_id":"massive-alarm-669fb7803ab072ceb0b5","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"99646e241c5c848f95da454dcbfd7698231492b08a38270a0c1651139e49f9d3","option_order":["false","true"],"gold":"true","raw_logits":[-0.666015625,0.058319091796875],"temperature":1.0,"type":"noul","source_id":"535","source_row_hash":"0e3fcc2ba0c2c5c5f577632298fa42814922bfd1141206f7474fd190dc965c7e","input_tokens":49,"truncated":false},
+ {"id":"massive-5492-choice","group_id":"massive-alarm-5791810e78edbebec8b4","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"55954676941878d2afda77659529a13bd38cd2f009e9eee4eb759261ff47ac73","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.234375,-3.1796875,1.62109375],"temperature":1.0,"type":"choice","source_id":"5492","source_row_hash":"6b09ef0b32c674bcb40f7ad0ca704a196efe0ea8ce77472e2e3088c05bd29ed3","input_tokens":53,"truncated":false},
+ {"id":"massive-5492-noul","group_id":"massive-alarm-5791810e78edbebec8b4","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c1cbc2786d808f88ca9a1d631a134993d9a3357acbad234c9de4ba4cff33ae20","option_order":["false","true"],"gold":"false","raw_logits":[-1.857421875,-0.35498046875],"temperature":1.0,"type":"noul","source_id":"5492","source_row_hash":"6b09ef0b32c674bcb40f7ad0ca704a196efe0ea8ce77472e2e3088c05bd29ed3","input_tokens":48,"truncated":false},
+ {"id":"massive-5673-choice","group_id":"massive-alarm-fa15e151bbd8376bf727","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"4ca5a1fef0f2d153d7bd6110a0fc74ceca4b41ad79287ee6961d656f74828ddc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.291015625,-3.931640625,2.373046875],"temperature":1.0,"type":"choice","source_id":"5673","source_row_hash":"42f49991de76cd94ab5ee5743f3e57ae2a14f2fb657869b455a94bfbbd05648f","input_tokens":52,"truncated":false},
+ {"id":"massive-5673-noul","group_id":"massive-alarm-fa15e151bbd8376bf727","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"d4cb011692189f194a6894f23e0488fa6be688c3166137780caf9244766206dc","option_order":["false","true"],"gold":"false","raw_logits":[-2.53125,0.496826171875],"temperature":1.0,"type":"noul","source_id":"5673","source_row_hash":"42f49991de76cd94ab5ee5743f3e57ae2a14f2fb657869b455a94bfbbd05648f","input_tokens":47,"truncated":false},
+ {"id":"massive-5675-choice","group_id":"massive-alarm-3568a7b610cf5fedeafe","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"c22de0ed7a39f40c2c13850693773a456275d5726a53d6e53dcd7355931f7c7f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.67578125,-4.43359375,1.3515625],"temperature":1.0,"type":"choice","source_id":"5675","source_row_hash":"d4455f4931eebb0ae5c429523260bb0da106c1fb1f59166ff83aa27192fd603d","input_tokens":53,"truncated":false},
+ {"id":"massive-5675-noul","group_id":"massive-alarm-3568a7b610cf5fedeafe","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"2e82b6e8cdeb672a76ee85b20814106355c89d5c9c981051cff180c0105ff9d3","option_order":["false","true"],"gold":"false","raw_logits":[-2.6015625,-0.00815582275390625],"temperature":1.0,"type":"noul","source_id":"5675","source_row_hash":"d4455f4931eebb0ae5c429523260bb0da106c1fb1f59166ff83aa27192fd603d","input_tokens":48,"truncated":false},
+ {"id":"massive-5736-choice","group_id":"massive-alarm-cad8645cf0d1993997ce","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d4a111329224fc89c0ce98f064feba839677a403249e06d1d2a8f49cb285191b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.5673828125,1.474609375,-0.52197265625],"temperature":1.0,"type":"choice","source_id":"5736","source_row_hash":"70f64d8c12527bf3143b24720e5e124b9c7c43de9bf0b78aca2fe33336e7f60d","input_tokens":54,"truncated":false},
+ {"id":"massive-5736-noul","group_id":"massive-alarm-cad8645cf0d1993997ce","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"cd7f7f33d54024ae047cc45e45135de22fddd92c2eff1bdc2cf57668f3ef0b9d","option_order":["false","true"],"gold":"false","raw_logits":[-0.8037109375,0.301025390625],"temperature":1.0,"type":"noul","source_id":"5736","source_row_hash":"70f64d8c12527bf3143b24720e5e124b9c7c43de9bf0b78aca2fe33336e7f60d","input_tokens":49,"truncated":false},
+ {"id":"massive-5891-choice","group_id":"massive-alarm-58fbb3df6930e3f46619","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"10f3ea8840e48ecbe378b6635f27574b818a01d405ff5f5435cda368088af7a3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.654296875,-1.7041015625,-3.41796875],"temperature":1.0,"type":"choice","source_id":"5891","source_row_hash":"13a21672cc8d63f41b4c4f3be3e9ee23d26cf4d483b68904d54849e76a9572ed","input_tokens":55,"truncated":false},
+ {"id":"massive-5891-noul","group_id":"massive-alarm-58fbb3df6930e3f46619","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"aebc66b1557724f2c8050b8b96f8bd79808521ed13b64bc0d421e5ddee9e2ed7","option_order":["false","true"],"gold":"false","raw_logits":[-2.353515625,0.875],"temperature":1.0,"type":"noul","source_id":"5891","source_row_hash":"13a21672cc8d63f41b4c4f3be3e9ee23d26cf4d483b68904d54849e76a9572ed","input_tokens":50,"truncated":false},
+ {"id":"massive-5906-choice","group_id":"massive-alarm-70d28c8cc29115db7bf6","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d429a04be92b35783c5220b12fb0d69dc0cc209556a6e30d4ccab1ab1b6957e6","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.2734375,-3.880859375,-4.16015625],"temperature":1.0,"type":"choice","source_id":"5906","source_row_hash":"c3492f95295cde551179df79a12fe2b92977fb44481de9b512aa59ed0900a40f","input_tokens":52,"truncated":false},
+ {"id":"massive-5906-noul","group_id":"massive-alarm-70d28c8cc29115db7bf6","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"73d6faa4a8c642c84be73a9055997aee319398417b64d904ff2cc240cdb18649","option_order":["false","true"],"gold":"true","raw_logits":[-2.685546875,1.5634765625],"temperature":1.0,"type":"noul","source_id":"5906","source_row_hash":"c3492f95295cde551179df79a12fe2b92977fb44481de9b512aa59ed0900a40f","input_tokens":47,"truncated":false},
+ {"id":"massive-5981-choice","group_id":"massive-alarm-3c4995900541ca97b2c9","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2226fcac61f5f77ed8bcf1443cada6e714f0219501e5b999600332ddff8f436b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.36328125,-1.1357421875,-2.537109375],"temperature":1.0,"type":"choice","source_id":"5981","source_row_hash":"601253fff66fad5594ffa03232ae14805096eeb3edddfabc89f23fdf86061586","input_tokens":53,"truncated":false},
+ {"id":"massive-5981-noul","group_id":"massive-alarm-3c4995900541ca97b2c9","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"757dbc799dab322bc0fdd47f72321799a7049226e9dd8e045289b95fd6aa6c52","option_order":["false","true"],"gold":"false","raw_logits":[-2.056640625,1.291015625],"temperature":1.0,"type":"noul","source_id":"5981","source_row_hash":"601253fff66fad5594ffa03232ae14805096eeb3edddfabc89f23fdf86061586","input_tokens":48,"truncated":false},
+ {"id":"massive-60-choice","group_id":"massive-alarm-94ab087f60890852e463","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"dfc4bfc4a569b92c621afb5cc8c39261cd7db70bd6d3ee9f1fa4f709bc840b7f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.384765625,-2.48046875,1.759765625],"temperature":1.0,"type":"choice","source_id":"60","source_row_hash":"0a0aea9a63bd5ab159c323cdd5274587f022744194c0d7b63de7f75fc36db068","input_tokens":51,"truncated":false},
+ {"id":"massive-60-noul","group_id":"massive-alarm-94ab087f60890852e463","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"8d2c25f4e5a2219cdb422d36d08c2f2c51bde324bd447a0785ba5716405cc86a","option_order":["false","true"],"gold":"false","raw_logits":[-3.03515625,1.7685546875],"temperature":1.0,"type":"noul","source_id":"60","source_row_hash":"0a0aea9a63bd5ab159c323cdd5274587f022744194c0d7b63de7f75fc36db068","input_tokens":46,"truncated":false},
+ {"id":"massive-603-choice","group_id":"massive-alarm-2fe60ba4951e76d11029","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a2c3805f89c00f884c27440a1dc6ca929b7eeb892ce4ceada9d89cae658a68e2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.365234375,-4.1015625,0.50048828125],"temperature":1.0,"type":"choice","source_id":"603","source_row_hash":"c421b5916d3212e4b01829e3c79d953d94f710d02093a9cae17e52e58d84f7e8","input_tokens":53,"truncated":false},
+ {"id":"massive-603-noul","group_id":"massive-alarm-2fe60ba4951e76d11029","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"16ed49a37ca746de796382b34b307c96bd23e13298e2da96f4ba2c4d43c40a1e","option_order":["false","true"],"gold":"true","raw_logits":[-0.6171875,0.98193359375],"temperature":1.0,"type":"noul","source_id":"603","source_row_hash":"c421b5916d3212e4b01829e3c79d953d94f710d02093a9cae17e52e58d84f7e8","input_tokens":48,"truncated":false},
+ {"id":"massive-611-choice","group_id":"massive-alarm-8563c786dc56bc354169","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"7d47fe013fa4abd1bef6768aaa6e056adc5c74153c2b553727f66e3455fc540d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[7.2578125,-4.58203125,-4.5859375],"temperature":1.0,"type":"choice","source_id":"611","source_row_hash":"bcebdb7a1783441cd8df39b63bf7fcb15b183ef32d3ed586988011b72526dd5e","input_tokens":56,"truncated":false},
+ {"id":"massive-611-noul","group_id":"massive-alarm-8563c786dc56bc354169","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"2c9aee36cc50bdb10c9f6773fa1550f9c16e930556d25a6985c09c8505a49de6","option_order":["false","true"],"gold":"true","raw_logits":[-2.419921875,0.92041015625],"temperature":1.0,"type":"noul","source_id":"611","source_row_hash":"bcebdb7a1783441cd8df39b63bf7fcb15b183ef32d3ed586988011b72526dd5e","input_tokens":51,"truncated":false},
+ {"id":"massive-6161-choice","group_id":"massive-alarm-ce6e77c02022e073ef83","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a8c2d9d0d4c08ca16e412474a4c2e9403ffab98cbe9aa1d0e3d144dbc1b52a1a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.0234375,-3.31640625,1.15234375],"temperature":1.0,"type":"choice","source_id":"6161","source_row_hash":"3b9476d18f0671741201b748e7aae8b4bef4da841422e3890047663b8cfc6b58","input_tokens":49,"truncated":false},
+ {"id":"massive-6161-noul","group_id":"massive-alarm-ce6e77c02022e073ef83","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"de9c4de3d736330a69d1500ab613cb567c5d8a860bdd585ae9492f5e000f903b","option_order":["false","true"],"gold":"false","raw_logits":[-1.6689453125,0.68359375],"temperature":1.0,"type":"noul","source_id":"6161","source_row_hash":"3b9476d18f0671741201b748e7aae8b4bef4da841422e3890047663b8cfc6b58","input_tokens":44,"truncated":false},
+ {"id":"massive-7042-choice","group_id":"massive-alarm-d393bf328d9e34b61362","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"fbeee415bfc936dd946bd415f67e0202af1611d6e2e36b7a7c5975d736165f02","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.8984375,-4.5703125,-3.0546875],"temperature":1.0,"type":"choice","source_id":"7042","source_row_hash":"c88b745c72a0ae2239775dec1abf7adfa60007cd0d466769d82d72f70bfbfc16","input_tokens":55,"truncated":false},
+ {"id":"massive-7042-noul","group_id":"massive-alarm-d393bf328d9e34b61362","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"df2947e1b837c93ad24d1ba998064dae52f9b9deb1ef572a8b7b2b5f12ea8595","option_order":["false","true"],"gold":"true","raw_logits":[-1.6767578125,0.1278076171875],"temperature":1.0,"type":"noul","source_id":"7042","source_row_hash":"c88b745c72a0ae2239775dec1abf7adfa60007cd0d466769d82d72f70bfbfc16","input_tokens":50,"truncated":false},
+ {"id":"massive-7277-choice","group_id":"massive-alarm-debb470de19756d91704","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"24fbe87da2055f9d886b5876c64a68454a41f99da5507c2686870465a513f63e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.0390625,-3.416015625,-1.0498046875],"temperature":1.0,"type":"choice","source_id":"7277","source_row_hash":"aeae185a00d916458ae3ab6aa4bfff1cd313a140872c45d27d15d576fe0ca9bc","input_tokens":53,"truncated":false},
+ {"id":"massive-7277-noul","group_id":"massive-alarm-debb470de19756d91704","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"05b9feb244fca27da4be4e37c2cf62fdd13a90d5927b297ec2467bee1ebcaf42","option_order":["false","true"],"gold":"true","raw_logits":[-2.09375,1.1689453125],"temperature":1.0,"type":"noul","source_id":"7277","source_row_hash":"aeae185a00d916458ae3ab6aa4bfff1cd313a140872c45d27d15d576fe0ca9bc","input_tokens":48,"truncated":false},
+ {"id":"massive-7865-choice","group_id":"massive-alarm-d5d345224d210f90f9f5","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"8b0593a1fdf0d6c7ced2ac1338010fe12e88ba4292f508dc4866529053d416c3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.99609375,-3.4453125,-2.880859375],"temperature":1.0,"type":"choice","source_id":"7865","source_row_hash":"1d2893dc1fc8769432ee4ae1da4f838791060f36eeb8a5465921654fea2966a2","input_tokens":53,"truncated":false},
+ {"id":"massive-7865-noul","group_id":"massive-alarm-d5d345224d210f90f9f5","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"ade30874b49054351cc4d461647189dd5673e5a4a54e10ba7a5316569792c804","option_order":["false","true"],"gold":"true","raw_logits":[-2.541015625,1.515625],"temperature":1.0,"type":"noul","source_id":"7865","source_row_hash":"1d2893dc1fc8769432ee4ae1da4f838791060f36eeb8a5465921654fea2966a2","input_tokens":48,"truncated":false},
+ {"id":"massive-8252-choice","group_id":"massive-alarm-b0a22b8198f4993e1291","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"ba3abaddc76eb45141e707cb4d2cae8e7bbd610553b4c9cbfcb65bc0ee921400","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.5078125,-3.552734375,-1.6689453125],"temperature":1.0,"type":"choice","source_id":"8252","source_row_hash":"47b417240a61f167aba697da9695e406562d95fbcfa8317d40ff52b052c7329b","input_tokens":50,"truncated":false},
+ {"id":"massive-8252-noul","group_id":"massive-alarm-b0a22b8198f4993e1291","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"f2f944c5c1af54dd0f1ea98d42d7a2f19324eb3f9c9be5ca3bba7a6701a567b6","option_order":["false","true"],"gold":"true","raw_logits":[-2.232421875,1.2470703125],"temperature":1.0,"type":"noul","source_id":"8252","source_row_hash":"47b417240a61f167aba697da9695e406562d95fbcfa8317d40ff52b052c7329b","input_tokens":45,"truncated":false},
+ {"id":"massive-8374-choice","group_id":"massive-alarm-b2c391a264a7fc0e8d05","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a54daba394b794d67ecb3fb82069387e93349d371a89e7d827f80635326ef79b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.12408447265625,-2.046875,-0.1451416015625],"temperature":1.0,"type":"choice","source_id":"8374","source_row_hash":"f9b7880829291847e8117d7cc48838e1f2289986e30778ecb70f1426263e361f","input_tokens":53,"truncated":false},
+ {"id":"massive-8374-noul","group_id":"massive-alarm-b2c391a264a7fc0e8d05","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"81070c78d0e7723baaaf3f84c1722d81c18fe2e65502cfa00afa997dffefb8b0","option_order":["false","true"],"gold":"true","raw_logits":[-0.9482421875,1.0],"temperature":1.0,"type":"noul","source_id":"8374","source_row_hash":"f9b7880829291847e8117d7cc48838e1f2289986e30778ecb70f1426263e361f","input_tokens":48,"truncated":false},
+ {"id":"massive-838-choice","group_id":"massive-alarm-9c1aa24d8856b1a9b594","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2e7152c29865640d059a093638cf15c424a87c28b0fbe0919bfc7f40c72e596b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.068359375,-4.296875,1.1064453125],"temperature":1.0,"type":"choice","source_id":"838","source_row_hash":"0374f1afd3ddf573375d73c97e68b801e7847739ab120bddcddb5f24ddf9301f","input_tokens":52,"truncated":false},
+ {"id":"massive-838-noul","group_id":"massive-alarm-9c1aa24d8856b1a9b594","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"5553f4f5fc0c112d6e39cc75d646adbf627ea97450649ed1d5fc19aef431748b","option_order":["false","true"],"gold":"false","raw_logits":[-1.46875,-0.341796875],"temperature":1.0,"type":"noul","source_id":"838","source_row_hash":"0374f1afd3ddf573375d73c97e68b801e7847739ab120bddcddb5f24ddf9301f","input_tokens":47,"truncated":false},
+ {"id":"massive-8881-choice","group_id":"massive-alarm-a3097c674dea080a2da5","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d54982cbf06da42f2e2f43ca280cb876161678636d5bb6e8ffee7a0d3bb93e7a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.8544921875,-2.931640625,-2.2109375],"temperature":1.0,"type":"choice","source_id":"8881","source_row_hash":"b4a67a7ddf8a2064162255cea468172e3b6117bc5912380b2561538d15fc942f","input_tokens":55,"truncated":false},
+ {"id":"massive-8881-noul","group_id":"massive-alarm-a3097c674dea080a2da5","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"158e662345ac34c7894aaae13fa572924b0d5f2caacf6e3b84764fb118746db5","option_order":["false","true"],"gold":"true","raw_logits":[-2.541015625,1.11328125],"temperature":1.0,"type":"noul","source_id":"8881","source_row_hash":"b4a67a7ddf8a2064162255cea468172e3b6117bc5912380b2561538d15fc942f","input_tokens":50,"truncated":false},
+ {"id":"massive-8931-choice","group_id":"massive-alarm-db8555fe3197239b3206","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d980fa7dedacf5e72ae37665538e2793b4f022673a8d82eb2e41284118fa51d0","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.3046875,-2.083984375,-1.1513671875],"temperature":1.0,"type":"choice","source_id":"8931","source_row_hash":"8f70a2fc9af29a53f173113c232962f4df7836f331b2282b37caf1ccab49a781","input_tokens":52,"truncated":false},
+ {"id":"massive-8931-noul","group_id":"massive-alarm-db8555fe3197239b3206","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"7ec3b962a5c5a78cb83727987c51167dc8c8d7c76e5cbc8931b315728f12ca47","option_order":["false","true"],"gold":"true","raw_logits":[-1.7548828125,1.1884765625],"temperature":1.0,"type":"noul","source_id":"8931","source_row_hash":"8f70a2fc9af29a53f173113c232962f4df7836f331b2282b37caf1ccab49a781","input_tokens":47,"truncated":false},
+ {"id":"massive-9113-choice","group_id":"massive-alarm-397aba5493c3b2aa0106","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"cbf50a8bce382e1c8069c25f24d9187c0581bbf3a4557c96ba8e8c0bd4d094e4","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.126953125,-4.140625,0.7939453125],"temperature":1.0,"type":"choice","source_id":"9113","source_row_hash":"fb4976f4148eaf6a56d495a37fe34a0ea3cbc0cde2a2b924dfb2bb9d43e3fcc9","input_tokens":55,"truncated":false},
+ {"id":"massive-9113-noul","group_id":"massive-alarm-397aba5493c3b2aa0106","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"1eb2651db8e5fb0ffe9c87a86a1eea9a93c803613b6109ed7b243775301390bc","option_order":["false","true"],"gold":"true","raw_logits":[-0.2366943359375,0.02410888671875],"temperature":1.0,"type":"noul","source_id":"9113","source_row_hash":"fb4976f4148eaf6a56d495a37fe34a0ea3cbc0cde2a2b924dfb2bb9d43e3fcc9","input_tokens":50,"truncated":false},
+ {"id":"massive-919-choice","group_id":"massive-alarm-4c0a77d504ba07897c11","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1df3beb000463e7bc08b13bb810b877d176680c61686b53599cffc5b0d579beb","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.984375,-3.20703125,4.3359375],"temperature":1.0,"type":"choice","source_id":"919","source_row_hash":"b899f7728a260ed0af75e47ce729fb61bac2aa810bbdcc2b39e068dd142e1140","input_tokens":52,"truncated":false},
+ {"id":"massive-919-noul","group_id":"massive-alarm-4c0a77d504ba07897c11","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"65b4e822aafcca31aac32995a5110244feba92743a197f2bb18d469b77384537","option_order":["false","true"],"gold":"false","raw_logits":[-1.474609375,0.272705078125],"temperature":1.0,"type":"noul","source_id":"919","source_row_hash":"b899f7728a260ed0af75e47ce729fb61bac2aa810bbdcc2b39e068dd142e1140","input_tokens":47,"truncated":false},
+ {"id":"massive-0-choice","group_id":"massive-alarm-a37bbbb0764cc5bf8d8c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"77b0b32f31535aa2bdd9f6f747335478e4a43d19f2018d740cec1c1fa264a584","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.77197265625,-3.3671875,-0.1776123046875],"temperature":1.0,"type":"choice","source_id":"0","source_row_hash":"c7526effce043a65aeb540d2fd4ed2c638ea92efb82184525bfa946c59419162","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.77197265625,-3.3671875,-0.1776123046875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-0-noul","group_id":"massive-alarm-a37bbbb0764cc5bf8d8c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"35ff02590dfdcdf3d2f174132d6178af71b02166065f1ef047c1e306f4cae5f9","option_order":["false","true"],"gold":"true","raw_logits":[-1.6298828125,0.81640625],"temperature":1.0,"type":"noul","source_id":"0","source_row_hash":"c7526effce043a65aeb540d2fd4ed2c638ea92efb82184525bfa946c59419162","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.6298828125,0.81640625],"calibrated_temperature":5.0},
+ {"id":"massive-1205-choice","group_id":"massive-alarm-46ec519726ad99259312","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c5336a0dbeaab10f6d75e474cd8ca690a4a17b614be1c36aa9bc422f3420b0e3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.78125,-4.015625,2.580078125],"temperature":1.0,"type":"choice","source_id":"1205","source_row_hash":"c6f272625d9dddb2157b268ff169f20764ccb2c15d8194a208dd4572661cf287","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-1.78125,-4.015625,2.580078125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1205-noul","group_id":"massive-alarm-46ec519726ad99259312","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ad37d94fa29dae7c5a19037407667cf3ec16629415f0f66c6d35b1bf2d99bd10","option_order":["false","true"],"gold":"false","raw_logits":[-1.56640625,0.179443359375],"temperature":1.0,"type":"noul","source_id":"1205","source_row_hash":"c6f272625d9dddb2157b268ff169f20764ccb2c15d8194a208dd4572661cf287","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.56640625,0.179443359375],"calibrated_temperature":5.0},
+ {"id":"massive-1264-choice","group_id":"massive-alarm-81721bc34387bf04c532","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"5d1b1f024710e0ccc55d6f8617daa19c339f01457cfb0e6ce8f41ad766f25b55","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[1.2314453125,-1.3837890625,-0.91796875],"temperature":1.0,"type":"choice","source_id":"1264","source_row_hash":"b0e2b6e0b5645a28c5ef36a7b84a2f2c1967d0af1db72136cb632466ac4a2bde","input_tokens":53,"truncated":false,"calibrated_raw_logits":[1.2314453125,-1.3837890625,-0.91796875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1264-noul","group_id":"massive-alarm-81721bc34387bf04c532","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"132b85b5df1b397bcfc73c5408e767840171c95421a29b96e6f3ead9162b793b","option_order":["false","true"],"gold":"false","raw_logits":[-2.1015625,0.94873046875],"temperature":1.0,"type":"noul","source_id":"1264","source_row_hash":"b0e2b6e0b5645a28c5ef36a7b84a2f2c1967d0af1db72136cb632466ac4a2bde","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.1015625,0.94873046875],"calibrated_temperature":5.0},
+ {"id":"massive-1452-choice","group_id":"massive-alarm-426bdf23716beff075f4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ba17e7a041b87e59af379c45db0671a59b60e5fa051c7555195d8a9da3aa2337","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.24609375,-3.59375,1.53515625],"temperature":1.0,"type":"choice","source_id":"1452","source_row_hash":"1bfe246bb5276edcdcb98bb0446151ac77373bd9a92f38c219359eb38e6f7682","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-3.24609375,-3.59375,1.53515625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1452-noul","group_id":"massive-alarm-426bdf23716beff075f4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"12849a0b88a8ff2a993968e6b0182681c95a4694070c5ccacee0ed5f2b6524a0","option_order":["false","true"],"gold":"false","raw_logits":[-1.767578125,0.370849609375],"temperature":1.0,"type":"noul","source_id":"1452","source_row_hash":"1bfe246bb5276edcdcb98bb0446151ac77373bd9a92f38c219359eb38e6f7682","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.767578125,0.370849609375],"calibrated_temperature":5.0},
+ {"id":"massive-1454-choice","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ce41769466fb7357d3ccff960d17d722b3d6847a9cb11833b48f3c85dca068d5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.6337890625,-3.45703125,1.048828125],"temperature":1.0,"type":"choice","source_id":"1454","source_row_hash":"145608279739b62c2458975fa2f71167df19ca5eba37351354515f842abe0aee","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-1.6337890625,-3.45703125,1.048828125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1454-noul","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"792f47e0d63956a5accbee0429b61de43a74bda6d3f490cf082803c5b8f38138","option_order":["false","true"],"gold":"false","raw_logits":[-2.0078125,0.259521484375],"temperature":1.0,"type":"noul","source_id":"1454","source_row_hash":"145608279739b62c2458975fa2f71167df19ca5eba37351354515f842abe0aee","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.0078125,0.259521484375],"calibrated_temperature":5.0},
+ {"id":"massive-1483-choice","group_id":"massive-alarm-2c21587cc19c72b110b3","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"f30db95f41dc4f0696ec72519901b57e25e672544e091d5cb2dc86ea24665f7e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[6.015625,-2.25390625,-2.994140625],"temperature":1.0,"type":"choice","source_id":"1483","source_row_hash":"890f145f8f9f0b5fd1d7091f1f3c2e12575f505e096fb255d1ead352d3634b9a","input_tokens":51,"truncated":false,"calibrated_raw_logits":[6.015625,-2.25390625,-2.994140625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1483-noul","group_id":"massive-alarm-2c21587cc19c72b110b3","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7ddce951250db5a4434c1ab2a0c87281815f837602586c62fb1730e47234864b","option_order":["false","true"],"gold":"false","raw_logits":[-1.54296875,0.85107421875],"temperature":1.0,"type":"noul","source_id":"1483","source_row_hash":"890f145f8f9f0b5fd1d7091f1f3c2e12575f505e096fb255d1ead352d3634b9a","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.54296875,0.85107421875],"calibrated_temperature":5.0},
+ {"id":"massive-1635-choice","group_id":"massive-alarm-f9e0722a15d811b9f803","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"bb88865e7bc83986e78d070eaa64317d656ec97f98105b2ad26d92a38d668e08","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.658203125,0.01424407958984375,-2.0234375],"temperature":1.0,"type":"choice","source_id":"1635","source_row_hash":"d1953237537debb810d6f0e9c57dfd7c06257fcb558e7b2a16b137f96c4304d0","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.658203125,0.01424407958984375,-2.0234375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1635-noul","group_id":"massive-alarm-f9e0722a15d811b9f803","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ea73bed5f72a1db4db41433a4c84c6d710acf14be98e446f68d74d9b1dcdfd05","option_order":["false","true"],"gold":"false","raw_logits":[-1.5322265625,0.79248046875],"temperature":1.0,"type":"noul","source_id":"1635","source_row_hash":"d1953237537debb810d6f0e9c57dfd7c06257fcb558e7b2a16b137f96c4304d0","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.5322265625,0.79248046875],"calibrated_temperature":5.0},
+ {"id":"massive-1646-choice","group_id":"massive-alarm-62749a68e3e345ba0baf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"64d7678b68d45c87ca45fc5bf39f26b3d45490f848ea4a4b5a760af9d994e293","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.42578125,-0.8388671875,-0.951171875],"temperature":1.0,"type":"choice","source_id":"1646","source_row_hash":"45be821060a8270e88285729f71d4851684eccb3f8e08dc3af41d53e22bd4a93","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.42578125,-0.8388671875,-0.951171875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1646-noul","group_id":"massive-alarm-62749a68e3e345ba0baf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fea796b609111eb3c000991a208e87211ae8c359a7cdc07e754bc56caa22d373","option_order":["false","true"],"gold":"false","raw_logits":[-2.109375,1.1884765625],"temperature":1.0,"type":"noul","source_id":"1646","source_row_hash":"45be821060a8270e88285729f71d4851684eccb3f8e08dc3af41d53e22bd4a93","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.109375,1.1884765625],"calibrated_temperature":5.0},
+ {"id":"massive-1658-choice","group_id":"massive-alarm-364266652280dd7c910e","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0230e908a3762610b091476be9f4a05d8480c6e8cbaa6f719a31d29d363d5a58","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.298828125,-0.3212890625,-0.67919921875],"temperature":1.0,"type":"choice","source_id":"1658","source_row_hash":"93fa8ba611a374efaa1f7aaead3325da80c2330ae0ae622640f247c61e01d54b","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.298828125,-0.3212890625,-0.67919921875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1658-noul","group_id":"massive-alarm-364266652280dd7c910e","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"84713593dc3c74a749551857971c03d1a91d64f10e9b64a4c0f9dcde5e4a3ea7","option_order":["false","true"],"gold":"false","raw_logits":[-1.306640625,0.36962890625],"temperature":1.0,"type":"noul","source_id":"1658","source_row_hash":"93fa8ba611a374efaa1f7aaead3325da80c2330ae0ae622640f247c61e01d54b","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.306640625,0.36962890625],"calibrated_temperature":5.0},
+ {"id":"massive-1675-choice","group_id":"massive-alarm-bf4ac6ee8fec3f0b558c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1466e3c3f90ac2ac30f3f4f1b0d94ead57c08f3d7497b6ef1f9ea9c4eac8711f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.48046875,-4.32421875,2.890625],"temperature":1.0,"type":"choice","source_id":"1675","source_row_hash":"4d208d74d0efe7e2c8ca898d3b22951bcde4612a05aa052742a78e2af0f54efa","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-3.48046875,-4.32421875,2.890625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1675-noul","group_id":"massive-alarm-bf4ac6ee8fec3f0b558c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"275b4e3cc0377df561399ef1fad6be14b832e24bd56aae9ad0a547a7d358c00e","option_order":["false","true"],"gold":"false","raw_logits":[-2.11328125,0.400146484375],"temperature":1.0,"type":"noul","source_id":"1675","source_row_hash":"4d208d74d0efe7e2c8ca898d3b22951bcde4612a05aa052742a78e2af0f54efa","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.11328125,0.400146484375],"calibrated_temperature":5.0},
+ {"id":"massive-1719-choice","group_id":"massive-alarm-3aeb032bc1ab13756f0a","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8c5933a9aeb20b97adbd1e5b98c5cbd42e62e879e8e780ffa59c46e4d607a736","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.390625,-3.01171875,-1.6962890625],"temperature":1.0,"type":"choice","source_id":"1719","source_row_hash":"c920466329cad750fe4b3605a5197b2eb9ea1b411cb18b89cd499d0a209dd0da","input_tokens":56,"truncated":false,"calibrated_raw_logits":[4.390625,-3.01171875,-1.6962890625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1719-noul","group_id":"massive-alarm-3aeb032bc1ab13756f0a","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fd8b7628ff216af2d1a0577794f912897a4b1852599ad9d32b74f55fe23bfdc1","option_order":["false","true"],"gold":"true","raw_logits":[-1.40625,0.93798828125],"temperature":1.0,"type":"noul","source_id":"1719","source_row_hash":"c920466329cad750fe4b3605a5197b2eb9ea1b411cb18b89cd499d0a209dd0da","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.40625,0.93798828125],"calibrated_temperature":5.0},
+ {"id":"massive-174-choice","group_id":"massive-alarm-ce0229b7c6149d882cf3","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7839990611efc7091e313b70ab12446f38e919bf3775e40e3288cb228c8e2b77","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.22265625,-2.859375,-1.3046875],"temperature":1.0,"type":"choice","source_id":"174","source_row_hash":"e8f8762e0d5998c6602731393d2fab6fdfb7662662c9a73a02e95041799cc0a9","input_tokens":54,"truncated":false,"calibrated_raw_logits":[4.22265625,-2.859375,-1.3046875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-174-noul","group_id":"massive-alarm-ce0229b7c6149d882cf3","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"46d0f88a1bb77721bef837b24d61db6244bac9678ca6744ac32fa010615a69c6","option_order":["false","true"],"gold":"true","raw_logits":[-2.4375,1.5888671875],"temperature":1.0,"type":"noul","source_id":"174","source_row_hash":"e8f8762e0d5998c6602731393d2fab6fdfb7662662c9a73a02e95041799cc0a9","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.4375,1.5888671875],"calibrated_temperature":5.0},
+ {"id":"massive-1774-choice","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6b00cec1b29c2fa5a974ad219e812cc757b1e41937665cfacf265bb6c5a0a327","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.6337890625,-3.45703125,1.048828125],"temperature":1.0,"type":"choice","source_id":"1774","source_row_hash":"e2d9115544309a1c262318bb19f1d82615d1bb4704ad53d26c3bb5265d1cd2f0","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-1.6337890625,-3.45703125,1.048828125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1774-noul","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"10fce9ced8e6eec5b8b870dbf1b532e302afac2b8bc85d93119e264ea2ffebe3","option_order":["false","true"],"gold":"false","raw_logits":[-2.0078125,0.259521484375],"temperature":1.0,"type":"noul","source_id":"1774","source_row_hash":"e2d9115544309a1c262318bb19f1d82615d1bb4704ad53d26c3bb5265d1cd2f0","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.0078125,0.259521484375],"calibrated_temperature":5.0},
+ {"id":"massive-1862-choice","group_id":"massive-alarm-869c1474b1c82e78b00c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"81e679b9cc2bda219fc616a52ccb9d7eecdad4f0c7350f44c21a5b12e38f9f44","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.253173828125,-0.1468505859375,-0.475341796875],"temperature":1.0,"type":"choice","source_id":"1862","source_row_hash":"d7dcb916b78b9417322277ed5947fe9386a9b6a258ec5d7237bf0a8db3eab495","input_tokens":51,"truncated":false,"calibrated_raw_logits":[0.253173828125,-0.1468505859375,-0.475341796875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1862-noul","group_id":"massive-alarm-869c1474b1c82e78b00c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"03ca4bdaf3b5b365cb529d8d6fcd255ea2c26a37f058442867ac3939e66adcf1","option_order":["false","true"],"gold":"false","raw_logits":[-1.8515625,1.146484375],"temperature":1.0,"type":"noul","source_id":"1862","source_row_hash":"d7dcb916b78b9417322277ed5947fe9386a9b6a258ec5d7237bf0a8db3eab495","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.8515625,1.146484375],"calibrated_temperature":5.0},
+ {"id":"massive-1913-choice","group_id":"massive-alarm-04a340be86134fb825f2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0c21d545d8a862238e988f7cb58faa8ea1930a3d9adddb2d93adc69289d959ef","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.48974609375,-2.974609375,-0.37744140625],"temperature":1.0,"type":"choice","source_id":"1913","source_row_hash":"fd24179d6c9dc537d667cb6ff6fe25ab9e00652ce36f045a39da3a2eb3efeeae","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-0.48974609375,-2.974609375,-0.37744140625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1913-noul","group_id":"massive-alarm-04a340be86134fb825f2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f821536cce50ca455cf0e83aa0589f117ccdc48ac96ee7b0de3c0e3f00fe5885","option_order":["false","true"],"gold":"true","raw_logits":[-2.65234375,1.6552734375],"temperature":1.0,"type":"noul","source_id":"1913","source_row_hash":"fd24179d6c9dc537d667cb6ff6fe25ab9e00652ce36f045a39da3a2eb3efeeae","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.65234375,1.6552734375],"calibrated_temperature":5.0},
+ {"id":"massive-1982-choice","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"046ef58708776eb61d36aba5282ff98022582353215e6eae4cb389226c6c9ff2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.10546875,-3.75,-2.833984375],"temperature":1.0,"type":"choice","source_id":"1982","source_row_hash":"68a92316a11c74af3d569a87cc55e8d8e9f116b771db9601d3c65b44d659be25","input_tokens":55,"truncated":false,"calibrated_raw_logits":[5.10546875,-3.75,-2.833984375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-1982-noul","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"6f13803fd13041a079ba5f448a57ab23e1d609a8fd17f4c9b68635b312890988","option_order":["false","true"],"gold":"true","raw_logits":[-2.30859375,2.40625],"temperature":1.0,"type":"noul","source_id":"1982","source_row_hash":"68a92316a11c74af3d569a87cc55e8d8e9f116b771db9601d3c65b44d659be25","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-2.30859375,2.40625],"calibrated_temperature":5.0},
+ {"id":"massive-201-choice","group_id":"massive-alarm-34510ceb5116eee89abf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1919418590df3569bb082e98b9368556569025bf81fd8c5c80d0f62fb8012279","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.8603515625,-2.9609375,3.517578125],"temperature":1.0,"type":"choice","source_id":"201","source_row_hash":"fec125430204fc240f59fa0b8afc5ab33fb231589e377ca142563556d36313c1","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.8603515625,-2.9609375,3.517578125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-201-noul","group_id":"massive-alarm-34510ceb5116eee89abf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e682c7af0260299ab1b72d73ba69134a323fd6c0a4488f9e8ebb899d12e2f95c","option_order":["false","true"],"gold":"false","raw_logits":[-2.462890625,0.419921875],"temperature":1.0,"type":"noul","source_id":"201","source_row_hash":"fec125430204fc240f59fa0b8afc5ab33fb231589e377ca142563556d36313c1","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.462890625,0.419921875],"calibrated_temperature":5.0},
+ {"id":"massive-2134-choice","group_id":"massive-alarm-f8e7f38a4927528ba9b2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8cb6e812da09a75b6e72019b09c11e0a85c207aa5c8dbfddc570fd92d30458de","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.0035247802734375,-3.1875,-0.9208984375],"temperature":1.0,"type":"choice","source_id":"2134","source_row_hash":"7d7e16a4e1ed61a8de9442d3e01f17fafdc055b6548e09778de80844d7b5df78","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-0.0035247802734375,-3.1875,-0.9208984375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2134-noul","group_id":"massive-alarm-f8e7f38a4927528ba9b2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"9689620330e6aaa79089562fe3f74c74152b155ba69f48bbb97341c234944c63","option_order":["false","true"],"gold":"true","raw_logits":[-2.033203125,1.36328125],"temperature":1.0,"type":"noul","source_id":"2134","source_row_hash":"7d7e16a4e1ed61a8de9442d3e01f17fafdc055b6548e09778de80844d7b5df78","input_tokens":43,"truncated":false,"calibrated_raw_logits":[-2.033203125,1.36328125],"calibrated_temperature":5.0},
+ {"id":"massive-2166-choice","group_id":"massive-alarm-9519286825bfef55a4e0","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"69ab8ef1a844ffe7974da7b965063744250ea864ceb8d5a1efbdbdaa3022c36c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.90625,-4.01171875,-3.65234375],"temperature":1.0,"type":"choice","source_id":"2166","source_row_hash":"06505fac3fedd536d679acbcbc516ba66217322dc627d35a70fed7066fc76658","input_tokens":54,"truncated":false,"calibrated_raw_logits":[5.90625,-4.01171875,-3.65234375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2166-noul","group_id":"massive-alarm-9519286825bfef55a4e0","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"473a9f5612897736e32cafd3b6a9730e7740e1758fb072b71087bd827cdaf1b6","option_order":["false","true"],"gold":"true","raw_logits":[-2.603515625,1.41796875],"temperature":1.0,"type":"noul","source_id":"2166","source_row_hash":"06505fac3fedd536d679acbcbc516ba66217322dc627d35a70fed7066fc76658","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.603515625,1.41796875],"calibrated_temperature":5.0},
+ {"id":"massive-2167-choice","group_id":"massive-alarm-d02ea237a24fb1becb66","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cde8b8d2e32f6683425e46e437ca44af31aa51bb38d8e675498c818dde0ec4d8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-1.3681640625,-3.48046875,0.047393798828125],"temperature":1.0,"type":"choice","source_id":"2167","source_row_hash":"bc3911b9a1c3926a1c45f4eb47de667f89e300ef4b21a13428a67c529a0efa24","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.3681640625,-3.48046875,0.047393798828125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2167-noul","group_id":"massive-alarm-d02ea237a24fb1becb66","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"83343991208595707f936ac6728585bd5637a32b678c39ae9802bf00e123d2f7","option_order":["false","true"],"gold":"true","raw_logits":[-1.54296875,1.0595703125],"temperature":1.0,"type":"noul","source_id":"2167","source_row_hash":"bc3911b9a1c3926a1c45f4eb47de667f89e300ef4b21a13428a67c529a0efa24","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.54296875,1.0595703125],"calibrated_temperature":5.0},
+ {"id":"massive-2168-choice","group_id":"massive-alarm-cff75109d691bab986bf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7d2f0a6e8d4f34fa7bbf37a4a5aed96e8976d9c64cfe47bbc2c9660bdaaf199c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-0.76611328125,-3.515625,2.119140625],"temperature":1.0,"type":"choice","source_id":"2168","source_row_hash":"43fe92fb79734cdcfc39bb3d6f1df8e11c61fff1e834e468a137c272fb88ddb4","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-0.76611328125,-3.515625,2.119140625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2168-noul","group_id":"massive-alarm-cff75109d691bab986bf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"b6eb787be2412c61dd4e39b1920656d4a1bd9f081a93f2fea79db2f421bd3f37","option_order":["false","true"],"gold":"false","raw_logits":[-1.7783203125,0.79736328125],"temperature":1.0,"type":"noul","source_id":"2168","source_row_hash":"43fe92fb79734cdcfc39bb3d6f1df8e11c61fff1e834e468a137c272fb88ddb4","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.7783203125,0.79736328125],"calibrated_temperature":5.0},
+ {"id":"massive-2334-choice","group_id":"massive-alarm-68e901c570a8bc247ea5","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"dad5cc765a8b621e7d161351f5c7d96e7990e74890f935c8da2bcc8f4465dcaf","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.072265625,-4.1484375,-0.0853271484375],"temperature":1.0,"type":"choice","source_id":"2334","source_row_hash":"37ed15aaa5179f417ed1db5f5e5fbf3fd407b5f570abad133bf0c99a1f824b8a","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.072265625,-4.1484375,-0.0853271484375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2334-noul","group_id":"massive-alarm-68e901c570a8bc247ea5","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"979a7e2f21c7c3cc727090ad2a84fe0ff99b663a18c7fa9599ef652f08bb46d2","option_order":["false","true"],"gold":"true","raw_logits":[-0.97021484375,1.2099609375],"temperature":1.0,"type":"noul","source_id":"2334","source_row_hash":"37ed15aaa5179f417ed1db5f5e5fbf3fd407b5f570abad133bf0c99a1f824b8a","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-0.97021484375,1.2099609375],"calibrated_temperature":5.0},
+ {"id":"massive-2564-choice","group_id":"massive-alarm-606907415c29daf2edfb","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c44f45e943b2a653cda222a15ba4d17ed7ac9b9d5bae688e436eca315eaa7e16","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.66796875,-3.73046875,-2.88671875],"temperature":1.0,"type":"choice","source_id":"2564","source_row_hash":"95d1ba4090b52ba0ce5fb3444edef0250d596ad13f4f69a621c4a90930827e9f","input_tokens":56,"truncated":false,"calibrated_raw_logits":[4.66796875,-3.73046875,-2.88671875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2564-noul","group_id":"massive-alarm-606907415c29daf2edfb","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"87f3caebd55ff64483153128b3dc7a98c34a40ae17290470a5b5123949a92735","option_order":["false","true"],"gold":"true","raw_logits":[-2.638671875,1.4833984375],"temperature":1.0,"type":"noul","source_id":"2564","source_row_hash":"95d1ba4090b52ba0ce5fb3444edef0250d596ad13f4f69a621c4a90930827e9f","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.638671875,1.4833984375],"calibrated_temperature":5.0},
+ {"id":"massive-2673-choice","group_id":"massive-alarm-8691ee9b0cf94dc2728b","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2e1c9599701f771117cd159e0ce73abb95b349419aacf2f200688397632c2c17","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.77294921875,-4.08203125,-2.396484375],"temperature":1.0,"type":"choice","source_id":"2673","source_row_hash":"1afaa7047e42a574d8e3230ec7c390cd32494a5058aadd8ceba92aa8d8381c72","input_tokens":58,"truncated":false,"calibrated_raw_logits":[-0.77294921875,-4.08203125,-2.396484375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2673-noul","group_id":"massive-alarm-8691ee9b0cf94dc2728b","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7a310326662b1ceb900e0ce2c672d76155807ff8771cd5e70b6705cdbf11793a","option_order":["false","true"],"gold":"false","raw_logits":[-1.7099609375,0.22412109375],"temperature":1.0,"type":"noul","source_id":"2673","source_row_hash":"1afaa7047e42a574d8e3230ec7c390cd32494a5058aadd8ceba92aa8d8381c72","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-1.7099609375,0.22412109375],"calibrated_temperature":5.0},
+ {"id":"massive-2870-choice","group_id":"massive-alarm-8a430eb6516ce866e134","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ad6afdc347ef21ce011640d137edaa887c55f4963941c347b23f6b7b21a67dd2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.9970703125,4.4453125,-2.201171875],"temperature":1.0,"type":"choice","source_id":"2870","source_row_hash":"30d1945137ce312fac5d58fb2da24a2c2971e65f2403ad3ac1204ea1451af255","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-0.9970703125,4.4453125,-2.201171875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2870-noul","group_id":"massive-alarm-8a430eb6516ce866e134","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"9d2b0868a4c40cd0a685f0a6b8d495f5d7d98653eab390d7b4b1421dac4b84f1","option_order":["false","true"],"gold":"false","raw_logits":[-1.0615234375,0.373046875],"temperature":1.0,"type":"noul","source_id":"2870","source_row_hash":"30d1945137ce312fac5d58fb2da24a2c2971e65f2403ad3ac1204ea1451af255","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.0615234375,0.373046875],"calibrated_temperature":5.0},
+ {"id":"massive-2957-choice","group_id":"massive-alarm-ee34ced4068dccdafbd4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2fc5caa4ae2985c5045322f402b6fbaec115c14788046392c7a1b7eb9e7278ee","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.16796875,-0.132080078125,-0.9990234375],"temperature":1.0,"type":"choice","source_id":"2957","source_row_hash":"cad2ff93096738a0cc95d453cd87af036fed8518b9c6804a76474027052392b2","input_tokens":52,"truncated":false,"calibrated_raw_logits":[4.16796875,-0.132080078125,-0.9990234375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2957-noul","group_id":"massive-alarm-ee34ced4068dccdafbd4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"c01ab9e555be8266d496a98398689b99aa6cb691f9d6a87897ebcd8dd5ee0a67","option_order":["false","true"],"gold":"false","raw_logits":[-1.783203125,1.5283203125],"temperature":1.0,"type":"noul","source_id":"2957","source_row_hash":"cad2ff93096738a0cc95d453cd87af036fed8518b9c6804a76474027052392b2","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.783203125,1.5283203125],"calibrated_temperature":5.0},
+ {"id":"massive-2983-choice","group_id":"massive-alarm-80f7027d6c0b934a23f1","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"92c9f6b64792017733c2fa0d9e61f8d9579da249ea18cb26cc1de02a7df19f04","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.037109375,-3.71875,0.493408203125],"temperature":1.0,"type":"choice","source_id":"2983","source_row_hash":"51967e6f2526f9032885c83079a3b964f6d61de7941722049a83749bad8643b9","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-3.037109375,-3.71875,0.493408203125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2983-noul","group_id":"massive-alarm-80f7027d6c0b934a23f1","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a91925e0946afa49892ffde2b9cb42fe9240c5b085c14bff1f9663280a3b296f","option_order":["false","true"],"gold":"false","raw_logits":[-2.271484375,0.56787109375],"temperature":1.0,"type":"noul","source_id":"2983","source_row_hash":"51967e6f2526f9032885c83079a3b964f6d61de7941722049a83749bad8643b9","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.271484375,0.56787109375],"calibrated_temperature":5.0},
+ {"id":"massive-2985-choice","group_id":"massive-alarm-cadeebcba526c5ce11e1","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c77bbecf7a4f64203fd3018a54e31f1f7e265dd20b4402e784ae564bf479b282","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.66796875,-4.046875,2.580078125],"temperature":1.0,"type":"choice","source_id":"2985","source_row_hash":"45de5ba6489c1472d66739caf602f19f3dddeb4654edb0633098362cb7245dc5","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-3.66796875,-4.046875,2.580078125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-2985-noul","group_id":"massive-alarm-cadeebcba526c5ce11e1","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ce7d0fa964fdf4b79e9e0090ed6dab1bf8d2a0ba07586c8de0979042fd97fabe","option_order":["false","true"],"gold":"false","raw_logits":[-2.65625,0.5087890625],"temperature":1.0,"type":"noul","source_id":"2985","source_row_hash":"45de5ba6489c1472d66739caf602f19f3dddeb4654edb0633098362cb7245dc5","input_tokens":43,"truncated":false,"calibrated_raw_logits":[-2.65625,0.5087890625],"calibrated_temperature":5.0},
+ {"id":"massive-3161-choice","group_id":"massive-alarm-76d4b32f23cb9a972b27","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"70f06e64a39a6ef6fe0fc89ca848b8a2d66e40dbe52d56c3b6b62aa198f16381","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.1484375,2.0390625,-0.23193359375],"temperature":1.0,"type":"choice","source_id":"3161","source_row_hash":"21b811432a21e010c74460ef7e1b4573c17346e6fd3f6b869abdd9ef6ae0c1ef","input_tokens":56,"truncated":false,"calibrated_raw_logits":[2.1484375,2.0390625,-0.23193359375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3161-noul","group_id":"massive-alarm-76d4b32f23cb9a972b27","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"2bbbb3a8362746db393fa624e9043f8dad23e400182090154aade1921dc28ffb","option_order":["false","true"],"gold":"false","raw_logits":[-1.357421875,0.58740234375],"temperature":1.0,"type":"noul","source_id":"3161","source_row_hash":"21b811432a21e010c74460ef7e1b4573c17346e6fd3f6b869abdd9ef6ae0c1ef","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.357421875,0.58740234375],"calibrated_temperature":5.0},
+ {"id":"massive-3162-choice","group_id":"massive-alarm-6e7b23cce190bc2d85c5","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ea7591a2f602f59dcb876b7ae44597587400890381314433fe0308a51e1f0d6d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.9912109375,-0.8330078125,-0.1859130859375],"temperature":1.0,"type":"choice","source_id":"3162","source_row_hash":"d15cb4a55b87aa0a3910749116e4161f2684c7546aa4c19dc3c4f99f0f375afe","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.9912109375,-0.8330078125,-0.1859130859375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3162-noul","group_id":"massive-alarm-6e7b23cce190bc2d85c5","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"de8da33771b147c3186d74aa65038831e1cb9ca49cd32cbe4c043568bcde65ae","option_order":["false","true"],"gold":"false","raw_logits":[-1.0283203125,-0.179443359375],"temperature":1.0,"type":"noul","source_id":"3162","source_row_hash":"d15cb4a55b87aa0a3910749116e4161f2684c7546aa4c19dc3c4f99f0f375afe","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.0283203125,-0.179443359375],"calibrated_temperature":5.0},
+ {"id":"massive-3213-choice","group_id":"massive-alarm-130f5b17e0470747d338","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0799db4758f49463128dbabd40c943aed44fcc559c87572c05e7ab6d128b4789","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.97900390625,-3.73828125,-0.10302734375],"temperature":1.0,"type":"choice","source_id":"3213","source_row_hash":"9fd345a0d9dc2aed398ed21031246ff893fe651a70a0d745ba90398f720b9acb","input_tokens":53,"truncated":false,"calibrated_raw_logits":[0.97900390625,-3.73828125,-0.10302734375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3213-noul","group_id":"massive-alarm-130f5b17e0470747d338","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f43c35f39a7dc113b60b52e5f60073b04f43737051ee83161483c56e9fe2cc57","option_order":["false","true"],"gold":"true","raw_logits":[-2.564453125,1.2177734375],"temperature":1.0,"type":"noul","source_id":"3213","source_row_hash":"9fd345a0d9dc2aed398ed21031246ff893fe651a70a0d745ba90398f720b9acb","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.564453125,1.2177734375],"calibrated_temperature":5.0},
+ {"id":"massive-3398-choice","group_id":"massive-alarm-da466cfc858133a8c89c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"dcb2df6d7f361c5165fff930443cdd8d616f8bfac79a1cfc8eef312ee5956872","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.0,-4.55859375,-3.73046875],"temperature":1.0,"type":"choice","source_id":"3398","source_row_hash":"bf8610ee8a05e9800dd90fb7461d52940fcfc1bf777b1aef8138784628240668","input_tokens":54,"truncated":false,"calibrated_raw_logits":[5.0,-4.55859375,-3.73046875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3398-noul","group_id":"massive-alarm-da466cfc858133a8c89c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f268081c1d5a7aa3525ee6e1106a49e8f95cde1b4b9f2cee63a2f1b497c0ca98","option_order":["false","true"],"gold":"true","raw_logits":[-1.8974609375,1.115234375],"temperature":1.0,"type":"noul","source_id":"3398","source_row_hash":"bf8610ee8a05e9800dd90fb7461d52940fcfc1bf777b1aef8138784628240668","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-1.8974609375,1.115234375],"calibrated_temperature":5.0},
+ {"id":"massive-3435-choice","group_id":"massive-alarm-462210a662d596b5be80","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"fd24fd1ddefae756012ea9809480034da52e1590ecb9f855fea742309bb004ad","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.26171875,-3.630859375,1.640625],"temperature":1.0,"type":"choice","source_id":"3435","source_row_hash":"e0d15beb0c523aab28095c68f8a36e66e1f0ce73e302239dc482aad749d2fc21","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-3.26171875,-3.630859375,1.640625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3435-noul","group_id":"massive-alarm-462210a662d596b5be80","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"da919c2441ffa633bc40be48a80679231ee8f87a5a09323f8d1b0de2feb94ef9","option_order":["false","true"],"gold":"false","raw_logits":[-1.9365234375,0.5263671875],"temperature":1.0,"type":"noul","source_id":"3435","source_row_hash":"e0d15beb0c523aab28095c68f8a36e66e1f0ce73e302239dc482aad749d2fc21","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.9365234375,0.5263671875],"calibrated_temperature":5.0},
+ {"id":"massive-3694-choice","group_id":"massive-alarm-ba350675a0496c49938e","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"f1f8330521612e78a2cde0c6656f2fba239789708588e8a80b2fc20b737e7845","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.71875,-4.5234375,-4.33984375],"temperature":1.0,"type":"choice","source_id":"3694","source_row_hash":"d1eb20fde014fb2b17be65874b712b3c63c6f59e4a2a59b4dcf2d38b52cce3b5","input_tokens":56,"truncated":false,"calibrated_raw_logits":[6.71875,-4.5234375,-4.33984375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3694-noul","group_id":"massive-alarm-ba350675a0496c49938e","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a62d58a5600dea097aa91ce2c7c14ed15568eee6c9bfd4c6389f8e279899de18","option_order":["false","true"],"gold":"true","raw_logits":[-2.154296875,1.7275390625],"temperature":1.0,"type":"noul","source_id":"3694","source_row_hash":"d1eb20fde014fb2b17be65874b712b3c63c6f59e4a2a59b4dcf2d38b52cce3b5","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.154296875,1.7275390625],"calibrated_temperature":5.0},
+ {"id":"massive-3695-choice","group_id":"massive-alarm-b52edfb5a71ea1dcee30","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"202cb568bede45bcb290436753573690741c6ce9aa2bd2488fea63edefc3c144","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.623046875,1.001953125,1.2080078125],"temperature":1.0,"type":"choice","source_id":"3695","source_row_hash":"5f84198d661ad46d3f9fd8c6e1da9ee2f4afe6fa1c77ec4b8ecbb1a18e582de6","input_tokens":52,"truncated":false,"calibrated_raw_logits":[1.623046875,1.001953125,1.2080078125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3695-noul","group_id":"massive-alarm-b52edfb5a71ea1dcee30","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"96938fc7026fb8a5848686387930d87db6f09856773963de5c13c5adc8ceaf17","option_order":["false","true"],"gold":"true","raw_logits":[-1.865234375,1.1982421875],"temperature":1.0,"type":"noul","source_id":"3695","source_row_hash":"5f84198d661ad46d3f9fd8c6e1da9ee2f4afe6fa1c77ec4b8ecbb1a18e582de6","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.865234375,1.1982421875],"calibrated_temperature":5.0},
+ {"id":"massive-3702-choice","group_id":"massive-alarm-d974ab158bb24355505f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7e3fea92fc16c779c9bda789dafacaa6be5cbe622e4f5560bfa3fdc82a74801e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[6.21484375,-4.44140625,-4.55859375],"temperature":1.0,"type":"choice","source_id":"3702","source_row_hash":"e618c546f5bea2ac0fe003c4c1dc9c54f82f1a3ab0ae22346f48031406b951c4","input_tokens":55,"truncated":false,"calibrated_raw_logits":[6.21484375,-4.44140625,-4.55859375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3702-noul","group_id":"massive-alarm-d974ab158bb24355505f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f74bf9955f1d89ea5586cb69e06a3ef816cc3cb2a5b7151d8273a1723839c8e2","option_order":["false","true"],"gold":"false","raw_logits":[-2.099609375,1.6357421875],"temperature":1.0,"type":"noul","source_id":"3702","source_row_hash":"e618c546f5bea2ac0fe003c4c1dc9c54f82f1a3ab0ae22346f48031406b951c4","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-2.099609375,1.6357421875],"calibrated_temperature":5.0},
+ {"id":"massive-3761-choice","group_id":"massive-alarm-906532f048bd9c89f168","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0599547b574a8679aa3e878e382da40df6e16c9d94f84a30b2d769fbe4b85de0","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-1.2392578125,-3.041015625,-0.41552734375],"temperature":1.0,"type":"choice","source_id":"3761","source_row_hash":"ca9d72e81a4b3f3d7e2235562d7d91f1f303f58748ed6e425cb54f84d73dd906","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.2392578125,-3.041015625,-0.41552734375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3761-noul","group_id":"massive-alarm-906532f048bd9c89f168","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4c43d963e2942b0e4b14bba53a12af4bfc6c6e9765a4a95b0b98a038cb326f38","option_order":["false","true"],"gold":"true","raw_logits":[-1.900390625,1.345703125],"temperature":1.0,"type":"noul","source_id":"3761","source_row_hash":"ca9d72e81a4b3f3d7e2235562d7d91f1f303f58748ed6e425cb54f84d73dd906","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.900390625,1.345703125],"calibrated_temperature":5.0},
+ {"id":"massive-3771-choice","group_id":"massive-alarm-c45435c75f999b77b736","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"25a21359b2b1ed7367720091eb1992eb1dad9c87afdf98e01b88ad12e7c8b6bd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.8818359375,3.08203125,-1.3564453125],"temperature":1.0,"type":"choice","source_id":"3771","source_row_hash":"2e4a7daa393af56cd536025155d462dbecd3c006ec8952136555253adea64973","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-0.8818359375,3.08203125,-1.3564453125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3771-noul","group_id":"massive-alarm-c45435c75f999b77b736","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"65677218b234245fd090491dad4d645564fa868f49e1c5ff60d2157d2d64e3d0","option_order":["false","true"],"gold":"false","raw_logits":[-1.66796875,0.7587890625],"temperature":1.0,"type":"noul","source_id":"3771","source_row_hash":"2e4a7daa393af56cd536025155d462dbecd3c006ec8952136555253adea64973","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.66796875,0.7587890625],"calibrated_temperature":5.0},
+ {"id":"massive-3829-choice","group_id":"massive-alarm-f60b61ceed3880a795df","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7b86cc5b86e8345e60c6261aabec21980fd9dfa33baff9d168197fd184a36dec","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-0.89892578125,-4.5,3.939453125],"temperature":1.0,"type":"choice","source_id":"3829","source_row_hash":"cbdd61b9e23a630e30d582423ee8353b28ecafc5c9b93cb0b7c7d327d1e9ac5c","input_tokens":54,"truncated":false,"calibrated_raw_logits":[-0.89892578125,-4.5,3.939453125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3829-noul","group_id":"massive-alarm-f60b61ceed3880a795df","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"591cecb6a8d39597824aa757e6a24856a9a87ab456991e3c95b4abf9246b2b5c","option_order":["false","true"],"gold":"false","raw_logits":[-2.30078125,0.60546875],"temperature":1.0,"type":"noul","source_id":"3829","source_row_hash":"cbdd61b9e23a630e30d582423ee8353b28ecafc5c9b93cb0b7c7d327d1e9ac5c","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.30078125,0.60546875],"calibrated_temperature":5.0},
+ {"id":"massive-3870-choice","group_id":"massive-alarm-cc3a484250d2efcc2043","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1fb7c02d1b63c62c515f630c6f6b1a190f806d8933fa807c965f98c542ea3734","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.9140625,0.95361328125,-0.422119140625],"temperature":1.0,"type":"choice","source_id":"3870","source_row_hash":"1eb812098d9c31e9b8c54e259a28eb60b0a677a306bbd79412ca17e0fd267621","input_tokens":56,"truncated":false,"calibrated_raw_logits":[2.9140625,0.95361328125,-0.422119140625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3870-noul","group_id":"massive-alarm-cc3a484250d2efcc2043","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4b85b5fb27bfdd2ab902c1aeebbb8681e8bba5cd4ab7ecb99502da2692776530","option_order":["false","true"],"gold":"false","raw_logits":[-2.1953125,1.1767578125],"temperature":1.0,"type":"noul","source_id":"3870","source_row_hash":"1eb812098d9c31e9b8c54e259a28eb60b0a677a306bbd79412ca17e0fd267621","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.1953125,1.1767578125],"calibrated_temperature":5.0},
+ {"id":"massive-3884-choice","group_id":"massive-alarm-021f9f46cfb6e0326d91","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"607244bfee195714e4c96cab20b3b0e4c30e225e8835abe4fc93cbf89fd30998","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.5439453125,-0.671875,-2.466796875],"temperature":1.0,"type":"choice","source_id":"3884","source_row_hash":"51b89a83ff9ab846f795d6b36a0f992d19c4db4c0a9f3ed29cdd233b99bfa59f","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.5439453125,-0.671875,-2.466796875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3884-noul","group_id":"massive-alarm-021f9f46cfb6e0326d91","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"0481094ed9ba36cd8746ca74f5c7144fd855684f2e353ce3d0763a903f4d024d","option_order":["false","true"],"gold":"false","raw_logits":[-1.958984375,0.455322265625],"temperature":1.0,"type":"noul","source_id":"3884","source_row_hash":"51b89a83ff9ab846f795d6b36a0f992d19c4db4c0a9f3ed29cdd233b99bfa59f","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.958984375,0.455322265625],"calibrated_temperature":5.0},
+ {"id":"massive-3886-choice","group_id":"massive-alarm-661b37ed9fdd982c94bf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"27c0ff2ef671d83abaebeff897b97cd739dc1770c284ea8bd367da824127f203","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.759765625,-1.4873046875,0.60546875],"temperature":1.0,"type":"choice","source_id":"3886","source_row_hash":"d6664d601ddccd6310963193b24469d0e9d111c0a7f3f23bf551b85d96113ccf","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-1.759765625,-1.4873046875,0.60546875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3886-noul","group_id":"massive-alarm-661b37ed9fdd982c94bf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"1fc4b67408e23e34752d030fee7275c22033746a1eaecef7e6277bffd34047f8","option_order":["false","true"],"gold":"false","raw_logits":[-1.8095703125,1.1162109375],"temperature":1.0,"type":"noul","source_id":"3886","source_row_hash":"d6664d601ddccd6310963193b24469d0e9d111c0a7f3f23bf551b85d96113ccf","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.8095703125,1.1162109375],"calibrated_temperature":5.0},
+ {"id":"massive-3986-choice","group_id":"massive-alarm-f2693fd957a1a13798ca","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"d542a5fa84c401daab24a69a30c193f6a2a09bf9263e9bc7a51bf617211fd54c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.13671875,-3.943359375,-2.3984375],"temperature":1.0,"type":"choice","source_id":"3986","source_row_hash":"a52237c94225b1553a4b9b64b99a6de6c5747841054a315e869690c5c40c0042","input_tokens":51,"truncated":false,"calibrated_raw_logits":[6.13671875,-3.943359375,-2.3984375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-3986-noul","group_id":"massive-alarm-f2693fd957a1a13798ca","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"b8b2c739fdb3846f88484b30634ef774a8bf95921c0db9777d23d18c672698ad","option_order":["false","true"],"gold":"true","raw_logits":[-1.990234375,1.25390625],"temperature":1.0,"type":"noul","source_id":"3986","source_row_hash":"a52237c94225b1553a4b9b64b99a6de6c5747841054a315e869690c5c40c0042","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.990234375,1.25390625],"calibrated_temperature":5.0},
+ {"id":"massive-399-choice","group_id":"massive-alarm-5064cd0ab19f1ced3b63","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7f7f324de0a5abcdfb147204c078ff9f5e6627709efd66a92c90f2f11eda84c8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.845703125,-2.98046875,2.666015625],"temperature":1.0,"type":"choice","source_id":"399","source_row_hash":"1c0573b1dc35d2d56f63d762ce3123e67d36e4fe7a90b405693688e9f3c0fb74","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-2.845703125,-2.98046875,2.666015625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-399-noul","group_id":"massive-alarm-5064cd0ab19f1ced3b63","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"3f557ad186f1a17d7172adf20667e35f775f5335dbae9db5de749c0a85692384","option_order":["false","true"],"gold":"false","raw_logits":[-1.7451171875,0.732421875],"temperature":1.0,"type":"noul","source_id":"399","source_row_hash":"1c0573b1dc35d2d56f63d762ce3123e67d36e4fe7a90b405693688e9f3c0fb74","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.7451171875,0.732421875],"calibrated_temperature":5.0},
+ {"id":"massive-4017-choice","group_id":"massive-alarm-7bab44deabd44eeba1d4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c6aa025ee3f6243b63d4112e57528c1a0d09804b794a47a52f827cd845c22365","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.89453125,-3.54296875,1.318359375],"temperature":1.0,"type":"choice","source_id":"4017","source_row_hash":"b950ef4e3cfec6148b5f70b8cc2699e018e3443f68871089bb3b3bac77a915b3","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-2.89453125,-3.54296875,1.318359375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4017-noul","group_id":"massive-alarm-7bab44deabd44eeba1d4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"91e79a9efc33926738ab1aca86bf6652646cab200330672ca9fdd14ff4a13f95","option_order":["false","true"],"gold":"false","raw_logits":[-2.76953125,0.5654296875],"temperature":1.0,"type":"noul","source_id":"4017","source_row_hash":"b950ef4e3cfec6148b5f70b8cc2699e018e3443f68871089bb3b3bac77a915b3","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.76953125,0.5654296875],"calibrated_temperature":5.0},
+ {"id":"massive-4081-choice","group_id":"massive-alarm-0fbfcc011e34434db270","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"96a5e44b26bf23a409304bc33c251790f9f8bebf6fc652e9b1b60e8941003ce4","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-1.23828125,-2.173828125,-0.37451171875],"temperature":1.0,"type":"choice","source_id":"4081","source_row_hash":"ed08495bac8110f45e50260d015d1d52615f67a00d5c47ffcd960745bd32f98f","input_tokens":60,"truncated":false,"calibrated_raw_logits":[-1.23828125,-2.173828125,-0.37451171875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4081-noul","group_id":"massive-alarm-0fbfcc011e34434db270","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"db72d9a98ad646aab2a2727af0b22a3dd95d9afb6b7053284680c34d1e22848b","option_order":["false","true"],"gold":"true","raw_logits":[-1.77734375,0.5439453125],"temperature":1.0,"type":"noul","source_id":"4081","source_row_hash":"ed08495bac8110f45e50260d015d1d52615f67a00d5c47ffcd960745bd32f98f","input_tokens":55,"truncated":false,"calibrated_raw_logits":[-1.77734375,0.5439453125],"calibrated_temperature":5.0},
+ {"id":"massive-4121-choice","group_id":"massive-alarm-06d7dae0b6a5251a04b7","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"856f98d8094e873994e21182bb63f23d0513e3176c63e27cd351ac91cb60e697","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.19140625,-1.458984375,-1.919921875],"temperature":1.0,"type":"choice","source_id":"4121","source_row_hash":"905a75e6d0482805183ad0e81410418cc470ec3cb4cfabe1d76f9f30a337c809","input_tokens":52,"truncated":false,"calibrated_raw_logits":[4.19140625,-1.458984375,-1.919921875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4121-noul","group_id":"massive-alarm-06d7dae0b6a5251a04b7","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a09915d1ead2a4e4ef77bcfb9cfdc28ff092aab9317064470bf2d91fa130fcf9","option_order":["false","true"],"gold":"false","raw_logits":[-1.306640625,1.3212890625],"temperature":1.0,"type":"noul","source_id":"4121","source_row_hash":"905a75e6d0482805183ad0e81410418cc470ec3cb4cfabe1d76f9f30a337c809","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.306640625,1.3212890625],"calibrated_temperature":5.0},
+ {"id":"massive-4130-choice","group_id":"massive-alarm-e36c4391c2490fb6ea69","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"9e20290617b2c369552819c32b78e1c50dadbb853aaa45e3fbaa353b5ac9a1e1","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.81005859375,-0.61181640625,-0.282958984375],"temperature":1.0,"type":"choice","source_id":"4130","source_row_hash":"45b0bb6e019be688e56cc7d5540bc815c6cc6845e6f17adfa152bee81fedec9a","input_tokens":55,"truncated":false,"calibrated_raw_logits":[0.81005859375,-0.61181640625,-0.282958984375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4130-noul","group_id":"massive-alarm-e36c4391c2490fb6ea69","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ca8193f9310a7214cfdea09f2f789252a72d64e3aa770830d4d71eb920257fc9","option_order":["false","true"],"gold":"false","raw_logits":[-0.470458984375,0.58203125],"temperature":1.0,"type":"noul","source_id":"4130","source_row_hash":"45b0bb6e019be688e56cc7d5540bc815c6cc6845e6f17adfa152bee81fedec9a","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-0.470458984375,0.58203125],"calibrated_temperature":5.0},
+ {"id":"massive-4145-choice","group_id":"massive-alarm-193674ca7cd9463d8724","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"d3989f582d7f1cb6a535ec95163ad30df3e578d1b8610a327b515b423f69c862","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.52734375,-3.193359375,-2.767578125],"temperature":1.0,"type":"choice","source_id":"4145","source_row_hash":"4e06467ae4327b6843c2fd71dea717e128c9c5e776c173ba67f85c47f7caadb4","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.52734375,-3.193359375,-2.767578125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4145-noul","group_id":"massive-alarm-193674ca7cd9463d8724","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"eeda34b2c3c8364306cca7d520527d07f47fb1da66cbbb5627cff7d9a0127a78","option_order":["false","true"],"gold":"false","raw_logits":[-2.39453125,1.0224609375],"temperature":1.0,"type":"noul","source_id":"4145","source_row_hash":"4e06467ae4327b6843c2fd71dea717e128c9c5e776c173ba67f85c47f7caadb4","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.39453125,1.0224609375],"calibrated_temperature":5.0},
+ {"id":"massive-4319-choice","group_id":"massive-alarm-3a6e1e04dfdb2826f3e6","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"59054d0457ed9b0e150ca8e4946d327901f4ca6957ecfd3ece87322ba3eac9a5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.406494140625,-2.6796875,-0.0236053466796875],"temperature":1.0,"type":"choice","source_id":"4319","source_row_hash":"f81d298a66e00e967f0e503432b25d08e7c32fae30fc39b9d68651d1c7e22915","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.406494140625,-2.6796875,-0.0236053466796875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4319-noul","group_id":"massive-alarm-3a6e1e04dfdb2826f3e6","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"aeeb6f93631dbebaa9ddf60f34c6b43d9dc7a176e9667f7b3efb94eb3d775fa2","option_order":["false","true"],"gold":"true","raw_logits":[-1.3876953125,0.8994140625],"temperature":1.0,"type":"noul","source_id":"4319","source_row_hash":"f81d298a66e00e967f0e503432b25d08e7c32fae30fc39b9d68651d1c7e22915","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.3876953125,0.8994140625],"calibrated_temperature":5.0},
+ {"id":"massive-4375-choice","group_id":"massive-alarm-d0710f72b80d46f21bd2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"91da7137e905fecc5e2cdbd252323de33015b18aefbddd160effc10409a0a0ec","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.3505859375,0.5517578125,1.2265625],"temperature":1.0,"type":"choice","source_id":"4375","source_row_hash":"2d5824d9d29c077e250c5812fe3e84908d5bf837ce7bc903743164f8d11624dd","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.3505859375,0.5517578125,1.2265625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4375-noul","group_id":"massive-alarm-d0710f72b80d46f21bd2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fbbeae282b0e1b151a5b50a77f85d81ef6c08f60ec2811a7a2a773c2e789e581","option_order":["false","true"],"gold":"false","raw_logits":[-1.2119140625,0.95458984375],"temperature":1.0,"type":"noul","source_id":"4375","source_row_hash":"2d5824d9d29c077e250c5812fe3e84908d5bf837ce7bc903743164f8d11624dd","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.2119140625,0.95458984375],"calibrated_temperature":5.0},
+ {"id":"massive-4654-choice","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"77cebc8da0a8189f2262bdd3a39f31d61fc360a7956bcd0f65463b08dc21bf4f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.2578125,-2.5625,-2.025390625],"temperature":1.0,"type":"choice","source_id":"4654","source_row_hash":"b7e67a9331d7f20bc014914c88b8be8ba51ce9bf981915ffe1d2138b91b6a01e","input_tokens":53,"truncated":false,"calibrated_raw_logits":[4.2578125,-2.5625,-2.025390625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4654-noul","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"456e9b4d2821b1b3662f84db099f4a9923f03141ad9b289aa4cb01e7b434fac1","option_order":["false","true"],"gold":"true","raw_logits":[-1.9423828125,1.6611328125],"temperature":1.0,"type":"noul","source_id":"4654","source_row_hash":"b7e67a9331d7f20bc014914c88b8be8ba51ce9bf981915ffe1d2138b91b6a01e","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.9423828125,1.6611328125],"calibrated_temperature":5.0},
+ {"id":"massive-4673-choice","group_id":"massive-alarm-84fb06bfa21c395f5aaf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"bbc99315926deca3de03ece3f5e4184df6aa7262d5442d04979bf9dbf2a8cbd9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-2.296875,3.982421875,-0.8330078125],"temperature":1.0,"type":"choice","source_id":"4673","source_row_hash":"b9cefc1b3d178f9664ac742c0d5fd6c3541b41453ce33986bd45ded2e5baa606","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.296875,3.982421875,-0.8330078125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4673-noul","group_id":"massive-alarm-84fb06bfa21c395f5aaf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"29dce85d99aa74e32a858b60eb0de1ff518b8819f65f192a048517fb9a9d798b","option_order":["false","true"],"gold":"false","raw_logits":[-1.3115234375,0.343994140625],"temperature":1.0,"type":"noul","source_id":"4673","source_row_hash":"b9cefc1b3d178f9664ac742c0d5fd6c3541b41453ce33986bd45ded2e5baa606","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.3115234375,0.343994140625],"calibrated_temperature":5.0},
+ {"id":"massive-4712-choice","group_id":"massive-alarm-2c6ac9f498036d649d66","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"934f5e684a6cbf814f1536b5b6b017245af8638cf21ae0290e64f1a32bd58c4c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.005859375,-3.384765625,-0.11474609375],"temperature":1.0,"type":"choice","source_id":"4712","source_row_hash":"4386c04043d8c28ace7ee145b5b749efc7ff6ad5efb0664e6ffb4a359864a2ba","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.005859375,-3.384765625,-0.11474609375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4712-noul","group_id":"massive-alarm-2c6ac9f498036d649d66","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"17794909495f150f505eb1cd6a46d9bcaa7531479f2f9d0165ee8b8193811d67","option_order":["false","true"],"gold":"false","raw_logits":[-1.8779296875,0.9365234375],"temperature":1.0,"type":"noul","source_id":"4712","source_row_hash":"4386c04043d8c28ace7ee145b5b749efc7ff6ad5efb0664e6ffb4a359864a2ba","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.8779296875,0.9365234375],"calibrated_temperature":5.0},
+ {"id":"massive-4776-choice","group_id":"massive-alarm-44615da0c0274196323c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"5fec435b1670ec5cf5fb855e00e31fce4474fe0e0fc3105fcb24aeceaeaeb6b5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.238037109375,-1.462890625,-1.5185546875],"temperature":1.0,"type":"choice","source_id":"4776","source_row_hash":"81079ab4bf8ea6953a66c7c91e9a60e6fe22b3e698c003ffa5c411c0da2afe20","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.238037109375,-1.462890625,-1.5185546875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4776-noul","group_id":"massive-alarm-44615da0c0274196323c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"0fc3aba2af6c95f1c6f27d8149e96b826c80dd710f2427776cd09b5660c919cf","option_order":["false","true"],"gold":"false","raw_logits":[-1.685546875,1.1787109375],"temperature":1.0,"type":"noul","source_id":"4776","source_row_hash":"81079ab4bf8ea6953a66c7c91e9a60e6fe22b3e698c003ffa5c411c0da2afe20","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.685546875,1.1787109375],"calibrated_temperature":5.0},
+ {"id":"massive-4848-choice","group_id":"massive-alarm-a708c3be70feb0cd7499","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7522ec4a0488ba7e56fa08b528c3df20f40223c6894ac0dc46d8bd184912bcb3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.416015625,-2.8828125,-0.599609375],"temperature":1.0,"type":"choice","source_id":"4848","source_row_hash":"b0b8d755300f645d4383b66847a7614fea92b600c73c2cf5542dff3eca2e27bd","input_tokens":50,"truncated":false,"calibrated_raw_logits":[0.416015625,-2.8828125,-0.599609375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4848-noul","group_id":"massive-alarm-a708c3be70feb0cd7499","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"956ba45262eca715bf5c90a06427ede7e3364a644ee12016e6da052866a0845b","option_order":["false","true"],"gold":"true","raw_logits":[-1.8486328125,0.95556640625],"temperature":1.0,"type":"noul","source_id":"4848","source_row_hash":"b0b8d755300f645d4383b66847a7614fea92b600c73c2cf5542dff3eca2e27bd","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.8486328125,0.95556640625],"calibrated_temperature":5.0},
+ {"id":"massive-4851-choice","group_id":"massive-alarm-0fe1a1db9c374422445f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"fdeb0308d07e167e345d0809a6395c4969a801c33d5f80ca07ffed816020bbcb","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.86328125,-3.0859375,-2.537109375],"temperature":1.0,"type":"choice","source_id":"4851","source_row_hash":"9fe5af17055439dfa1b1610ca5fb2973b66459a1e0614af0e2452d2ce7a1e77c","input_tokens":50,"truncated":false,"calibrated_raw_logits":[4.86328125,-3.0859375,-2.537109375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4851-noul","group_id":"massive-alarm-0fe1a1db9c374422445f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"133b6950426a63d754fc346a4d151cbafe939b8e37e37fc290c101f6dcc77a0f","option_order":["false","true"],"gold":"false","raw_logits":[-2.02734375,1.529296875],"temperature":1.0,"type":"noul","source_id":"4851","source_row_hash":"9fe5af17055439dfa1b1610ca5fb2973b66459a1e0614af0e2452d2ce7a1e77c","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.02734375,1.529296875],"calibrated_temperature":5.0},
+ {"id":"massive-4921-choice","group_id":"massive-alarm-5c308695c6a8b9f486cd","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"72d43a129cdaffcad2fbed561838f632af3ac96bca5743580ba0401a3665095e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.8505859375,-3.353515625,1.2998046875],"temperature":1.0,"type":"choice","source_id":"4921","source_row_hash":"2c2b240390fb93b6986489979cea8c3377a2c30ff81aab5abde369bc1a921d66","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.8505859375,-3.353515625,1.2998046875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4921-noul","group_id":"massive-alarm-5c308695c6a8b9f486cd","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4437a78e7f0d9e73db14372aa4f2b1cb1ee2f75571651b05dcb50bef8dac6af6","option_order":["false","true"],"gold":"false","raw_logits":[-2.416015625,0.92822265625],"temperature":1.0,"type":"noul","source_id":"4921","source_row_hash":"2c2b240390fb93b6986489979cea8c3377a2c30ff81aab5abde369bc1a921d66","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.416015625,0.92822265625],"calibrated_temperature":5.0},
+ {"id":"massive-4930-choice","group_id":"massive-alarm-54d0fa25a96d8072f5b5","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cb8fffe4d7f15dd6507c6a0d7fa59fd180ba95cd1fb191f63aa8c1fe6aaf1b3b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.9716796875,-3.0546875,-0.66943359375],"temperature":1.0,"type":"choice","source_id":"4930","source_row_hash":"fd5035b32bf0e5c2a53512a0663af50295468117efba589c8277e039fbaace04","input_tokens":54,"truncated":false,"calibrated_raw_logits":[1.9716796875,-3.0546875,-0.66943359375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-4930-noul","group_id":"massive-alarm-54d0fa25a96d8072f5b5","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"5f559d4cf48d8ed51abf809f544bc6ed69e826262d2c3aa6a1980713386add37","option_order":["false","true"],"gold":"true","raw_logits":[-2.16796875,1.2373046875],"temperature":1.0,"type":"noul","source_id":"4930","source_row_hash":"fd5035b32bf0e5c2a53512a0663af50295468117efba589c8277e039fbaace04","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.16796875,1.2373046875],"calibrated_temperature":5.0},
+ {"id":"massive-5006-choice","group_id":"massive-alarm-36c1ee0067291e81dfe0","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c0cf2686b16917d7ab80250b7f585134b8169f9fdc84e27e9c480875e60f94a2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.9462890625,1.2626953125,-2.009765625],"temperature":1.0,"type":"choice","source_id":"5006","source_row_hash":"c23192b2816c3f5f07c27870d69fdf383d4bc79de8202365cce80d52d65ef321","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.9462890625,1.2626953125,-2.009765625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5006-noul","group_id":"massive-alarm-36c1ee0067291e81dfe0","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4a7e732be2948711919ca7d7eafa8af5c50381df15405ffb995261ae054c1fd5","option_order":["false","true"],"gold":"false","raw_logits":[-1.046875,0.1993408203125],"temperature":1.0,"type":"noul","source_id":"5006","source_row_hash":"c23192b2816c3f5f07c27870d69fdf383d4bc79de8202365cce80d52d65ef321","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.046875,0.1993408203125],"calibrated_temperature":5.0},
+ {"id":"massive-5043-choice","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"9209f20ab1cba0e22314b3adfb3a5459bc888eb018e2160a81d37579238baa7a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.076171875,-1.1318359375,-0.78173828125],"temperature":1.0,"type":"choice","source_id":"5043","source_row_hash":"837ce3dfa54a2caaa3feddb1add4699979a13624fa240aedd9d48eb841d7c3e3","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.076171875,-1.1318359375,-0.78173828125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5043-noul","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"24c1573dfcf63931fc5ed541e2ad4eb9e2eac99472bee9d3f85e0b3c80b7ee59","option_order":["false","true"],"gold":"true","raw_logits":[-2.505859375,1.9638671875],"temperature":1.0,"type":"noul","source_id":"5043","source_row_hash":"837ce3dfa54a2caaa3feddb1add4699979a13624fa240aedd9d48eb841d7c3e3","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.505859375,1.9638671875],"calibrated_temperature":5.0},
+ {"id":"massive-5051-choice","group_id":"massive-alarm-16e090610da060f7511f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"e5fbc20b0a5aad3f0006a418d26781efe4e3d935dd3ae44b4fcc7219d7760b64","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.53125,-3.443359375,-0.4462890625],"temperature":1.0,"type":"choice","source_id":"5051","source_row_hash":"f3bbc1f439396e98e81c6c680dcba8b9fbe57a12e24a67d449622e4042bb9671","input_tokens":50,"truncated":false,"calibrated_raw_logits":[0.53125,-3.443359375,-0.4462890625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5051-noul","group_id":"massive-alarm-16e090610da060f7511f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"426104e055a8c3026abeb0f10d97c3466a4cfacf4037b56e3596670dc21c7368","option_order":["false","true"],"gold":"true","raw_logits":[-2.826171875,1.2958984375],"temperature":1.0,"type":"noul","source_id":"5051","source_row_hash":"f3bbc1f439396e98e81c6c680dcba8b9fbe57a12e24a67d449622e4042bb9671","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.826171875,1.2958984375],"calibrated_temperature":5.0},
+ {"id":"massive-508-choice","group_id":"massive-alarm-41a421223f2457d4ae35","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2fc92d0e251aa21634ffc2b072bab5a2ca2442251f5e82ccf62197d9b4d9194d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.478515625,-3.337890625,-3.78515625],"temperature":1.0,"type":"choice","source_id":"508","source_row_hash":"71ecc0e574fd987cffcdeee6026d2de621df0225d0e6be68cf09db826a9bcd9b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[3.478515625,-3.337890625,-3.78515625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-508-noul","group_id":"massive-alarm-41a421223f2457d4ae35","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"61946bd8d19af979160d933e5b004a05b212d1f720b0866c522637f99f21ea7b","option_order":["false","true"],"gold":"false","raw_logits":[-2.60546875,0.544921875],"temperature":1.0,"type":"noul","source_id":"508","source_row_hash":"71ecc0e574fd987cffcdeee6026d2de621df0225d0e6be68cf09db826a9bcd9b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.60546875,0.544921875],"calibrated_temperature":5.0},
+ {"id":"massive-5082-choice","group_id":"massive-alarm-dff7933fbae6d32c0fc9","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"65e59f1014104082d5ba399967ee15984ae0fa87f363cd154ec0f46b5075265f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.66650390625,-3.208984375,0.7880859375],"temperature":1.0,"type":"choice","source_id":"5082","source_row_hash":"eb3b740e165f04e6c1419581c82a67b0e075496a1bb875d944161266c0f98fd6","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.66650390625,-3.208984375,0.7880859375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5082-noul","group_id":"massive-alarm-dff7933fbae6d32c0fc9","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4d65d5fa341600065d5f523f2936bd98d886c1b3ddd05ef2f97bca7116d2413b","option_order":["false","true"],"gold":"true","raw_logits":[-2.341796875,1.470703125],"temperature":1.0,"type":"noul","source_id":"5082","source_row_hash":"eb3b740e165f04e6c1419581c82a67b0e075496a1bb875d944161266c0f98fd6","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.341796875,1.470703125],"calibrated_temperature":5.0},
+ {"id":"massive-5103-choice","group_id":"massive-alarm-a0f523ed66d86dd66928","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1474862757e271b75a651b0a5c023ae431e3471fbd2b83e930cc010904b7f605","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.208984375,-4.21484375,2.35546875],"temperature":1.0,"type":"choice","source_id":"5103","source_row_hash":"5e4324ac53ee00213541d5386e0f8f520d4b7930fe26bd416545ed9b3681e81c","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-3.208984375,-4.21484375,2.35546875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5103-noul","group_id":"massive-alarm-a0f523ed66d86dd66928","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7cc1683f408cddf9993f59351a691ac692dcd2f2be18820e5d088f5d607d95e5","option_order":["false","true"],"gold":"false","raw_logits":[-1.99609375,-0.01267242431640625],"temperature":1.0,"type":"noul","source_id":"5103","source_row_hash":"5e4324ac53ee00213541d5386e0f8f520d4b7930fe26bd416545ed9b3681e81c","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.99609375,-0.01267242431640625],"calibrated_temperature":5.0},
+ {"id":"massive-5116-choice","group_id":"massive-alarm-eef8eb3baf2d9be9f032","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ff5afaf8010c0ec76afa02463532a835bf618957bb435b423b572a34d1f72b9b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.78125,-1.4833984375,2.59765625],"temperature":1.0,"type":"choice","source_id":"5116","source_row_hash":"81e95101c290ce21984ebbc4b02d5abbd6cc8ca4e784b97182671444871de55b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.78125,-1.4833984375,2.59765625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5116-noul","group_id":"massive-alarm-eef8eb3baf2d9be9f032","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e55857f2a894f6be93794ed4131d1b24b3d1e25bd0fb009c6bed1068297afa36","option_order":["false","true"],"gold":"false","raw_logits":[-1.5234375,-0.00075531005859375],"temperature":1.0,"type":"noul","source_id":"5116","source_row_hash":"81e95101c290ce21984ebbc4b02d5abbd6cc8ca4e784b97182671444871de55b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.5234375,-0.00075531005859375],"calibrated_temperature":5.0},
+ {"id":"massive-5122-choice","group_id":"massive-alarm-a7c3622f745708add50b","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"b8cecf6e8d365c8cca3910552fdf85dce9653479d4d50a00d9437b329657fca3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.7109375,-2.01953125,-0.6689453125],"temperature":1.0,"type":"choice","source_id":"5122","source_row_hash":"60429c8fba1d01a9b236a868780f5aab91fdfdfd7d7193d219e29dda4ff40f7b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.7109375,-2.01953125,-0.6689453125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5122-noul","group_id":"massive-alarm-a7c3622f745708add50b","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"3c3cccaa225b3031122a45bab64f407013dfce7331773fbb68584397b853f187","option_order":["false","true"],"gold":"false","raw_logits":[-1.8408203125,0.66845703125],"temperature":1.0,"type":"noul","source_id":"5122","source_row_hash":"60429c8fba1d01a9b236a868780f5aab91fdfdfd7d7193d219e29dda4ff40f7b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.8408203125,0.66845703125],"calibrated_temperature":5.0},
+ {"id":"massive-5292-choice","group_id":"massive-alarm-7479d40399b64ecb695e","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6b263f0198363465a79d92000b9f4549cac9e8fdec4c19381966d4af03853c2e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.376953125,-4.47265625,-4.5078125],"temperature":1.0,"type":"choice","source_id":"5292","source_row_hash":"813c05e407bea752a81f88e33c115e1e14729209d2ef4bb63bb371d3d394049c","input_tokens":50,"truncated":false,"calibrated_raw_logits":[3.376953125,-4.47265625,-4.5078125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5292-noul","group_id":"massive-alarm-7479d40399b64ecb695e","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"2bae5052bc8cf5c358997f4b0612f1f6d00852a3884983f2e3b12c9cacfa0e9d","option_order":["false","true"],"gold":"true","raw_logits":[-3.390625,1.625],"temperature":1.0,"type":"noul","source_id":"5292","source_row_hash":"813c05e407bea752a81f88e33c115e1e14729209d2ef4bb63bb371d3d394049c","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-3.390625,1.625],"calibrated_temperature":5.0},
+ {"id":"massive-533-choice","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"99a85134b83e2e949a3e0c54f4e95c4e40f8bf0a6b5f395df71cd8a3e029bdec","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.0703125,-4.25,-3.58203125],"temperature":1.0,"type":"choice","source_id":"533","source_row_hash":"eca3b5ac906fb8a35343d18eb18cad818f1db7770f69dcac2648292b16b0ffbc","input_tokens":52,"truncated":false,"calibrated_raw_logits":[6.0703125,-4.25,-3.58203125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-533-noul","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e0c12ffe2cd33b8d0a94ddaa9dab71028665b02ed35d0295366709686a915d67","option_order":["false","true"],"gold":"true","raw_logits":[-2.1328125,1.4267578125],"temperature":1.0,"type":"noul","source_id":"533","source_row_hash":"eca3b5ac906fb8a35343d18eb18cad818f1db7770f69dcac2648292b16b0ffbc","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.1328125,1.4267578125],"calibrated_temperature":5.0},
+ {"id":"massive-5365-choice","group_id":"massive-alarm-d0dbaab04348ee02b2e4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7dae85cf680af0945a163fff2386a84f8d4b0fe67d9c437a5527258473b91288","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.703125,-3.419921875,-1.2119140625],"temperature":1.0,"type":"choice","source_id":"5365","source_row_hash":"3e646d241a20c6d18260b38f64e3d5b4045894baca11ab213b09d65fc04fd4d5","input_tokens":53,"truncated":false,"calibrated_raw_logits":[4.703125,-3.419921875,-1.2119140625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5365-noul","group_id":"massive-alarm-d0dbaab04348ee02b2e4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a3d43d9d9cfd2a79d27de7f606676fe8fd10e9ede4c75b655407327e83fe27ab","option_order":["false","true"],"gold":"true","raw_logits":[-1.537109375,1.2392578125],"temperature":1.0,"type":"noul","source_id":"5365","source_row_hash":"3e646d241a20c6d18260b38f64e3d5b4045894baca11ab213b09d65fc04fd4d5","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.537109375,1.2392578125],"calibrated_temperature":5.0},
+ {"id":"massive-5475-choice","group_id":"massive-alarm-e7a25c599f07a27b07a8","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6ce27768ae10f0206a17038e7f7e3bd69bc0584f9ed44947f46347b4423df338","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.0107421875,-2.423828125,0.060760498046875],"temperature":1.0,"type":"choice","source_id":"5475","source_row_hash":"dc6c0165e59d0b35c1731ad73656915fbd6291deba80299bc358fc944a72978c","input_tokens":52,"truncated":false,"calibrated_raw_logits":[1.0107421875,-2.423828125,0.060760498046875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5475-noul","group_id":"massive-alarm-e7a25c599f07a27b07a8","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"264847e9ef6d6c6f317ea9dea8f6f8a17935af1b5fc265f2802d522dbf5af17d","option_order":["false","true"],"gold":"true","raw_logits":[-2.3203125,1.548828125],"temperature":1.0,"type":"noul","source_id":"5475","source_row_hash":"dc6c0165e59d0b35c1731ad73656915fbd6291deba80299bc358fc944a72978c","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.3203125,1.548828125],"calibrated_temperature":5.0},
+ {"id":"massive-5568-choice","group_id":"massive-alarm-9d8d8e3f6fb99e20fed8","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8af7c9ac22d839179599a532150c0575129bb3db346ed4fcaf23996f2727c981","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.759765625,-1.7978515625,-1.2451171875],"temperature":1.0,"type":"choice","source_id":"5568","source_row_hash":"e2678a9b8f76e5325e37853048a1af7ef56128900590959c918238a2a0a87f67","input_tokens":54,"truncated":false,"calibrated_raw_logits":[3.759765625,-1.7978515625,-1.2451171875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5568-noul","group_id":"massive-alarm-9d8d8e3f6fb99e20fed8","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"6bb9d7bc2c8f0080617c2a6a730f73b45ea25f3c7a0e4fafdef8dbbfc6067d91","option_order":["false","true"],"gold":"true","raw_logits":[-2.484375,1.4775390625],"temperature":1.0,"type":"noul","source_id":"5568","source_row_hash":"e2678a9b8f76e5325e37853048a1af7ef56128900590959c918238a2a0a87f67","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.484375,1.4775390625],"calibrated_temperature":5.0},
+ {"id":"massive-560-choice","group_id":"massive-alarm-3914f2c167cd3bc916a9","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8b4bacc83095f32adfbe08922536f8e8f085e1c778c9ed57ed2a65ce83ee6ff5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.1005859375,-3.044921875,1.4658203125],"temperature":1.0,"type":"choice","source_id":"560","source_row_hash":"198c7a67f853f0a20207e974e8b9f5c52eb85e496988929af5c4cd1b2ba14b7a","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.1005859375,-3.044921875,1.4658203125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-560-noul","group_id":"massive-alarm-3914f2c167cd3bc916a9","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"de516021eb9a4d76b704b912aab4923d0355631dc535f055c7f97319615c832e","option_order":["false","true"],"gold":"false","raw_logits":[-2.03515625,-0.5439453125],"temperature":1.0,"type":"noul","source_id":"560","source_row_hash":"198c7a67f853f0a20207e974e8b9f5c52eb85e496988929af5c4cd1b2ba14b7a","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.03515625,-0.5439453125],"calibrated_temperature":5.0},
+ {"id":"massive-5623-choice","group_id":"massive-alarm-ee24ef5af4d9c0920de7","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2a8568251fdcde56fbf31c4ee89e239a704fee954543e6ef90286e089bda6c6f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.417236328125,-1.1318359375,0.07928466796875],"temperature":1.0,"type":"choice","source_id":"5623","source_row_hash":"d74f814f601c02c5df4a0b5f23516c6bf156ad8051a476422eb6fdea1660ae5e","input_tokens":50,"truncated":false,"calibrated_raw_logits":[0.417236328125,-1.1318359375,0.07928466796875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5623-noul","group_id":"massive-alarm-ee24ef5af4d9c0920de7","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"6af842101731694a1cd8e31608ddc4a5a0e95ab625ae4ebe22e246073a8f6700","option_order":["false","true"],"gold":"true","raw_logits":[-1.201171875,1.654296875],"temperature":1.0,"type":"noul","source_id":"5623","source_row_hash":"d74f814f601c02c5df4a0b5f23516c6bf156ad8051a476422eb6fdea1660ae5e","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.201171875,1.654296875],"calibrated_temperature":5.0},
+ {"id":"massive-5635-choice","group_id":"massive-alarm-4beeb5931613d60f1341","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cc5c973ab31e3d3df68e2c8c5178bc62cccae85d6a38bef00637b677f9b3e2e0","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.031829833984375,-2.861328125,-0.1290283203125],"temperature":1.0,"type":"choice","source_id":"5635","source_row_hash":"40eb075b4d858f1411258909d4d77625f091cbc21c25f9c1484232129528cb67","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.031829833984375,-2.861328125,-0.1290283203125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5635-noul","group_id":"massive-alarm-4beeb5931613d60f1341","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"91e98a9d1309dfdefc5644e876e4141869104d2d08ee388a23a75d4dd1689a6a","option_order":["false","true"],"gold":"true","raw_logits":[-1.052734375,1.1396484375],"temperature":1.0,"type":"noul","source_id":"5635","source_row_hash":"40eb075b4d858f1411258909d4d77625f091cbc21c25f9c1484232129528cb67","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.052734375,1.1396484375],"calibrated_temperature":5.0},
+ {"id":"massive-59-choice","group_id":"massive-alarm-2276a9a439ac95d3573f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"89b470c0d2ae0a5f39585af5b6ff60255952f2dd6dc6ec3e4a84b0ae006b2ecc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-0.04351806640625,-2.509765625,1.7255859375],"temperature":1.0,"type":"choice","source_id":"59","source_row_hash":"cb6899117f503f233e7b525dd5cd06511f88ace9a9caaee665d48c413113a6d9","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.04351806640625,-2.509765625,1.7255859375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-59-noul","group_id":"massive-alarm-2276a9a439ac95d3573f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"872c7a88b6ca28b3a9194a3e4678c850b38ef642762b3fd454545679f04001f5","option_order":["false","true"],"gold":"false","raw_logits":[-1.94140625,0.13916015625],"temperature":1.0,"type":"noul","source_id":"59","source_row_hash":"cb6899117f503f233e7b525dd5cd06511f88ace9a9caaee665d48c413113a6d9","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.94140625,0.13916015625],"calibrated_temperature":5.0},
+ {"id":"massive-5902-choice","group_id":"massive-alarm-fb6df7f77542683a3ff2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7fbcd8af2c716bdd66961270eacaa6a96622bbcf1372261fd4a96f855553ec8d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.564453125,-0.93017578125,0.293212890625],"temperature":1.0,"type":"choice","source_id":"5902","source_row_hash":"d1fd61eb591646e537e465ded07ee0a48eeb5db31db81e0c4407cac197d93495","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-0.564453125,-0.93017578125,0.293212890625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5902-noul","group_id":"massive-alarm-fb6df7f77542683a3ff2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"c4be7c11de56ddc45a0601df7c80095583e9c5a7b51a97188443ff20f73e590e","option_order":["false","true"],"gold":"false","raw_logits":[-1.3876953125,1.02734375],"temperature":1.0,"type":"noul","source_id":"5902","source_row_hash":"d1fd61eb591646e537e465ded07ee0a48eeb5db31db81e0c4407cac197d93495","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.3876953125,1.02734375],"calibrated_temperature":5.0},
+ {"id":"massive-5974-choice","group_id":"massive-alarm-f7cd5de4d0683af203ce","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"e7e220e3f9639398f9c6e56b8f38093b2018ad88a562b3f898609eacfb615f75","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[6.62890625,-4.55078125,-3.935546875],"temperature":1.0,"type":"choice","source_id":"5974","source_row_hash":"b4c1084215b35f9d593d4300ec53920326c4f994cb5bbe678cbc274e8068d703","input_tokens":54,"truncated":false,"calibrated_raw_logits":[6.62890625,-4.55078125,-3.935546875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-5974-noul","group_id":"massive-alarm-f7cd5de4d0683af203ce","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"964d28a1e6cbe408f1d74d44572fe81dc51b8fac4e3784c40464b96c34439c4d","option_order":["false","true"],"gold":"false","raw_logits":[-2.146484375,1.2470703125],"temperature":1.0,"type":"noul","source_id":"5974","source_row_hash":"b4c1084215b35f9d593d4300ec53920326c4f994cb5bbe678cbc274e8068d703","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.146484375,1.2470703125],"calibrated_temperature":5.0},
+ {"id":"massive-6083-choice","group_id":"massive-alarm-950ec71310e9ece1aa7a","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"a11eee2509927e63186f13c256d44d20ed64932038a23b4a0806cab08a89405b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.064453125,-1.720703125,-0.33642578125],"temperature":1.0,"type":"choice","source_id":"6083","source_row_hash":"c98e1b18be265f4f3518a3d4f075393c40074e8f1aae799dbcefe012b867cd89","input_tokens":53,"truncated":false,"calibrated_raw_logits":[1.064453125,-1.720703125,-0.33642578125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-6083-noul","group_id":"massive-alarm-950ec71310e9ece1aa7a","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"5a396304607490821ff2e74144e5202da5c16f6aa3b19624edec62d3cbc0496a","option_order":["false","true"],"gold":"true","raw_logits":[-1.982421875,1.3046875],"temperature":1.0,"type":"noul","source_id":"6083","source_row_hash":"c98e1b18be265f4f3518a3d4f075393c40074e8f1aae799dbcefe012b867cd89","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.982421875,1.3046875],"calibrated_temperature":5.0},
+ {"id":"massive-6101-choice","group_id":"massive-alarm-a63b13c9add3acdf4c9f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"06d76dcf51d497e0590fe5b28fa322168bfb9dce9eb88440c989d8a502850fc5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.08184814453125,-1.6630859375,-0.9814453125],"temperature":1.0,"type":"choice","source_id":"6101","source_row_hash":"3dce7763719f2037e2c03a6492336b70e796b37c6eec38a0243a9e146c04f22e","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.08184814453125,-1.6630859375,-0.9814453125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-6101-noul","group_id":"massive-alarm-a63b13c9add3acdf4c9f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"2eccf49f3d9075b4515ed86d93c2ae48a73534099b204323ff87e723347cbfb7","option_order":["false","true"],"gold":"false","raw_logits":[-0.765625,1.0244140625],"temperature":1.0,"type":"noul","source_id":"6101","source_row_hash":"3dce7763719f2037e2c03a6492336b70e796b37c6eec38a0243a9e146c04f22e","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-0.765625,1.0244140625],"calibrated_temperature":5.0},
+ {"id":"massive-612-choice","group_id":"massive-alarm-6d2b063ac90aec6042cb","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"df2193b6cb5257904a68c032afc1bbe241c266f2fd966aff1d5f2543ef419e58","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.40576171875,-2.451171875,-0.64404296875],"temperature":1.0,"type":"choice","source_id":"612","source_row_hash":"437786bbb8397c686ef9c55a8e67c91d16d8087753ecc83dccbe5e64726224a6","input_tokens":57,"truncated":false,"calibrated_raw_logits":[0.40576171875,-2.451171875,-0.64404296875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-612-noul","group_id":"massive-alarm-6d2b063ac90aec6042cb","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"aa455667dd535b15199206b1efc70f5485bcda3c2b32ff04ff3fb4a6bcb5437c","option_order":["false","true"],"gold":"true","raw_logits":[-0.074951171875,1.0908203125],"temperature":1.0,"type":"noul","source_id":"612","source_row_hash":"437786bbb8397c686ef9c55a8e67c91d16d8087753ecc83dccbe5e64726224a6","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.074951171875,1.0908203125],"calibrated_temperature":5.0},
+ {"id":"massive-6162-choice","group_id":"massive-alarm-7c67db34bf4904167369","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6c494191c98e1d84f5a75d1fe41fc46f153edf386c185a7e596bf70f25d2bf15","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.43359375,-3.333984375,2.044921875],"temperature":1.0,"type":"choice","source_id":"6162","source_row_hash":"9476a44d1260488e881b3dc20d371d653c5e2f622f312a564be0ff9faebdc44a","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-3.43359375,-3.333984375,2.044921875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-6162-noul","group_id":"massive-alarm-7c67db34bf4904167369","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7078e785d1141162ba7cc4503f84962ca97b9c6fdb845b23e0fe12d393519eeb","option_order":["false","true"],"gold":"false","raw_logits":[-1.7138671875,0.31005859375],"temperature":1.0,"type":"noul","source_id":"6162","source_row_hash":"9476a44d1260488e881b3dc20d371d653c5e2f622f312a564be0ff9faebdc44a","input_tokens":43,"truncated":false,"calibrated_raw_logits":[-1.7138671875,0.31005859375],"calibrated_temperature":5.0},
+ {"id":"massive-63-choice","group_id":"massive-alarm-3f054a8e6d320a808954","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"b27424721301b616f34476c934b889aeef8fe2d25823afb618470a06db45d9cd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.4794921875,3.935546875,-1.7333984375],"temperature":1.0,"type":"choice","source_id":"63","source_row_hash":"f9f143eecf685ee8dcf485e26ca404fb60e1e6244ad6133efb0ff64c6889fc7b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.4794921875,3.935546875,-1.7333984375],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-63-noul","group_id":"massive-alarm-3f054a8e6d320a808954","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"13c4f2f436a4a1cad7db316725fa8870b5829f21dd47c292beb942e925bd11d2","option_order":["false","true"],"gold":"false","raw_logits":[-0.765625,0.442626953125],"temperature":1.0,"type":"noul","source_id":"63","source_row_hash":"f9f143eecf685ee8dcf485e26ca404fb60e1e6244ad6133efb0ff64c6889fc7b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-0.765625,0.442626953125],"calibrated_temperature":5.0},
+ {"id":"massive-796-choice","group_id":"massive-alarm-1b94ce3d96f2d223de77","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"01608bf1f2ba0a3fe29f6ae16e61c8ee20458e453613bf297000ffae65be5d0b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.71142578125,-2.318359375,-2.154296875],"temperature":1.0,"type":"choice","source_id":"796","source_row_hash":"c0c137c04060af51a4172b5b368190deb3009c12c64fea91b2be12f7b694f28f","input_tokens":51,"truncated":false,"calibrated_raw_logits":[0.71142578125,-2.318359375,-2.154296875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-796-noul","group_id":"massive-alarm-1b94ce3d96f2d223de77","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e658950e96a415d7a90f36a0e8dbb02edf10a90e0ec6b9bf9b85672831d343d4","option_order":["false","true"],"gold":"false","raw_logits":[-1.1796875,0.55810546875],"temperature":1.0,"type":"noul","source_id":"796","source_row_hash":"c0c137c04060af51a4172b5b368190deb3009c12c64fea91b2be12f7b694f28f","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.1796875,0.55810546875],"calibrated_temperature":5.0},
+ {"id":"massive-861-choice","group_id":"massive-alarm-105aeeed3519053d3f55","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"adb310a263a979b824132b06ef4ca2fd6f40f379cafa5c0583d2893422e48ffe","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[5.2890625,-3.55078125,-2.4453125],"temperature":1.0,"type":"choice","source_id":"861","source_row_hash":"7c204b7b739698b3a6f70ee107ef52e531a47e8d7b51ed6b839e3583bc83351a","input_tokens":50,"truncated":false,"calibrated_raw_logits":[5.2890625,-3.55078125,-2.4453125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-861-noul","group_id":"massive-alarm-105aeeed3519053d3f55","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a55bf3c906f2bba70c4aa1a2f07dc2ea87f35341ef6fe8be3d694dbabe98589e","option_order":["false","true"],"gold":"false","raw_logits":[-2.2734375,1.072265625],"temperature":1.0,"type":"noul","source_id":"861","source_row_hash":"7c204b7b739698b3a6f70ee107ef52e531a47e8d7b51ed6b839e3583bc83351a","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.2734375,1.072265625],"calibrated_temperature":5.0},
+ {"id":"massive-8731-choice","group_id":"massive-alarm-c773d4135b5f98edcb5c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"e992f3c9dd54dfa9e1f81ed8dea16513e434dff0114a353de45a25822f20fc4b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.36376953125,-4.3046875,-1.541015625],"temperature":1.0,"type":"choice","source_id":"8731","source_row_hash":"37b09493cdc06a1bc88cc3643150ae9782b3ed10eea61dba2cb9c6519f20d4a1","input_tokens":59,"truncated":false,"calibrated_raw_logits":[-0.36376953125,-4.3046875,-1.541015625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-8731-noul","group_id":"massive-alarm-c773d4135b5f98edcb5c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ad5e11d3a7f8e16dc300f0f163db1e89309331376cd7149fc62d155a16de6d35","option_order":["false","true"],"gold":"true","raw_logits":[-2.794921875,0.283203125],"temperature":1.0,"type":"noul","source_id":"8731","source_row_hash":"37b09493cdc06a1bc88cc3643150ae9782b3ed10eea61dba2cb9c6519f20d4a1","input_tokens":54,"truncated":false,"calibrated_raw_logits":[-2.794921875,0.283203125],"calibrated_temperature":5.0},
+ {"id":"massive-8741-choice","group_id":"massive-alarm-79a03eb5dc7bab70336f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"12f704d636ca16e99b76ca5cea7307740f6125027d09a9c89c544fe86c2342f3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.1640625,-4.1875,-3.794921875],"temperature":1.0,"type":"choice","source_id":"8741","source_row_hash":"062a50b981a7bf96310a69f314e1b5e9e03ca5f3c1ff474bf49fbf780ad21844","input_tokens":52,"truncated":false,"calibrated_raw_logits":[6.1640625,-4.1875,-3.794921875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-8741-noul","group_id":"massive-alarm-79a03eb5dc7bab70336f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fb565c561562c1581b7292d775cfc16f8ebab4e949c1ac31e7d00f377e4d9834","option_order":["false","true"],"gold":"true","raw_logits":[-1.9677734375,1.234375],"temperature":1.0,"type":"noul","source_id":"8741","source_row_hash":"062a50b981a7bf96310a69f314e1b5e9e03ca5f3c1ff474bf49fbf780ad21844","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.9677734375,1.234375],"calibrated_temperature":5.0},
+ {"id":"massive-884-choice","group_id":"massive-alarm-42f057d6c35ee3122781","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"92579ff62a0442cf6f564f0ed96f983bec83f3f69dfe99d3ce8bb1259c1ac9d5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.1171875,-1.9990234375,-1.5869140625],"temperature":1.0,"type":"choice","source_id":"884","source_row_hash":"55b0f00bd02e71e600e190c2bc9e46f2a1d354fe59c818a9cb43fd1332486182","input_tokens":51,"truncated":false,"calibrated_raw_logits":[4.1171875,-1.9990234375,-1.5869140625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-884-noul","group_id":"massive-alarm-42f057d6c35ee3122781","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4ab83fec7f87887afa7cfc2310e2c6a492f625fe0400fac7bc6ed4813067f280","option_order":["false","true"],"gold":"true","raw_logits":[-2.265625,1.5087890625],"temperature":1.0,"type":"noul","source_id":"884","source_row_hash":"55b0f00bd02e71e600e190c2bc9e46f2a1d354fe59c818a9cb43fd1332486182","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.265625,1.5087890625],"calibrated_temperature":5.0},
+ {"id":"massive-8867-choice","group_id":"massive-alarm-bfe246fb1d4b99a3f48a","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cbde715e981d6e934a2aac954045f8eae203ebf02c68432c7343a6bf8c3cc940","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.603515625,-3.880859375,-2.998046875],"temperature":1.0,"type":"choice","source_id":"8867","source_row_hash":"a513b328db26ac85863558c72614b0034a2f8177453c5d71df24760e4e766cd7","input_tokens":53,"truncated":false,"calibrated_raw_logits":[2.603515625,-3.880859375,-2.998046875],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-8867-noul","group_id":"massive-alarm-bfe246fb1d4b99a3f48a","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e2b236cc79afa43b3fc316499b938cc5aadf2150ff7a433da58df64c902ca760","option_order":["false","true"],"gold":"true","raw_logits":[-1.4541015625,0.333984375],"temperature":1.0,"type":"noul","source_id":"8867","source_row_hash":"a513b328db26ac85863558c72614b0034a2f8177453c5d71df24760e4e766cd7","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.4541015625,0.333984375],"calibrated_temperature":5.0},
+ {"id":"massive-8882-choice","group_id":"massive-alarm-84c38cd9104789d07c40","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ff0063357703eca9276281b428c4dda0916cc9e39a13740943d5e2776399f15e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.3515625,-2.701171875,-1.8369140625],"temperature":1.0,"type":"choice","source_id":"8882","source_row_hash":"e1c5f172f9f02cade5685772af79ebded1b74b376026d93341832931a32676f9","input_tokens":57,"truncated":false,"calibrated_raw_logits":[2.3515625,-2.701171875,-1.8369140625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-8882-noul","group_id":"massive-alarm-84c38cd9104789d07c40","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"94a0ac44c2af3923c53e676b3d98f5ee321b0cc3a3a31c051d034ec2c89445ce","option_order":["false","true"],"gold":"true","raw_logits":[-2.046875,1.22265625],"temperature":1.0,"type":"noul","source_id":"8882","source_row_hash":"e1c5f172f9f02cade5685772af79ebded1b74b376026d93341832931a32676f9","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-2.046875,1.22265625],"calibrated_temperature":5.0},
+ {"id":"massive-920-choice","group_id":"massive-alarm-84e033aebf70598798ff","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"d2f20e3d91af6e06b4f1acf721dcd73de93d27d83c024195da8ad52b1bceecdd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.517578125,-4.140625,4.17578125],"temperature":1.0,"type":"choice","source_id":"920","source_row_hash":"ed0a9ccdf3b6e428b4770ab41b3c2b59fa4327b9580291f077f18f49a54b44b2","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-3.517578125,-4.140625,4.17578125],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-920-noul","group_id":"massive-alarm-84e033aebf70598798ff","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4cc035ea2dff281db57f6de7c194ec38bfe20021c39aadf5fc817997e722b652","option_order":["false","true"],"gold":"false","raw_logits":[-2.470703125,-0.1820068359375],"temperature":1.0,"type":"noul","source_id":"920","source_row_hash":"ed0a9ccdf3b6e428b4770ab41b3c2b59fa4327b9580291f077f18f49a54b44b2","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.470703125,-0.1820068359375],"calibrated_temperature":5.0},
+ {"id":"massive-939-choice","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"b39b50798b10226d8212f2e2f9e3a8929c244b25102bcf6e856c5df4b13d4ee8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.4375,-3.669921875,-0.99853515625],"temperature":1.0,"type":"choice","source_id":"939","source_row_hash":"936109e1a09f4aecb014f31c6bbb213ce6de4023624f13dfe11e9c083dd0320e","input_tokens":54,"truncated":false,"calibrated_raw_logits":[5.4375,-3.669921875,-0.99853515625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-939-noul","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"dd6f955f414596fad6478d08e6f36bea3627253a74ec2d137249375a66c63afb","option_order":["false","true"],"gold":"true","raw_logits":[-2.650390625,0.978515625],"temperature":1.0,"type":"noul","source_id":"939","source_row_hash":"936109e1a09f4aecb014f31c6bbb213ce6de4023624f13dfe11e9c083dd0320e","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.650390625,0.978515625],"calibrated_temperature":5.0},
+ {"id":"massive-940-choice","group_id":"massive-alarm-5b03b657dea06a387027","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"58112a614bee3f995d34f5cd7698b4fc18f8696a65393c5d108082656961fb29","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.388671875,-2.8046875,-0.0709228515625],"temperature":1.0,"type":"choice","source_id":"940","source_row_hash":"78635f925b3274baf503c5823a54dfb84e1a144e79ad8a8ef3ffbd50edfc6c4b","input_tokens":50,"truncated":false,"calibrated_raw_logits":[1.388671875,-2.8046875,-0.0709228515625],"calibrated_temperature":3.1355035588563744},
+ {"id":"massive-940-noul","group_id":"massive-alarm-5b03b657dea06a387027","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"d534399a37dd93763633f7d9628ca5918f84cacf599473cd36b168ef8fcf2198","option_order":["false","true"],"gold":"true","raw_logits":[-2.1015625,1.1630859375],"temperature":1.0,"type":"noul","source_id":"940","source_row_hash":"78635f925b3274baf503c5823a54dfb84e1a144e79ad8a8ef3ffbd50edfc6c4b","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.1015625,1.1630859375],"calibrated_temperature":5.0}
+ ]
+}
diff --git a/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.md b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.md
new file mode 100644
index 00000000..2360a9c0
--- /dev/null
+++ b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.md
@@ -0,0 +1,43 @@
+# MASSIVE Chinese alarm calibration evaluation
+
+Measurement date: **2026-09-24**. Execution base: `23a17522aa4942da6cce53a995a275760320b691`; contribution: `37cc58465da5cdabddf1ff543234397d666ad546`.
+
+Historical inference was executed on 2026-09-24 from base commit 23a17522aa4942da6cce53a995a275760320b691 with a dirty contribution worktree subsequently recorded in 37cc58465da5cdabddf1ff543234397d666ad546. The recorded execution source-file hashes are authoritative for the measured code. These measurements were not rerun on the later upstream rebase or evaluation-only refactor.
+
+E0 and E1 use the same multilingual checkpoint and raw decision logits. E1 fits one temperature per question type on calibration data. No model weights are updated.
+
+| Condition | Slice | n | Accuracy | Macro F1 | NLL | Brier | ECE |
+|---|---|---:|---:|---:|---:|---:|---:|
+| E0 | overall | 186 | 0.532258 | 0.440840 | 1.383747 | 0.730071 | 0.366488 |
+| E0 | choice | 93 | 0.634409 | 0.580928 | 1.296043 | 0.520825 | 0.253426 |
+| E0 | noul | 93 | 0.430108 | 0.300752 | 1.471450 | 0.939317 | 0.494611 |
+| E1 | overall | 186 | 0.532258 | 0.440840 | 0.785033 | 0.506898 | 0.197529 |
+| E1 | choice | 93 | 0.634409 | 0.580928 | 0.837271 | 0.474032 | 0.169905 |
+| E1 | noul | 93 | 0.430108 | 0.300752 | 0.732796 | 0.539765 | 0.289635 |
+
+Positive scalar temperature preserves argmax: accuracy and F1 are unchanged.
+
+| Type | Calibration n | Temperature | Bound reached |
+|---|---:|---:|---|
+| choice | 64 | 3.135504 | False |
+| noul | 64 | 5.000000 | True |
+
+Paired differences are E1 minus E0; negative NLL, Brier and ECE differences are better.
+
+| Metric | Difference | 95% group bootstrap interval |
+|---|---:|---|
+| accuracy | +0.000000 | [+0.000000, +0.000000] |
+| macro_f1 | +0.000000 | [+0.000000, +0.000000] |
+| nll | -0.598713 | [-0.865774, -0.369099] |
+| brier | -0.223173 | [-0.298826, -0.152890] |
+| ece | -0.168958 | [-0.224995, -0.098547] |
+
+## Scope and limits
+
+- Test: 93 source utterances, 89 heuristic groups, 186 correlated decisions. Calibration: 64 source utterances, 62 groups, 128 decisions. The paired question types are not independent samples.
+- This is a derived MASSIVE 1.1 alarm subset with upstream human intent-review evidence, not the full benchmark, naturally occurring customer-service logs, or independent local human review.
+- One fixed split/seed; 1,000 group bootstrap draws. Intervals condition on the fixed checkpoint and fitted temperatures, omit calibration uncertainty, and are descriptive without multiplicity correction.
+- Unknown checkpoint exposure to MASSIVE/SLURP; grouping may miss semantic dependencies. These results do not establish general Chinese improvement or unchanged behavior on other tasks.
+- The baseline noul fit reaches T=5, the permitted upper bound. Calibration settings were fixed without choosing on test. Risk/coverage does not improve at every coverage.
+- Configured max_len=512, head_max_len=192, FP16. Historical test inputs were at most 60 tokens; this is not a 512-token stress measurement.
+- The compact evidence omits source text and retains source IDs/hashes. It can recompute numerical results and check pairing; full source-schema/annotation checks require the prepared public dataset.
diff --git a/research/benchmarks/zh_reliability/runtime.py b/research/benchmarks/zh_reliability/runtime.py
new file mode 100644
index 00000000..0eb6b5ae
--- /dev/null
+++ b/research/benchmarks/zh_reliability/runtime.py
@@ -0,0 +1,277 @@
+"""Environment diagnostics and a thin adapter over the repository's actual Agent."""
+import importlib.metadata
+import json
+import os
+from pathlib import Path
+import platform
+import shutil
+import subprocess
+import sys
+import time
+
+from .data import atomic_json, digest, file_hash, options
+
+MODEL_ID = 'convaiinnovations/laya-multilingual'
+INPUT_KEYS = ('input_ids', 'attention_mask', 'marker_pos', 'marker_mask', 'qtype')
+
+
+def command(args):
+ try:
+ p = subprocess.run(args, capture_output=True, text=True, timeout=20)
+ return p.stdout.strip() if p.returncode == 0 else p.stderr.strip()
+ except (OSError, subprocess.TimeoutExpired) as e:
+ return str(e)
+
+
+def environment():
+ result = {'python': sys.version, 'executable': sys.executable, 'os': platform.platform(),
+ 'commit': command(['git', 'rev-parse', 'HEAD']),
+ 'dirty': bool(command(['git', 'status', '--porcelain'])),
+ 'source_hashes': {str(p): file_hash(p) for p in sorted(Path(__file__).parent.glob('*.py'))},
+ 'environment': {k: os.environ[k] for k in ('CUDA_VISIBLE_DEVICES', 'CUDA_DEVICE_ORDER',
+ 'HF_HUB_OFFLINE', 'USE_TF', 'USE_TORCH', 'TOKENIZERS_PARALLELISM', 'OMP_NUM_THREADS')
+ if k in os.environ}, 'dependencies': {},
+ 'nvidia_smi': command(['nvidia-smi', '--query-gpu=index,name,uuid,memory.total,driver_version',
+ '--format=csv,noheader']), 'gpus': []}
+ for package in ('torch', 'transformers', 'numpy', 'safetensors', 'huggingface_hub', 'tokenizers'):
+ try:
+ result['dependencies'][package] = importlib.metadata.version(package)
+ except importlib.metadata.PackageNotFoundError:
+ result['dependencies'][package] = None
+ try:
+ import torch
+ result.update(cuda=torch.version.cuda, cudnn=torch.backends.cudnn.version(),
+ pytorch_arches=torch.cuda.get_arch_list())
+ for i in range(torch.cuda.device_count()):
+ p = torch.cuda.get_device_properties(i)
+ gpu = {'process_index': i, 'name': p.name, 'uuid': str(getattr(p, 'uuid', 'unavailable')),
+ 'memory_bytes': p.total_memory, 'capability': [p.major, p.minor]}
+ if gpu['uuid'] != 'unavailable' and not gpu['uuid'].startswith('GPU-'):
+ gpu['uuid'] = 'GPU-' + gpu['uuid']
+ try:
+ with torch.cuda.device(i):
+ x = torch.ones((16, 16), device=f'cuda:{i}')
+ value = (x @ x).sum().item()
+ torch.cuda.synchronize(i)
+ if value != 4096:
+ raise RuntimeError('CUDA tensor result mismatch')
+ del x
+ gpu['status'] = 'SUCCESS'
+ except Exception as e:
+ gpu.update(status='FAILED', reason=str(e))
+ # UUID matching avoids confusing process-local index with physical index.
+ matches = [line.split(',')[0].strip() for line in result['nvidia_smi'].splitlines()
+ if gpu['uuid'] != 'unavailable' and gpu['uuid'] in line]
+ gpu['physical_index'] = matches[0] if len(matches) == 1 else None
+ result['gpus'].append(gpu)
+ result['gpu_status'] = ('SUCCESS' if result['gpus'] and all(g['status'] == 'SUCCESS' for g in result['gpus'])
+ else 'FAILED' if result['gpus'] else 'NOT_RUN')
+ except Exception as e:
+ result.update(gpu_status='NOT_RUN', reason=str(e))
+ return result
+
+
+def select_device(selector, env):
+ matches = [g for g in env['gpus'] if selector.lower() in g['name'].lower() or selector == g['uuid']
+ or selector == f"cuda:{g['process_index']}"]
+ if len(matches) != 1 or matches[0]['status'] != 'SUCCESS':
+ raise RuntimeError('device must uniquely match one visible GPU with a successful CUDA operation: ' + selector)
+ return 'cuda:' + str(matches[0]['process_index'])
+
+
+def resolve_model(source, revision, offline, work):
+ """Make a private snapshot so Agent's tokenizer compatibility patch cannot touch shared caches."""
+ source_path = Path(source)
+ sha = None
+ if not source_path.is_dir():
+ if source != MODEL_ID:
+ raise ValueError('only the multilingual model is supported')
+ from huggingface_hub import HfApi, snapshot_download
+ if not offline:
+ sha = HfApi().model_info(MODEL_ID, revision=revision).sha
+ path = snapshot_download(MODEL_ID, revision=sha or revision, local_files_only=offline,
+ allow_patterns=['rl_agent_config.json', 'model.safetensors',
+ 'encoder/*.json', 'tokenizer/*'])
+ source_path = Path(path)
+ sha = source_path.name
+ lineage_path = source_path / 'zh_lineage.json'
+ lineage = json.loads(lineage_path.read_text()) if lineage_path.exists() else None
+ if lineage and (lineage['model_id'] != MODEL_ID or lineage.get('trained')):
+ raise ValueError('evaluation requires an original multilingual checkpoint')
+ cfg = json.loads((source_path / 'rl_agent_config.json').read_text())
+ # Local imports require explicit, inspectable multilingual identity, not just a folder name.
+ if not lineage and not any(x in str(cfg.get('encoder', '')).lower() for x in ('mmbert', 'multilingual')):
+ raise ValueError('local checkpoint config does not identify a multilingual encoder')
+ files = [source_path / 'model.safetensors', source_path / 'rl_agent_config.json']
+ files += sorted((source_path / 'encoder').glob('*.json')) + sorted((source_path / 'tokenizer').glob('*'))
+ hashes = {str(p.relative_to(source_path)): file_hash(p) for p in files if p.is_file()}
+ if lineage and lineage.get('weights_hash') and lineage['weights_hash'] != hashes['model.safetensors']:
+ raise ValueError('checkpoint weight hash mismatch')
+ # Preserve the base identity shape so existing calibration artifacts still match.
+ identity = {'model_id': MODEL_ID, 'revision': sha or (lineage or {}).get('base_revision'),
+ 'content_hash': digest(hashes), 'files': hashes,
+ 'trained': False}
+ target = Path(work) / 'model_input'
+ target.mkdir()
+ for name in hashes:
+ dst = target / name
+ dst.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copyfile(source_path / name, dst)
+ atomic_json(Path(work) / 'model_identity.json', identity)
+ return target, identity
+
+
+def ensure_device(agent, device):
+ import torch
+ expected = torch.device(device)
+ if agent.device != expected or any(p.device != expected for p in agent.model.parameters()):
+ raise RuntimeError('device mismatch / SDK CPU fallback: requested ' + device)
+
+
+class Adapter:
+ def __init__(self, agent, identity, max_len=512, head_max_len=192, dtype='fp16'):
+ import torch
+ self.agent, self.identity = agent, identity
+ self.max_len, self.head_max_len, self.dtype = max_len, head_max_len, dtype
+ self.device = str(agent.device)
+ if head_max_len < 16 or max_len <= head_max_len + 4:
+ raise ValueError('max_len must leave state space beyond head_max_len + 4')
+ agent.amp_enabled = dtype == 'fp16' and agent.device.type == 'cuda'
+ agent.dtype = torch.float16 if agent.amp_enabled else torch.float32
+ self.original = {'temperature_raw': agent.temperature_raw,
+ 'temperature_by_options_raw': agent.temperature_by_options_raw,
+ 'temperature': list(agent.temperature),
+ 'temperature_by_options': dict(agent.temperature_by_options),
+ 'lang_temperatures': dict(agent.lang_temperatures)}
+
+ def temperature(self, row):
+ from laya.common import QTYPES, temp_bucket
+ qt = QTYPES[row['question']['type']]
+ cfg = self.agent.lang_temperatures.get('zh', {'temperature': self.agent.temperature,
+ 'temperature_by_options': self.agent.temperature_by_options})
+ return cfg['temperature_by_options'].get(temp_bucket(qt, len(options(row))), cfg['temperature'][qt])
+
+ def prepare(self, row):
+ from laya.common import build_sequence, render_options
+ a, q = self.agent, row['question']
+ a._check_question(row['id'], q)
+ internal = a._to_internal(q)
+ item = a._encode_state(row['state'], [row['id']], {row['id']: internal},
+ self.max_len, self.head_max_len)[0]
+ state_ids = a.tok(row['state'].replace(a.tok.mask_token, ' '), add_special_tokens=False)['input_ids']
+ empty, markers = build_sequence(a.tok, '', internal, self.max_len, self.head_max_len, state_ids=[])
+ if len(markers) != len(options(row)) or any(item['ids'][m] != a.tok.mask_token_id for m in markers):
+ raise ValueError('option marker truncated')
+ retained = max(0, len(item['ids']) - len(empty))
+ head = a.tok(f"{internal['t']} question: {internal['ins'].replace(a.tok.mask_token, ' ')}",
+ add_special_tokens=False)['input_ids']
+ opt_full = [a.tok(' ' + o.replace(a.tok.mask_token, ' '), add_special_tokens=False)['input_ids']
+ for o in render_options(internal)]
+ head_cut = len(head) > markers[0] - 2
+ opt_cut = any(len(full) > (markers[i + 1] if i + 1 < len(markers) else len(empty) - 2) - m - 1
+ for i, (m, full) in enumerate(zip(markers, opt_full)))
+ item['label'] = options(row).index(row['label'])
+ tokens = {'input_tokens': len(item['ids']), 'state_tokens': len(state_ids), 'state_retained': retained,
+ 'option_count': len(markers), 'state_truncated': retained < len(state_ids),
+ 'instructions_truncated': head_cut, 'options_truncated': opt_cut,
+ 'truncated': retained < len(state_ids) or head_cut or opt_cut}
+ return item, tokens
+
+ def batch(self, items):
+ from laya.common import collate_items
+ return collate_items([items], self.agent.tok.pad_token_id)
+
+ def forward(self, batch):
+ import torch
+ ensure_device(self.agent, self.device)
+ with torch.autocast(device_type=self.agent.device.type, dtype=torch.float16,
+ enabled=self.dtype == 'fp16' and self.agent.device.type == 'cuda'):
+ logits, _action_logits = self.agent.model(
+ **{k: batch[k].to(self.agent.device) for k in INPUT_KEYS})
+ if not torch.isfinite(logits).all():
+ raise RuntimeError('nonfinite decision logits')
+ return logits
+
+ def predict(self, rows, condition, split):
+ import torch
+ from .metrics import probability_record
+ self.agent.model.eval()
+ results = []
+ # One state has exactly one decision sequence in this schema.
+ with torch.no_grad():
+ for r in rows:
+ item, tokens = self.prepare(r)
+ logits = self.forward(self.batch([item]))[0, :len(options(r))].float().cpu().tolist()
+ rec = probability_record(r, logits, self.temperature(r))
+ rec.update(tokens=tokens, condition=condition, split=split['assignments'][r['id']], model=self.identity)
+ results.append(rec)
+ return results
+
+ def parity(self, rows):
+ import numpy as np
+ from .metrics import probability_record
+ import torch
+ errors = []
+ self.agent.model.eval()
+ for row in rows:
+ item, _ = self.prepare(row)
+ with torch.no_grad():
+ z = self.forward(self.batch([item]))[0, :len(options(row))].cpu().tolist()
+ p = probability_record(row, z, self.temperature(row))['probabilities']
+ fallback_count = self.agent.cpu_fallback_count
+ answer = self.agent.system_one(row['state'], {row['id']: row['question']}, lang='zh',
+ max_len=self.max_len, head_max_len=self.head_max_len)['answers'][row['id']]
+ ensure_device(self.agent, self.device)
+ if self.agent.cpu_fallback_count != fallback_count:
+ raise RuntimeError('Agent parity used CPU fallback')
+ public = ([1 - answer['noul'], answer['noul']] if row['question']['type'] == 'noul'
+ else [answer['probabilities'][k] for k in options(row)])
+ error = float(np.max(np.abs(np.array(p) - public)))
+ if error > .000051:
+ raise AssertionError('Agent parity exceeds four-decimal rounding: ' + str(error))
+ errors.append(error)
+ return {'n': len(rows), 'max_absolute_error': max(errors), 'rounding_tolerance': .000051}
+
+
+def measure(adapter, rows, warmup=10, batches=30):
+ import numpy as np
+ import torch
+ if adapter.agent.device.type != 'cuda':
+ raise RuntimeError('GPU measurements prohibit CPU fallback')
+ if warmup < 1 or batches < 1 or not rows:
+ raise ValueError('measurement needs positive warmup/batches and nonempty samples')
+ device = adapter.agent.device
+ adapter.agent.model.eval()
+ prepared = [adapter.batch([adapter.prepare(r)[0]]) for r in rows]
+ prepared_gpu = [{k: b[k].to(device) for k in INPUT_KEYS} for b in prepared]
+ result = {'warmup_batches': warmup, 'measured_batches': batches, 'batch_states': 1,
+ 'batch_decisions': 1, 'independent_samples': len(rows),
+ 'tokens': [adapter.prepare(r)[1] for r in rows], 'repetitions': batches / len(rows),
+ 'model_stage_transfers': 'excludes H2D/D2H; inputs already on device'}
+ from .metrics import probability_record
+ with torch.no_grad():
+ for stage in ('end_to_end', 'model_stage'):
+ times = []
+ torch.cuda.synchronize(device)
+ torch.cuda.reset_peak_memory_stats(device)
+ for i in range(warmup + batches):
+ row = rows[i % len(rows)]
+ torch.cuda.synchronize(device)
+ start = time.perf_counter()
+ if stage == 'end_to_end':
+ item, _ = adapter.prepare(row)
+ z = adapter.forward(adapter.batch([item]))[0, :len(options(row))].cpu().tolist()
+ probability_record(row, z, adapter.temperature(row))
+ else:
+ adapter.forward(prepared_gpu[i % len(rows)])
+ torch.cuda.synchronize(device)
+ elapsed = time.perf_counter() - start
+ if i >= warmup:
+ times.append(elapsed)
+ result[stage] = {'p50_batch_ms': float(np.quantile(times, .5) * 1000),
+ 'p95_batch_ms': float(np.quantile(times, .95) * 1000),
+ 'state_per_s': batches / sum(times), 'decision_per_s': batches / sum(times),
+ 'max_memory_allocated': torch.cuda.max_memory_allocated(device),
+ 'max_memory_reserved': torch.cuda.max_memory_reserved(device),
+ 'memory_scope': 'inference including warmup, resident model and prepared inputs; excludes model loading'}
+ return result
diff --git a/tests/test_zh_annotation_provenance.py b/tests/test_zh_annotation_provenance.py
new file mode 100644
index 00000000..04e92219
--- /dev/null
+++ b/tests/test_zh_annotation_provenance.py
@@ -0,0 +1,150 @@
+"""Offline regressions for upstream review evidence; all rows here are synthetic."""
+import copy
+import json
+from pathlib import Path
+import sys
+import tempfile
+import unittest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+from research.benchmarks.zh_reliability.data import partition, read_data
+
+
+class UpstreamAnnotationProvenanceTests(unittest.TestCase):
+ def setUp(self):
+ self.directory = tempfile.TemporaryDirectory()
+ self.addCleanup(self.directory.cleanup)
+ self.path = Path(self.directory.name) / 'data.jsonl'
+ self.row = {
+ 'id': 'synthetic_alarm_1', 'group_id': 'synthetic_source_1', 'task_id': 'alarm_operation',
+ 'state': '设置明早的闹钟', 'split': 'train',
+ 'question': {'type': 'choice', 'instructions': '选择闹钟操作',
+ 'criteria': {'alarm_set': '设置闹钟', 'alarm_remove': '删除闹钟'}},
+ 'label': 'alarm_set', 'tags': [],
+ 'provenance': {
+ 'kind': 'public_human_annotated', 'human_verified': False,
+ 'source': 'synthetic source for offline validation only', 'license': 'CC-BY-4.0',
+ 'annotation_status': 'upstream intent judgments; no local human review',
+ 'source_intent': 'alarm_set',
+ 'upstream_annotation': {
+ 'dataset': 'MASSIVE', 'version': '1.1', 'source_id': '1', 'intent': 'alarm_set',
+ 'source_row_hash': 'b' * 64,
+ 'evidence_url': 'https://github.com/alexa/massive/blob/' + 'a' * 40 + '/README.md#data-format',
+ 'judgments': [{'worker_id': str(i), 'intent_score': 1} for i in range(3)],
+ },
+ },
+ }
+
+ def read(self, rows, formal=True):
+ self.path.write_text(''.join(json.dumps(r, ensure_ascii=False) + '\n' for r in rows), encoding='utf-8')
+ return read_data(self.path, formal=formal)
+
+ def test_source_review_keeps_local_review_false_and_preserves_group_splits(self):
+ rows = []
+ for i, split in enumerate(('train', 'dev', 'calibration', 'test')):
+ source = copy.deepcopy(self.row)
+ source.update(id=str(i), group_id='source_' + str(i), split=split)
+ source['provenance']['upstream_annotation']['source_id'] = str(i)
+ binary = copy.deepcopy(source)
+ binary.update(id=str(i) + '_binary', task_id='alarm_set_binary', label='true')
+ binary['question'] = {'type': 'noul', 'instructions': '是否设置闹钟?',
+ 'criteria': {'false': '不是设置闹钟', 'true': '设置闹钟'}}
+ rows.extend((source, binary))
+ loaded = self.read(rows)
+ self.assertTrue(all(r['provenance']['human_verified'] is False for r in loaded))
+ manifest = partition(loaded, formal=True)
+ self.assertEqual(manifest['group_counts'], {s: 1 for s in ('train', 'dev', 'calibration', 'test')})
+ self.assertEqual(manifest['assignments'], {r['id']: r['split'] for r in rows})
+ rows[-1]['split'] = 'train'
+ with self.assertRaisesRegex(ValueError, 'split a group'):
+ partition(self.read(rows), formal=True)
+
+ def test_missing_or_invalid_identity_evidence_is_rejected(self):
+ changes = {
+ 'dataset': (None, 'Other'), 'version': (None, '1.0'),
+ 'source_id': (None, 1, '', ' '), 'intent': (None, '', 'alarm_remove'),
+ 'source_row_hash': (None, '', 'b' * 63, 'g' * 64),
+ 'evidence_url': (None, 'https://github.com/alexa/massive/blob/main/README.md',
+ 'https://example.com/' + 'a' * 40 + '/README.md',
+ 'https://github.com/alexa/massive/blob/' + 'a' * 40 + '/README.md?branch=main'),
+ }
+ for field, values in changes.items():
+ for value in values:
+ with self.subTest(field=field, value=value):
+ row = copy.deepcopy(self.row)
+ row['provenance']['upstream_annotation'][field] = value
+ with self.assertRaisesRegex(ValueError, 'source-review evidence'):
+ self.read([row])
+ for value in (None, [], 'claimed review'):
+ row = copy.deepcopy(self.row)
+ row['provenance']['upstream_annotation'] = value
+ with self.assertRaisesRegex(ValueError, 'source-review evidence'):
+ self.read([row])
+
+ def test_contested_missing_or_duplicate_judgments_are_rejected(self):
+ valid = self.row['provenance']['upstream_annotation']['judgments']
+ invalid = [None, {}, [], valid[:2], valid[:2] + [valid[0]], valid + [None]]
+ for score in (0, 2, True, 1.0, '1', None):
+ judgments = copy.deepcopy(valid)
+ judgments[2]['intent_score'] = score
+ invalid.append(judgments)
+ for worker in ('', ' ', None, 2):
+ judgments = copy.deepcopy(valid)
+ judgments[2]['worker_id'] = worker
+ invalid.append(judgments)
+ invalid.append(valid + [{'worker_id': 'fourth', 'intent_score': 0}])
+ for judgments in invalid:
+ with self.subTest(judgments=judgments):
+ row = copy.deepcopy(self.row)
+ row['provenance']['upstream_annotation']['judgments'] = judgments
+ with self.assertRaisesRegex(ValueError, 'source-review evidence'):
+ self.read([row])
+
+ def test_public_kind_cannot_claim_local_human_review(self):
+ row = copy.deepcopy(self.row)
+ row['provenance']['human_verified'] = True
+ for formal in (False, True):
+ with self.subTest(formal=formal), self.assertRaisesRegex(ValueError, 'source-review evidence'):
+ self.read([row], formal=formal)
+
+ def test_formal_source_license_status_and_split_are_still_required(self):
+ for field in ('source', 'license', 'annotation_status'):
+ row = copy.deepcopy(self.row)
+ del row['provenance'][field]
+ with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'formal data requires'):
+ self.read([row])
+ row = copy.deepcopy(self.row)
+ del row['split']
+ with self.assertRaisesRegex(ValueError, 'formal data requires'):
+ self.read([row])
+ row = copy.deepcopy(self.row)
+ del row['provenance']['source_intent']
+ with self.assertRaisesRegex(ValueError, 'source-review evidence'):
+ self.read([row])
+
+ def test_pinned_raw_document_url_is_accepted(self):
+ row = copy.deepcopy(self.row)
+ row['provenance']['upstream_annotation']['evidence_url'] = (
+ 'https://raw.githubusercontent.com/alexa/massive/' + 'a' * 40 + '/README.md')
+ self.assertEqual(len(self.read([row])), 1)
+
+ def test_existing_local_verification_and_fixture_rules_are_preserved(self):
+ row = copy.deepcopy(self.row)
+ row['provenance']['kind'] = 'user_annotated'
+ row['provenance']['human_verified'] = True
+ del row['provenance']['upstream_annotation']
+ self.assertEqual(len(self.read([row])), 1)
+ row['provenance']['kind'] = 'agent_authored_fixture'
+ with self.assertRaisesRegex(ValueError, 'cannot claim human verification'):
+ self.read([row])
+ row['provenance']['human_verified'] = False
+ self.assertEqual(len(self.read([row], formal=False)), 1)
+ with self.assertRaisesRegex(ValueError, 'formal data requires'):
+ self.read([row])
+ row['provenance']['kind'] = 'unverified_external'
+ with self.assertRaisesRegex(ValueError, 'formal data requires'):
+ self.read([row])
+
+
+if __name__ == '__main__':
+ unittest.main(verbosity=2)
diff --git a/tests/test_zh_massive.py b/tests/test_zh_massive.py
new file mode 100644
index 00000000..5b149544
--- /dev/null
+++ b/tests/test_zh_massive.py
@@ -0,0 +1,440 @@
+"""Offline MASSIVE converter checks using fabricated, test-only source records."""
+import copy
+import io
+from pathlib import Path
+import sys
+import tarfile
+import tempfile
+import unittest
+from unittest import mock
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+from research.benchmarks.zh_reliability import prepare_massive as massive
+from research.benchmarks.zh_reliability.data import atomic_json, digest, file_hash, partition, read_data
+
+
+class MassiveConversionTests(unittest.TestCase):
+ def setUp(self):
+ self.directory = tempfile.TemporaryDirectory()
+ self.addCleanup(self.directory.cleanup)
+ self.path = Path(self.directory.name)
+ self.source = {'archive_sha256': 'b' * 64}
+ self.chinese, self.english = [], []
+ for i, split in enumerate(('train', 'train', 'train', 'train', 'dev', 'test')):
+ self.add(str(i), split, tuple(massive.INTENTS)[i % 3])
+
+ def add(self, source_id, split, intent='alarm_set', utterance=None, template=None):
+ chinese = {
+ 'id': source_id, 'locale': 'zh-CN', 'partition': split, 'scenario': 'alarm', 'intent': intent,
+ 'utt': utterance or '仅供测试的合成句子' + source_id,
+ 'annot_utt': '仅供测试的合成句子' + source_id,
+ 'judgments': [{'worker_id': str(i), 'intent_score': 1} for i in range(3)],
+ }
+ english = {**copy.deepcopy(chinese), 'locale': 'en-US',
+ 'utt': 'Fabricated source for test only ' + source_id,
+ 'annot_utt': template or 'Fabricated source for test only ' + source_id}
+ self.chinese.append(chinese)
+ self.english.append(english)
+ return chinese, english
+
+ def convert(self, **kwargs):
+ return massive.convert(self.chinese, self.english, self.source, **kwargs)
+
+ @staticmethod
+ def by_source(rows):
+ result = {}
+ for row in rows:
+ source_id = row['provenance']['upstream_annotation']['source_id']
+ result.setdefault(source_id, {})[row['question']['type']] = row
+ return result
+
+ def test_exact_labels_shared_groups_and_upstream_provenance(self):
+ rows, diagnostics, summary = self.convert()
+ self.assertEqual(diagnostics, [])
+ self.assertEqual(summary['derived_decisions'], 2 * len(self.chinese))
+ for source in self.chinese:
+ pair = self.by_source(rows)[source['id']]
+ choice, noul = pair['choice'], pair['noul']
+ self.assertEqual(choice['label'], source['intent'])
+ self.assertEqual(choice['question']['criteria'], massive.INTENTS)
+ self.assertEqual(noul['label'], 'true' if source['intent'] == 'alarm_set' else 'false')
+ self.assertEqual(list(noul['question']['criteria']), ['false', 'true'])
+ self.assertEqual(choice['group_id'], noul['group_id'])
+ self.assertEqual(choice['split'], noul['split'])
+ self.assertEqual(choice['state'], source['utt'])
+ self.assertEqual(noul['state'], source['utt'])
+ provenance = choice['provenance']
+ self.assertIs(provenance['human_verified'], False)
+ self.assertIs(provenance['local_human_review'], False)
+ self.assertEqual(provenance['kind'], 'public_human_annotated')
+ evidence = provenance['upstream_annotation']
+ self.assertEqual(evidence['intent'], source['intent'])
+ self.assertEqual(evidence['source_row_hash'], digest(source))
+ self.assertEqual(evidence['judgments'], source['judgments'])
+ self.assertEqual(evidence['evidence_url'], massive.EVIDENCE_URL)
+ massive.write_jsonl(self.path / 'derived.jsonl', rows)
+ validated = read_data(self.path / 'derived.jsonl', formal=True)
+ self.assertEqual(partition(validated, formal=True)['counts'],
+ {s: summary['by_split'][s]['decisions'] for s in summary['by_split']})
+
+ def test_official_dev_test_are_preserved_and_calibration_is_only_from_train(self):
+ rows, _, summary = self.convert(calibration_fraction=.5)
+ pairs = self.by_source(rows)
+ self.assertEqual(pairs['4']['choice']['split'], 'dev')
+ self.assertEqual(pairs['5']['choice']['split'], 'test')
+ source_splits = {r['id']: r['partition'] for r in self.chinese}
+ for source_id, pair in pairs.items():
+ if pair['choice']['split'] == 'calibration':
+ self.assertEqual(source_splits[source_id], 'train')
+ else:
+ self.assertEqual(pair['choice']['split'], source_splits[source_id])
+ self.assertEqual(summary['by_split']['train']['source_rows'], 2)
+ self.assertEqual(summary['by_split']['calibration']['source_rows'], 2)
+ self.assertEqual(sum(x['source_rows'] for x in summary['by_split'].values()), summary['retained_source_rows'])
+ self.assertEqual(sum(x['decisions'] for x in summary['by_split'].values()), len(rows))
+ self.assertEqual(sum(x['groups'] for x in summary['by_split'].values()), len({r['group_id'] for r in rows}))
+
+ def test_english_delexicalized_template_overlap_quarantines_train(self):
+ self.add('overlap_train', 'train', template='Test only: set an alarm at [time : seven]')
+ self.add('overlap_test', 'test', template='Test only: set an alarm at [time : eleven]')
+ rows, diagnostics, summary = self.convert()
+ pairs = self.by_source(rows)
+ self.assertNotIn('overlap_train', pairs)
+ self.assertEqual(pairs['overlap_test']['choice']['split'], 'test')
+ excluded = next(r for r in diagnostics if r['source_id'] == 'overlap_train')
+ self.assertIn('shares_source_template_or_text_with_test', excluded['reasons'])
+ self.assertEqual(excluded['group_id'], pairs['overlap_test']['choice']['group_id'])
+ self.assertEqual(summary['retained_source_rows'] + summary['diagnostic_source_rows'], len(self.chinese))
+
+ def test_chinese_normalization_and_transitive_template_grouping(self):
+ self.add('chain_train', 'train', utterance='测试:ABC!', template='Synthetic bridge one [time : seven]')
+ self.add('chain_dev', 'dev', utterance='测 试 abc', template='Synthetic bridge two [time : eight]')
+ self.add('chain_test', 'test', utterance='仅供测试的第三句', template='Synthetic bridge two [time : ten]')
+ rows, diagnostics, _ = self.convert()
+ pairs = self.by_source(rows)
+ self.assertNotIn('chain_train', pairs)
+ self.assertNotIn('chain_dev', pairs)
+ owner_group = pairs['chain_test']['choice']['group_id']
+ excluded = [r for r in diagnostics if r['source_id'].startswith('chain_')]
+ self.assertEqual({r['source_id'] for r in excluded}, {'chain_train', 'chain_dev'})
+ self.assertTrue(all(r['group_id'] == owner_group for r in excluded))
+ self.assertTrue(all('shares_source_template_or_text_with_test' in r['reasons'] for r in excluded))
+
+ def test_duplicate_chinese_sources_in_one_partition_share_a_group(self):
+ self.add('duplicate_a', 'test', utterance='测试:相同句子。')
+ self.add('duplicate_b', 'test', utterance='测试 相同句子')
+ rows, diagnostics, summary = self.convert()
+ pairs = self.by_source(rows)
+ self.assertEqual(pairs['duplicate_a']['choice']['group_id'], pairs['duplicate_b']['choice']['group_id'])
+ self.assertEqual(summary['by_split']['test']['source_rows'], 3)
+ self.assertEqual(summary['by_split']['test']['groups'], 2)
+ self.assertEqual(diagnostics, [])
+
+ def test_conflicting_chinese_labels_are_retained_only_in_diagnostics(self):
+ self.add('conflict_a', 'test', intent='alarm_set', utterance='测试:相同但冲突的句子')
+ self.add('conflict_b', 'test', intent='alarm_remove', utterance='测试 相同但冲突的句子。')
+ rows, diagnostics, summary = self.convert()
+ pairs = self.by_source(rows)
+ self.assertNotIn('conflict_a', pairs)
+ self.assertNotIn('conflict_b', pairs)
+ excluded = [r for r in diagnostics if r['source_id'].startswith('conflict_')]
+ self.assertEqual(len(excluded), 2)
+ self.assertTrue(all('conflicting_intents_for_identical_normalized_chinese' in r['reasons'] for r in excluded))
+ self.assertEqual(summary['excluded_reasons']['conflicting_intents_for_identical_normalized_chinese'], 2)
+
+ def test_contested_or_missing_reviews_are_preserved_in_diagnostics(self):
+ for source_id, score in (('contested', 0), ('reasonable_only', 2), ('boolean_not_score', True)):
+ row, _ = self.add(source_id, 'test')
+ row['judgments'][2]['intent_score'] = score
+ missing, _ = self.add('missing_review', 'test')
+ del missing['judgments']
+ duplicate, _ = self.add('duplicate_review', 'test')
+ duplicate['judgments'][2] = copy.deepcopy(duplicate['judgments'][0])
+ original = copy.deepcopy(self.chinese)
+ rows, diagnostics, summary = self.convert()
+ expected = {'contested', 'reasonable_only', 'boolean_not_score', 'missing_review', 'duplicate_review'}
+ self.assertEqual({r['source_id'] for r in diagnostics}, expected)
+ self.assertTrue(expected.isdisjoint(self.by_source(rows)))
+ self.assertTrue(all('upstream_intent_review_not_unanimous_or_missing' in r['reasons'] for r in diagnostics))
+ source_lookup = {r['id']: r for r in original}
+ self.assertTrue(all(d['source_row'] == source_lookup[d['source_id']] for d in diagnostics))
+ self.assertEqual(self.chinese, original)
+ self.assertEqual(summary['diagnostic_source_rows'], len(expected))
+
+ def test_source_mismatch_is_rejected(self):
+ variants = [('locale', 'zh-TW'), ('partition', 'test'), ('intent', 'alarm_remove')]
+ for field, value in variants:
+ english = copy.deepcopy(self.english)
+ english[0][field] = value
+ with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'source mismatch'):
+ massive.convert(self.chinese, english, self.source)
+ with self.assertRaisesRegex(ValueError, 'source mismatch'):
+ massive.convert(self.chinese, self.english[1:], self.source)
+ chinese = copy.deepcopy(self.chinese)
+ chinese[0]['locale'] = 'zh-TW'
+ with self.assertRaisesRegex(ValueError, 'source mismatch'):
+ massive.convert(chinese, self.english, self.source)
+
+ def test_deterministic_rerun_and_source_input_not_mutated(self):
+ source_before = copy.deepcopy((self.chinese, self.english, self.source))
+ first = self.convert(seed=7, calibration_fraction=.5)
+ second = self.convert(seed=7, calibration_fraction=.5)
+ self.assertEqual(first, second)
+ self.assertEqual((self.chinese, self.english, self.source), source_before)
+ reversed_result = massive.convert(list(reversed(self.chinese)), list(reversed(self.english)), self.source,
+ seed=7, calibration_fraction=.5)
+ self.assertEqual(first, reversed_result)
+
+ def test_insufficient_groups_and_invalid_calibration_fraction_fail(self):
+ for fraction in (0, 1, -.1, 1.1):
+ with self.subTest(fraction=fraction), self.assertRaisesRegex(ValueError, 'calibration fraction'):
+ self.convert(calibration_fraction=fraction)
+ with self.assertRaisesRegex(ValueError, 'insufficient independent train groups'):
+ massive.convert(self.chinese[3:], self.english[3:], self.source)
+
+
+class MassiveSourceChecksTests(unittest.TestCase):
+ def setUp(self):
+ self.directory = tempfile.TemporaryDirectory()
+ self.addCleanup(self.directory.cleanup)
+ self.path = Path(self.directory.name)
+
+ def make_source(self):
+ source = self.path / 'source'
+ source.mkdir()
+ for name in massive.SOURCE_FILES:
+ text = 'FABRICATED TEST-ONLY DOCUMENT ' + name
+ if name == 'NOTICE.md':
+ text += '\nCC BY 4.0\n'
+ if name == 'DATA_LICENSE':
+ text += '\nAttribution 4.0 International\n'
+ (source / name).write_text(text, encoding='utf-8')
+ archive = source / 'massive-1.1.tar.gz'
+ archive.write_bytes(b'FABRICATED TEST-ONLY ARCHIVE')
+ metadata = {'archive_url': massive.ARCHIVE_URL, 'publisher_revision': massive.PUBLISHER_REVISION,
+ 'archive_sha256': file_hash(archive),
+ 'files': {name: file_hash(source / name) for name in massive.SOURCE_FILES}}
+ atomic_json(source / 'source_manifest.json', metadata)
+ return source, metadata
+
+ def test_source_file_and_archive_checksum_mismatch_are_rejected(self):
+ source, metadata = self.make_source()
+ self.assertEqual(massive.check_source(source), metadata)
+ text = (source / 'zh-CN.jsonl').read_bytes()
+ (source / 'zh-CN.jsonl').write_bytes(text + b'changed')
+ with self.assertRaisesRegex(ValueError, 'source file hash mismatch: zh-CN.jsonl'):
+ massive.check_source(source)
+ (source / 'zh-CN.jsonl').write_bytes(text)
+ (source / 'massive-1.1.tar.gz').write_bytes(b'changed archive')
+ with self.assertRaisesRegex(ValueError, 'source archive hash mismatch'):
+ massive.check_source(source)
+
+ def test_version_and_notice_mismatch_are_rejected(self):
+ source, metadata = self.make_source()
+ for field, value in (('archive_url', 'https://example.com/not-massive'), ('publisher_revision', 'main')):
+ changed = {**metadata, field: value}
+ atomic_json(source / 'source_manifest.json', changed)
+ with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'unexpected dataset version'):
+ massive.check_source(source)
+ (source / 'NOTICE.md').write_text('test-only document without required notice')
+ metadata['files']['NOTICE.md'] = file_hash(source / 'NOTICE.md')
+ atomic_json(source / 'source_manifest.json', metadata)
+ with self.assertRaisesRegex(ValueError, 'missing publisher dataset-license notice'):
+ massive.check_source(source)
+
+ @staticmethod
+ def archive_bytes(extra=None):
+ out = io.BytesIO()
+ members = [('release/data/zh-CN.jsonl', b'{"test_only":true}\n'),
+ ('release/data/en-US.jsonl', b'{"test_only":true}\n'),
+ ('release/LICENSE', b'FABRICATED TEST ONLY Attribution 4.0 International')]
+ if extra:
+ members.append(extra)
+ with tarfile.open(fileobj=out, mode='w:gz') as archive:
+ for name, value in members:
+ member = tarfile.TarInfo(name)
+ if value is None:
+ member.type = tarfile.SYMTYPE
+ member.linkname = '../../outside'
+ archive.addfile(member)
+ else:
+ member.size = len(value)
+ archive.addfile(member, io.BytesIO(value))
+ return out.getvalue()
+
+ def responses(self, archive):
+ def open_url(url, timeout):
+ self.assertGreater(timeout, 0)
+ if url == massive.ARCHIVE_URL:
+ return io.BytesIO(archive)
+ self.assertTrue(url.startswith(massive.RAW_DOCS))
+ return io.BytesIO(b'FABRICATED TEST-ONLY DOCUMENT CC BY 4.0')
+ return open_url
+
+ def test_download_reads_selected_regular_members_without_extracting_paths(self):
+ archive = self.archive_bytes(('../outside', b'not allowed outside source'))
+ destination = self.path / 'download'
+ with mock.patch.object(massive.urllib.request, 'urlopen', side_effect=self.responses(archive)):
+ massive.fetch_source(destination)
+ self.assertFalse((self.path / 'outside').exists())
+ self.assertFalse((destination / 'release').exists())
+ self.assertEqual((destination / 'zh-CN.jsonl').read_text(), '{"test_only":true}\n')
+ self.assertEqual(massive.check_source(destination)['archive_sha256'], file_hash(destination / 'massive-1.1.tar.gz'))
+
+ def test_download_rejects_selected_symlinks_and_duplicate_members(self):
+ for i, extra in enumerate((('other/data/zh-CN.jsonl', None),
+ ('other/data/en-US.jsonl', b'duplicate test-only member'))):
+ archive = self.archive_bytes(extra)
+ with self.subTest(extra=extra), mock.patch.object(massive.urllib.request, 'urlopen',
+ side_effect=self.responses(archive)):
+ with self.assertRaisesRegex(ValueError, 'unexpected/duplicate archive member'):
+ massive.fetch_source(self.path / ('download_' + str(i)))
+
+
+class MassiveParquetTests(unittest.TestCase):
+ """Fabricated HF-shaped records and transport mocks; no Parquet library or network."""
+ def setUp(self):
+ self.directory = tempfile.TemporaryDirectory()
+ self.addCleanup(self.directory.cleanup)
+ self.path = Path(self.directory.name)
+ self.features = {
+ 'scenario': {'_type': 'ClassLabel', 'names': ['music', 'alarm']},
+ 'intent': {'_type': 'ClassLabel', 'names': ['alarm_query', 'alarm_remove', 'alarm_set']},
+ }
+ self.row = {
+ 'id': 'test-only-1', 'locale': 'zh-CN', 'partition': 'dev', 'scenario': 1, 'intent': 2,
+ 'utt': '仅供测试:请设置AM七点的闹钟!',
+ 'annot_utt': '仅供测试:请设置[time : AM七点]的闹钟!',
+ 'judgments': {'worker_id': ['0', '1', '2'], 'intent_score': [1, 1, 1],
+ 'grammar_score': [3, 4, 3]},
+ 'slot_method': {'slot': ['time'], 'method': ['localization']},
+ }
+
+ def decode(self, row=None, features=None):
+ return massive.decode_parquet_row(self.row if row is None else row,
+ self.features if features is None else features,
+ 'zh-CN', 'validation')
+
+ def test_decoder_uses_embedded_labels_and_transposes_sequences_without_text_changes(self):
+ original = copy.deepcopy(self.row)
+ result = self.decode()
+ self.assertEqual(result['scenario'], 'alarm')
+ self.assertEqual(result['intent'], 'alarm_set')
+ self.assertEqual(result['utt'], original['utt'])
+ self.assertEqual(result['annot_utt'], original['annot_utt'])
+ self.assertEqual(result['partition'], 'dev')
+ self.assertEqual(result['judgments'], [
+ {'worker_id': '0', 'intent_score': 1, 'grammar_score': 3},
+ {'worker_id': '1', 'intent_score': 1, 'grammar_score': 4},
+ {'worker_id': '2', 'intent_score': 1, 'grammar_score': 3},
+ ])
+ self.assertEqual(result['slot_method'], [{'slot': 'time', 'method': 'localization'}])
+ self.assertEqual(self.row, original)
+ self.row['slot_method'] = {'slot': [], 'method': []}
+ self.assertEqual(self.decode()['slot_method'], [])
+
+ def test_decoder_rejects_invalid_label_indices_and_feature_type(self):
+ for field in ('scenario', 'intent'):
+ for value in (-1, len(self.features[field]['names']), True, 1.0, '1', None):
+ row = copy.deepcopy(self.row)
+ row[field] = value
+ with self.subTest(field=field, value=value), self.assertRaisesRegex(ValueError, 'invalid parquet class label'):
+ self.decode(row)
+ features = copy.deepcopy(self.features)
+ features[field]['_type'] = 'Value'
+ with self.assertRaisesRegex(ValueError, 'invalid parquet class label'):
+ self.decode(features=features)
+
+ def test_decoder_rejects_wrong_locale_or_partition(self):
+ for field, value in (('locale', 'en-US'), ('partition', 'validation'), ('partition', 'test')):
+ row = copy.deepcopy(self.row)
+ row[field] = value
+ with self.subTest(field=field, value=value), self.assertRaisesRegex(ValueError, 'locale/partition mismatch'):
+ self.decode(row)
+ row = copy.deepcopy(self.row)
+ row['partition'] = 'train'
+ self.assertEqual(massive.decode_parquet_row(row, self.features, 'zh-CN', 'train')['partition'], 'train')
+
+ def test_decoder_rejects_malformed_or_unequal_sequence_columns(self):
+ for field in ('judgments', 'slot_method'):
+ for value in (None, [], {}, {'one': 'not a list'}, {'one': [1], 'two': [1, 2]}):
+ row = copy.deepcopy(self.row)
+ row[field] = value
+ with self.subTest(field=field, value=value), self.assertRaisesRegex(ValueError, 'parquet sequence'):
+ self.decode(row)
+
+ def make_source(self):
+ directory = self.path / 'source'
+ directory.mkdir()
+ for name in massive.SOURCE_FILES + massive.PARQUET_FILES:
+ text = 'FABRICATED TEST-ONLY SOURCE ' + name
+ if name == 'NOTICE.md':
+ text += ' CC BY 4.0'
+ if name == 'DATA_LICENSE':
+ text += ' Attribution 4.0 International'
+ (directory / name).write_text(text)
+ metadata = {
+ 'archive_url': massive.ARCHIVE_URL, 'publisher_revision': massive.PUBLISHER_REVISION,
+ 'transport': 'hf_parquet', 'mirror_revision': massive.PARQUET_REVISION,
+ 'mirror_base_url': massive.PARQUET_BASE, 'license_text_url': massive.LICENSE_TEXT_URL,
+ 'files': {name: file_hash(directory / name) for name in massive.SOURCE_FILES + massive.PARQUET_FILES},
+ }
+ atomic_json(directory / 'source_manifest.json', metadata)
+ return directory, metadata
+
+ def test_source_check_requires_fixed_mirror_metadata_and_parquet_hashes(self):
+ directory, metadata = self.make_source()
+ self.assertEqual(massive.check_source(directory), metadata)
+ for field, value in (('mirror_revision', 'main'), ('mirror_base_url', 'https://example.com/'),
+ ('license_text_url', 'https://example.com/LICENSE')):
+ atomic_json(directory / 'source_manifest.json', {**metadata, field: value})
+ with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'unexpected parquet source revision'):
+ massive.check_source(directory)
+ atomic_json(directory / 'source_manifest.json', metadata)
+ for name in massive.PARQUET_FILES:
+ target = directory / name
+ original = target.read_bytes()
+ target.write_bytes(original + b'CHANGED')
+ with self.subTest(name=name), self.assertRaisesRegex(ValueError, 'source file hash mismatch'):
+ massive.check_source(directory)
+ target.write_bytes(original)
+
+ def test_mirror_fetch_uses_pinned_urls_and_publishes_completed_files(self):
+ directory = self.path / 'download'
+ expected = {f'{locale}-{split}.parquet': f'{massive.PARQUET_BASE}{locale}/{split}/0000.parquet'
+ for locale in ('zh-CN', 'en-US') for split in ('train', 'validation', 'test')}
+ expected.update({name: massive.RAW_DOCS + name for name in massive.SOURCE_FILES[3:]})
+ expected['DATA_LICENSE'] = massive.LICENSE_TEXT_URL
+ def open_url(url, timeout):
+ self.assertIn(url, expected.values())
+ self.assertGreater(timeout, 0)
+ return io.BytesIO(('FABRICATED TEST-ONLY ' + url).encode())
+ with mock.patch.object(massive.urllib.request, 'urlopen', side_effect=open_url) as download, \
+ mock.patch.object(massive, 'prepare_parquet_source', return_value=directory) as prepare:
+ self.assertEqual(massive.fetch_parquet_source(directory), directory)
+ self.assertEqual(download.call_count, len(expected))
+ prepare.assert_called_once_with(directory)
+ self.assertEqual({p.name for p in directory.iterdir()}, set(expected))
+ self.assertFalse(list(directory.glob('*.part')))
+ for name, url in expected.items():
+ self.assertEqual((directory / name).read_bytes(), ('FABRICATED TEST-ONLY ' + url).encode())
+
+ def test_interrupted_mirror_download_is_not_published_or_prepared(self):
+ class InterruptedStream(io.BytesIO):
+ def read(self, size=-1):
+ data = super().read(size)
+ if not data:
+ raise OSError('fabricated download interruption')
+ return data
+ directory = self.path / 'download'
+ with mock.patch.object(massive.urllib.request, 'urlopen', return_value=InterruptedStream(b'test only')), \
+ mock.patch.object(massive, 'prepare_parquet_source') as prepare:
+ with self.assertRaisesRegex(OSError, 'fabricated download interruption'):
+ massive.fetch_parquet_source(directory)
+ prepare.assert_not_called()
+ self.assertFalse((directory / massive.PARQUET_FILES[0]).exists())
+ self.assertEqual((directory / (massive.PARQUET_FILES[0] + '.part')).read_bytes(), b'test only')
+
+
+if __name__ == '__main__':
+ unittest.main(verbosity=2)
diff --git a/tests/test_zh_reliability.py b/tests/test_zh_reliability.py
new file mode 100644
index 00000000..e58ee703
--- /dev/null
+++ b/tests/test_zh_reliability.py
@@ -0,0 +1,383 @@
+"""Offline logic tests; tiny random BERT is NOT a real multilingual Laya experiment."""
+import copy
+import contextlib
+import io
+import json
+import math
+import os
+from pathlib import Path
+import sys
+import tempfile
+import unittest
+from unittest.mock import patch
+
+os.environ.setdefault('USE_TF', '0')
+os.environ.setdefault('USE_TORCH', '1')
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+import numpy as np
+import torch
+from safetensors.torch import save_file
+from tokenizers import Tokenizer
+from tokenizers.models import WordLevel
+from tokenizers.pre_tokenizers import Whitespace
+from transformers import BertConfig, BertModel, PreTrainedTokenizerFast
+
+from research.benchmarks.zh_reliability.calibration import calibrate, fit_temperature, validate_calibration
+from research.benchmarks.zh_reliability.data import FIXTURES, atomic_json, options, partition, read_data
+from research.benchmarks.zh_reliability.metrics import log_probs, paired_bootstrap, probability_record, quality, summarize
+from research.benchmarks.zh_reliability.runtime import Adapter, ensure_device, select_device
+from laya.agent import Agent
+from laya.common import DecisionModel, collate_items
+
+
+class DataMetricsTests(unittest.TestCase):
+ def setUp(self):
+ self.rows = read_data(FIXTURES)
+ self.split = partition(self.rows)
+ self.model = {'content_hash': 'mock-only', 'trained': False}
+
+ def prediction(self, row=None, logits=(0., 0.), t=1.):
+ row = row or self.rows[0]
+ p = probability_record(row, logits, t)
+ p.update(split=self.split['assignments'][row['id']], model=self.model)
+ return p
+
+ def test_schema_groups_manifest_and_relations(self):
+ self.assertEqual(len(self.rows), 80)
+ self.assertEqual(len(self.split['groups']), 40)
+ self.assertEqual(self.split['counts'], {'test': 12, 'train': 48, 'dev': 8, 'calibration': 12})
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / 'split.json'
+ saved = partition(self.rows, path)
+ self.assertEqual(partition(self.rows, path, seed=1), saved)
+ changed = copy.deepcopy(self.rows)
+ changed[0]['state'] += '!'
+ with self.assertRaisesRegex(ValueError, 'hash'):
+ partition(changed, path)
+ saved['assignments'][self.rows[0]['id']] = 'train'
+ atomic_json(path, saved)
+ with self.assertRaisesRegex(ValueError, 'leakage'):
+ partition(self.rows, path)
+
+ def test_schema_rejections(self):
+ for change in ({'label': 'unknown'}, {'provenance': {}}, {'tags': 'negation'},
+ {'question': {'type': 'score', 'instructions': 'bad'}},
+ {'provenance': {'kind': 'agent_authored', 'human_verified': True}}):
+ with self.subTest(change=change), tempfile.TemporaryDirectory() as tmp:
+ rows = copy.deepcopy(self.rows)
+ rows[0].update(change)
+ path = Path(tmp) / 'data.jsonl'
+ path.write_text('\n'.join(json.dumps(r) for r in rows))
+ with self.assertRaises(ValueError):
+ read_data(path)
+ with self.assertRaises(ValueError):
+ read_data(FIXTURES, formal=True)
+
+ def test_template_family_may_not_cross_groups(self):
+ rows = copy.deepcopy(self.rows)
+ rows[2]['provenance']['template_family'] = rows[0]['provenance']['template_family']
+ with tempfile.TemporaryDirectory() as tmp:
+ p = Path(tmp) / 'data.jsonl'
+ p.write_text('\n'.join(json.dumps(r) for r in rows))
+ with self.assertRaisesRegex(ValueError, 'family crosses'):
+ read_data(p)
+
+ def test_formal_requires_fixed_licensed_verified_data(self):
+ rows = copy.deepcopy(self.rows)
+ for r in rows:
+ r['provenance'].update(kind='user_supplied', human_verified=True, source='test source',
+ license='CC0', annotation_status='double reviewed')
+ r['split'] = self.split['assignments'][r['id']]
+ with tempfile.TemporaryDirectory() as tmp:
+ p = Path(tmp) / 'real.jsonl'
+ p.write_text('\n'.join(json.dumps(r) for r in rows))
+ accepted = read_data(p, formal=True)
+ self.assertEqual(partition(accepted, formal=True)['assignments'], self.split['assignments'])
+ accepted[0]['split'] = 'train'
+ with self.assertRaisesRegex(ValueError, 'split a group'):
+ partition(accepted, formal=True)
+
+ def test_choice_permutation_preserves_semantic_mapping(self):
+ row = self.rows[40]
+ before = probability_record(row, [4., 3., 2., 1.], 1.)
+ perm = copy.deepcopy(row)
+ perm['question']['criteria'] = dict(reversed(list(perm['question']['criteria'].items())))
+ after = probability_record(perm, [1., 2., 3., 4.], 1.)
+ self.assertEqual(before['predicted'], after['predicted'])
+ self.assertEqual(before['gold'], after['gold'])
+ for key, value in zip(before['option_order'], before['probabilities']):
+ self.assertAlmostEqual(value, dict(zip(after['option_order'], after['probabilities']))[key])
+
+ def test_noul_probability_is_true_despite_display_labels(self):
+ row = copy.deepcopy(self.rows[0])
+ row['question']['labels'] = {'false': '拒绝', 'true': '接受'}
+ row['question']['criteria'] = dict(reversed(list(row['question']['criteria'].items())))
+ p = self.prediction(row, [0., math.log(3)])
+ self.assertEqual(p['option_order'], ['false', 'true'])
+ self.assertAlmostEqual(p['p_true'], .75)
+ self.assertAlmostEqual(p['confidence'], .75)
+
+ def test_valid_mask_and_temperature_invariance(self):
+ expected = log_probs([1., 2.], 1.)
+ np.testing.assert_allclose(expected, log_probs([1., 99., 2.], 1., [True, False, True]))
+ np.testing.assert_allclose(np.exp(expected), torch.softmax(torch.tensor([1., 2.], dtype=torch.float64), 0))
+ for t in (.5, 1., 2., 5.):
+ self.assertEqual(log_probs([1., 4., -2.], t).argmax(), 1)
+ for t in (0., -.1, .1, 6., float('nan'), float('inf')):
+ with self.assertRaises(ValueError):
+ log_probs([0., 1.], t)
+
+ def test_hand_computable_nll_brier_ece_and_confidence(self):
+ p = self.prediction(logits=(0., math.log(3))) # gold true, correct with p=.75
+ m = quality([p])
+ self.assertAlmostEqual(m['nll'], -math.log(.75))
+ self.assertAlmostEqual(m['brier'], .125)
+ self.assertAlmostEqual(m['ece'], .25)
+ self.assertAlmostEqual(p['confidence'], .75) # not 1 - entropy/log(2)
+ self.assertEqual(m['accuracy'], 1.)
+ uniform = self.prediction()
+ self.assertEqual(uniform['confidence'], .5)
+ self.assertIsNone(quality([])['nll'])
+ json.dumps(summarize([]), allow_nan=False)
+
+ def test_nll_does_not_underflow_via_probabilities(self):
+ p = self.prediction(logits=(10000., -10000.))
+ self.assertEqual(quality([p])['nll'], 20000.)
+
+ def test_risk_ties_do_not_depend_on_labels(self):
+ a, b = self.prediction(), self.prediction()
+ a.update(id='a', gold='false')
+ b.update(id='b', gold='true')
+ self.assertEqual(quality([b, a])['risk_coverage'][0]['risk'], 0)
+ a['gold'], b['gold'] = 'true', 'false'
+ self.assertEqual(quality([b, a])['risk_coverage'][0]['risk'], 1)
+ self.assertEqual(quality([a, b])['risk_coverage'], quality([b, a])['risk_coverage'])
+
+ def test_task_local_macro_f1(self):
+ a, b = self.prediction(), self.prediction()
+ a.update(task_id='a', gold='false', predicted='false')
+ b.update(task_id='b', gold='true', predicted='true')
+ # Each task has one perfectly predicted class and one absent class: F1=.5 each.
+ self.assertEqual(quality([a, b])['macro_f1'], .5)
+
+ def calibration_rows(self):
+ return [self.prediction(r, [0.] * len(options(r))) for r in self.rows
+ if self.split['assignments'][r['id']] == 'calibration']
+
+ def test_fit_export_reload_no_double_scaling_or_bucket_shadowing(self):
+ rows = self.calibration_rows()
+ original = {'temperature': [2., 3., 4.], 'temperature_by_options': {'choice:3-5': 5., 'score:2': 2.},
+ 'lang_temperatures': {}}
+ a = calibrate(rows, self.model, self.split, original, minimum=1, smoke=True)
+ with tempfile.TemporaryDirectory() as tmp:
+ p = Path(tmp) / 'cal.json'
+ atomic_json(p, a)
+ a = json.loads(p.read_text())
+ cfg = validate_calibration(a, self.model, self.split, True)
+ self.assertEqual(cfg['zh']['temperature'][1], 3.)
+ self.assertNotIn('choice:3-5', cfg['zh']['temperature_by_options'])
+ self.assertEqual(cfg['zh']['temperature_by_options']['score:2'], 2.)
+ altered = copy.deepcopy(rows)
+ for r in altered:
+ r['temperature'] = 5.
+ r['probabilities'] = [0.] * len(r['raw_logits'])
+ self.assertEqual(fit_temperature(rows, 1), fit_temperature(altered, 1))
+ with self.assertRaisesRegex(ValueError, 'smoke'):
+ validate_calibration(a, self.model, self.split)
+ with self.assertRaisesRegex(ValueError, 'model or data'):
+ validate_calibration(a, {'content_hash': 'other'}, self.split, True)
+ bad = copy.deepcopy(self.split)
+ bad['data_hash'] = 'wrong'
+ with self.assertRaises(ValueError):
+ validate_calibration(a, self.model, bad, True)
+ bad = copy.deepcopy(a)
+ bad['calibration_hash'] = 'wrong'
+ with self.assertRaises(ValueError):
+ validate_calibration(bad, self.model, self.split, True)
+ bad = copy.deepcopy(a)
+ bad['lang_temperatures']['zh']['temperature'][0] = .1
+ with self.assertRaises(ValueError):
+ validate_calibration(bad, self.model, self.split, True)
+
+ def test_fit_bounds_skipped_and_partition_guard(self):
+ rows = self.calibration_rows()
+ self.assertEqual(fit_temperature(rows)['status'], 'SKIPPED')
+ a = self.prediction(logits=(10., -10.))
+ result = fit_temperature([a] * 30)
+ self.assertEqual(result['temperature'], 5.)
+ self.assertTrue(result['boundary_hit'])
+ self.assertLessEqual(result['nll_fitted'], result['nll_t1'])
+ rows[0]['split'] = 'test'
+ with self.assertRaisesRegex(ValueError, 'calibration partition'):
+ calibrate(rows, self.model, self.split, {})
+
+ def test_bootstrap_pairs_and_insufficient_groups(self):
+ rows = self.calibration_rows()
+ self.assertEqual(paired_bootstrap(rows, rows)['status'], 'SKIPPED')
+ result = paired_bootstrap(rows, rows, repeats=10, min_groups=2)
+ self.assertEqual(result['after_minus_before']['nll']['ci95'], [0., 0.])
+ with self.assertRaises(ValueError):
+ paired_bootstrap(rows, rows[:-1])
+
+ def test_visible_device_selector_and_cpu_fallback(self):
+ env = {'gpus': [{'name': 'TITAN RTX', 'process_index': 0, 'uuid': 'GPU-b', 'status': 'SUCCESS'},
+ {'name': 'RTX 2080 Ti', 'process_index': 1, 'uuid': 'GPU-a', 'status': 'FAILED'}]}
+ self.assertEqual(select_device('GPU-b', env), 'cuda:0')
+ with self.assertRaises(RuntimeError):
+ select_device('2080', env)
+ with self.assertRaises(RuntimeError):
+ select_device('physical:0', env)
+ fake = type('A', (), {'device': torch.device('cpu')})()
+ with self.assertRaisesRegex(RuntimeError, 'fallback'):
+ ensure_device(fake, 'cuda:0')
+
+ def test_cli_failure_is_nonzero_and_cannot_reuse_success(self):
+ from research.benchmarks.zh_reliability.__main__ import main
+ with tempfile.TemporaryDirectory() as tmp:
+ run = Path(tmp) / 'failed'
+ with patch('research.benchmarks.zh_reliability.__main__.environment', return_value={'gpus': [], 'gpu_status': 'NOT_RUN'}):
+ self.assertEqual(main(['import-data', '--data', str(run / 'missing'), '--run-dir', str(run)]), 1)
+ m = json.loads((run / 'manifest.json').read_text())
+ self.assertEqual(m['status'], 'FAILED')
+ self.assertEqual(m['error']['stage'], 'data')
+ self.assertEqual(main(['import-data', '--run-dir', str(run)]), 2)
+
+ def test_no_cuda_smoke_and_report_saved_results(self):
+ from research.benchmarks.zh_reliability.__main__ import main
+ with tempfile.TemporaryDirectory() as tmp:
+ run, out = Path(tmp) / 'no-gpu', Path(tmp) / 'report'
+ with patch('research.benchmarks.zh_reliability.__main__.environment', return_value={'gpus': [], 'gpu_status': 'NOT_RUN'}):
+ self.assertEqual(main(['smoke', '--gpu', '--run-dir', str(run)]), 0)
+ self.assertEqual(json.loads((run / 'manifest.json').read_text())['status'], 'SKIPPED')
+ self.assertEqual(main(['report', '--source', str(run), '--run-dir', str(out)]), 0)
+ self.assertEqual((run / 'split_manifest.json').read_bytes(), (out / 'split_manifest.json').read_bytes())
+ self.assertEqual((run / 'metrics.json').read_bytes(), (out / 'metrics.json').read_bytes())
+ self.assertIn('SKIPPED', (out / 'report.md').read_text())
+
+ def test_all_cli_help(self):
+ from research.benchmarks.zh_reliability.__main__ import parser
+ for name in ('check-env', 'import-data', 'smoke', 'eval', 'calibrate', 'report'):
+ with self.subTest(name=name), contextlib.redirect_stdout(io.StringIO()), self.assertRaises(SystemExit) as e:
+ parser().parse_args([name, '--help'])
+ self.assertEqual(e.exception.code, 0)
+
+
+class TinyEvaluationTests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ torch.set_num_threads(1)
+ cls.tmp = tempfile.TemporaryDirectory()
+ cls.addClassCleanup(cls.tmp.cleanup)
+ cls.base = Path(cls.tmp.name) / 'tiny'
+ cls.base.mkdir()
+ torch.manual_seed(7)
+ config = BertConfig(vocab_size=16, hidden_size=16, num_hidden_layers=1,
+ num_attention_heads=1, intermediate_size=32)
+ config.save_pretrained(cls.base / 'encoder')
+ backend = Tokenizer(WordLevel({'[PAD]': 0, '[UNK]': 1, '[CLS]': 2, '[SEP]': 3, '[MASK]': 4,
+ 'false': 5, 'true': 6, 'billing': 7, 'technical': 8,
+ 'shipping': 9, 'other': 10}, unk_token='[UNK]'))
+ backend.pre_tokenizer = Whitespace()
+ tok = PreTrainedTokenizerFast(tokenizer_object=backend, pad_token='[PAD]', unk_token='[UNK]',
+ cls_token='[CLS]', sep_token='[SEP]', mask_token='[MASK]')
+ tok.save_pretrained(cls.base / 'tokenizer')
+ model = DecisionModel(BertModel(config), head_layers=1)
+ save_file(model.state_dict(), cls.base / 'model.safetensors')
+ atomic_json(cls.base / 'rl_agent_config.json', {'encoder': 'tiny-test-only', 'head_layers': 1,
+ 'act_costs': {'act': 0}, 'temperature': [2., 1., 3.],
+ 'temperature_by_options': {'choice:3-5': 1.5}, 'max_len': 128, 'head_max_len': 64})
+ cls.rows = read_data(FIXTURES)
+ cls.split = partition(cls.rows)
+
+ def adapter(self):
+ return Adapter(Agent(str(self.base), device='cpu', fast=False, compile=False),
+ {'model_id': 'tiny-mock-only', 'revision': None, 'content_hash': 'tiny', 'trained': False},
+ max_len=128, head_max_len=64, dtype='fp32')
+
+ def test_raw_forward_official_parity_and_mask(self):
+ a = self.adapter()
+ result = a.parity([self.rows[0], self.rows[40]])
+ self.assertLessEqual(result['max_absolute_error'], .000051)
+ items = [a.prepare(r)[0] for r in (self.rows[0], self.rows[40])]
+ b = collate_items([items], a.agent.tok.pad_token_id)
+ self.assertEqual(b['marker_mask'].tolist(), [[True, True, False, False], [True] * 4])
+ a.agent.model.eval()
+ with torch.no_grad():
+ z = a.forward(b)
+ self.assertTrue(torch.equal(z[0, 2:], torch.full((2,), -1e4)))
+
+ def test_mock_decision_not_action_and_noul_display_labels(self):
+ a = self.adapter()
+ row = copy.deepcopy(self.rows[0])
+ row['question']['labels'] = {'false': '不', 'true': '是'}
+ with patch.object(a.agent.model, 'forward', return_value=(torch.tensor([[0., 3.]]), torch.tensor([[999., -999.]]))):
+ p = a.predict([row], 'E0', self.split)[0]
+ self.assertAlmostEqual(p['p_true'], 1 / (1 + math.exp(-1)))
+ a.parity([row])
+
+ def test_calibrated_evaluation_keeps_logits_and_restores_baseline(self):
+ from research.benchmarks.zh_reliability.__main__ import evaluate
+ adapter = self.adapter()
+ weights = {name: value.clone() for name, value in adapter.agent.model.state_dict().items()}
+ baseline, base_metrics = evaluate(adapter, self.rows, self.split)
+ calibration_rows, _ = evaluate(adapter, self.rows, self.split, selection='calibration')
+ artifact = calibrate(calibration_rows, adapter.identity, self.split, adapter.original,
+ minimum=1, smoke=True)
+ calibrated, calibrated_metrics = evaluate(adapter, self.rows, self.split, artifact, smoke=True)
+ self.assertTrue(all(row['condition'] == 'E0' and row['split'] == 'test' for row in baseline))
+ self.assertTrue(all(row['condition'] == 'E1' and row['split'] == 'test' for row in calibrated))
+ self.assertEqual([row['raw_logits'] for row in baseline], [row['raw_logits'] for row in calibrated])
+ self.assertEqual([row['predicted'] for row in baseline], [row['predicted'] for row in calibrated])
+ self.assertEqual(base_metrics['overall']['accuracy'], calibrated_metrics['overall']['accuracy'])
+ for before, after in zip(baseline, calibrated):
+ fitted = artifact['fitted'][after['question']['type']]['temperature']
+ np.testing.assert_allclose(after['log_probabilities'], log_probs(before['raw_logits'], fitted))
+ restored, restored_metrics = evaluate(adapter, self.rows, self.split)
+ self.assertEqual(restored, baseline)
+ self.assertEqual(restored_metrics, base_metrics)
+ self.assertTrue(all(torch.equal(value, weights[name])
+ for name, value in adapter.agent.model.state_dict().items()))
+
+ def test_official_parity_rejects_restored_device_cpu_fallback(self):
+ adapter = self.adapter()
+ original = adapter.agent.system_one
+ def fallback(*args, **kwargs):
+ result = original(*args, **kwargs)
+ adapter.agent.cpu_fallback_count += 1
+ return result
+ with patch.object(adapter.agent, 'system_one', side_effect=fallback):
+ with self.assertRaisesRegex(RuntimeError, 'fallback'):
+ adapter.parity(self.rows[:1])
+
+ def test_token_lengths_state_truncation_and_missing_marker(self):
+ a = self.adapter()
+ r = copy.deepcopy(self.rows[0])
+ r['state'] = 'hello ' * 500
+ item, tokens = a.prepare(r)
+ self.assertTrue(tokens['state_truncated'])
+ self.assertEqual(tokens['input_tokens'], 128)
+ self.assertLess(tokens['state_retained'], tokens['state_tokens'])
+ a.max_len = 4
+ with self.assertRaises(ValueError):
+ a.prepare(r)
+
+
[email protected](os.environ.get('LAYA_ZH_REAL_MODEL'), 'opt-in: set LAYA_ZH_REAL_MODEL to multilingual local weights')
+class RealCheckpointTests(unittest.TestCase):
+ def test_official_parity(self):
+ if not torch.cuda.is_available():
+ self.skipTest('CUDA unavailable')
+ # Keep opt-in load within a private copy; never let Agent patch a shared snapshot.
+ from research.benchmarks.zh_reliability.runtime import resolve_model
+ with tempfile.TemporaryDirectory() as tmp:
+ source, identity = resolve_model(os.environ['LAYA_ZH_REAL_MODEL'], 'main', True, tmp)
+ agent = Agent(str(source), device='cuda:0', fast=False, compile=False)
+ ensure_device(agent, 'cuda:0')
+ a = Adapter(agent, identity, 256, 192, 'fp16')
+ rows = read_data(FIXTURES)
+ a.parity(rows[:2] + rows[40:42])
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/tests/test_zh_results.py b/tests/test_zh_results.py
new file mode 100644
index 00000000..35931d15
--- /dev/null
+++ b/tests/test_zh_results.py
@@ -0,0 +1,95 @@
+"""Recompute the published E0/E1 evidence and reject inconsistent archived results."""
+import copy
+import json
+from pathlib import Path
+import sys
+import unittest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from research.benchmarks.zh_reliability.aggregate_massive import render_report, verify_artifact
+from research.benchmarks.zh_reliability.data import digest
+
+RESULTS = Path(__file__).resolve().parents[1] / 'research/benchmarks/zh_reliability/results'
+
+
+class PublishedResultsTests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ cls.artifact = json.loads((RESULTS / 'massive_alarm_2026-09-24.json').read_text(encoding='utf-8'))
+ cls.result = verify_artifact(cls.artifact)
+
+ def altered(self):
+ return copy.deepcopy(self.artifact)
+
+ def test_published_table_recomputes_from_decisions(self):
+ self.assertEqual(self.result['counts']['test_source_utterances'], 93)
+ self.assertEqual(self.result['counts']['test_groups'], 89)
+ self.assertEqual(self.result['counts']['test_decisions'], 186)
+ before, after = (self.result[c]['overall'] for c in ('E0', 'E1'))
+ self.assertAlmostEqual(before['accuracy'], 99 / 186)
+ self.assertEqual(before['accuracy'], after['accuracy'])
+ self.assertEqual(before['macro_f1'], after['macro_f1'])
+ self.assertAlmostEqual(before['nll'], 1.383747, places=6)
+ self.assertAlmostEqual(after['nll'], .785033, places=6)
+ self.assertEqual(self.result['fitted']['noul']['temperature'], 5.)
+ self.assertTrue(self.result['fitted']['noul']['boundary_hit'])
+ paired = self.result['paired']['E1-E0']
+ self.assertEqual(paired['repeats'], 1000)
+ self.assertEqual(paired['after_minus_before']['accuracy']['ci95'], [0., 0.])
+ self.assertLess(paired['after_minus_before']['nll']['ci95'][1], 0.)
+ self.assertEqual(render_report(self.artifact, self.result),
+ (RESULTS / 'massive_alarm_2026-09-24.md').read_text(encoding='utf-8'))
+
+ def test_records_hash_rejects_changed_logits(self):
+ artifact = self.altered()
+ artifact['records'][0]['raw_logits'][0] += 1.
+ with self.assertRaisesRegex(ValueError, 'record hash mismatch'):
+ verify_artifact(artifact, bootstrap=False)
+
+ def test_rehashed_logits_must_match_between_conditions(self):
+ artifact = self.altered()
+ record = next(r for r in artifact['records'] if r['split'] == 'test')
+ record['calibrated_raw_logits'][0] += 1.
+ artifact['records_sha256'] = digest(artifact['records'])
+ with self.assertRaisesRegex(ValueError, 'changed raw decision logits'):
+ verify_artifact(artifact, bootstrap=False)
+
+ def test_rehashed_duplicate_ids_are_rejected(self):
+ artifact = self.altered()
+ artifact['records'][1]['id'] = artifact['records'][0]['id']
+ artifact['records_sha256'] = digest(artifact['records'])
+ with self.assertRaisesRegex(ValueError, 'duplicate decision IDs'):
+ verify_artifact(artifact, bootstrap=False)
+
+ def test_rehashed_cross_partition_group_is_rejected(self):
+ artifact = self.altered()
+ calibration = next(r for r in artifact['records'] if r['split'] == 'calibration')
+ record = next(r for r in artifact['records'] if r['split'] == 'test')
+ record['group_id'] = calibration['group_id']
+ artifact['records_sha256'] = digest(artifact['records'])
+ with self.assertRaisesRegex(ValueError, 'group leakage'):
+ verify_artifact(artifact, bootstrap=False)
+
+ def test_source_pair_hash_must_match(self):
+ artifact = self.altered()
+ artifact['records'][0]['source_row_hash'] = '0' * 64
+ artifact['records_sha256'] = digest(artifact['records'])
+ with self.assertRaisesRegex(ValueError, 'source utterance split/group/hash mismatch'):
+ verify_artifact(artifact, bootstrap=False)
+
+ def test_temperature_is_refitted_from_calibration_only(self):
+ artifact = self.altered()
+ artifact['calibration']['fitted']['choice']['temperature'] += .1
+ with self.assertRaisesRegex(ValueError, 'fitted temperature/objective mismatch'):
+ verify_artifact(artifact, bootstrap=False)
+
+ def test_stored_metric_cannot_override_recomputed_value(self):
+ artifact = self.altered()
+ artifact['summary']['E1']['overall']['accuracy'] = 1.
+ with self.assertRaisesRegex(ValueError, 'stored summary mismatch: E1'):
+ verify_artifact(artifact, bootstrap=False)
+
+
+if __name__ == '__main__':
+ unittest.main()