Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

Revert "Merge pull request #392 from Fanrito/feat/chinese-reliability-benchmark"

Nandakishor2c736329c2ac
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index bc30a389..c97f733d 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -156,7 +156,6 @@ jobs:
python tests/test_langchain.py
python tests/test_portability.py
python tests/test_training.py
- python -m unittest discover -s tests -p 'test_zh_*.py' -v
python tests/test_example_server_limits.py
python tests/test_blank_lang_routing.py
python tests/test_export_onnx_safety.py
@@ -369,9 +368,9 @@ jobs:
python-version: "3.11"
- run: pip install ruff
- name: Ruff
- run: ruff check laya/ research/benchmarks/zh_reliability tests/test_zh_*.py --select=E9,F63,F7,F82,F401,F811 --line-length=120
+ run: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
- name: Byte-compile every module
- run: python -m compileall -q laya/ tests/ research/benchmarks/zh_reliability
+ run: python -m compileall -q laya/ tests/
typescript-sdk:
name: TypeScript SDK (node${{ matrix.node }})
diff --git a/.gitignore b/.gitignore
index 42c9377a..75442a6f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -80,6 +80,3 @@ research/*_benchmark_results.json
# local checkpoint weights, kept out of the tree
# (fetch them with: python verify/checkpoints.py --fetch)
models/
-
-# Local Chinese reliability runs (model copies, data, diagnostics)
-/runs/
diff --git a/research/benchmarks/zh_reliability/README.md b/research/benchmarks/zh_reliability/README.md
deleted file mode 100644
index 45f6aa7d..00000000
--- a/research/benchmarks/zh_reliability/README.md
+++ /dev/null
@@ -1,183 +0,0 @@
-# Chinese alarm-intent evaluation and temperature calibration
-
-This optional research benchmark measures `choice` and `noul` decisions through Laya's
-existing local Agent, sequence builder and multilingual checkpoint. E0 uses the checkpoint's
-runtime temperatures; E1 fits one bounded scalar temperature per question type on a separate
-calibration partition. It does not update model weights or change the public runtime API.
-
-The contribution is deliberately narrower than a general Chinese benchmark: a three-intent
-MASSIVE alarm subset, grouping checks, auditable calibration/test separation, and paired
-before/after reliability metrics. It complements the broader Chinese evaluation in
-[#557](https://github.com/NandhaKishorM/laya/pull/557), short-command prompt ablations in
-[#364](https://github.com/NandhaKishorM/laya/pull/364), and the generic evaluation harness in
-[#382](https://github.com/NandhaKishorM/laya/pull/382).
-
-## Inspect and recompute the historical result on CPU
-
-[The result table](results/massive_alarm_2026-09-24.md) is generated from
-[compact decision evidence](results/massive_alarm_2026-09-24.json). The JSON contains 128
-calibration and 186 test records with source IDs, group IDs, labels, option order, raw logits,
-actual temperatures and source hashes. Test records retain both runs' raw logits so the verifier
-checks that calibration changed only temperature. It includes original file hashes, model
-identity, selected dependency versions and numerical summaries. Source text, weights, local
-absolute paths, GPU UUIDs and credentials are omitted.
-
-**These are historical measurements from 2026-09-24**, executed on base commit `23a1752`
-with the contribution subsequently recorded in `37cc584`. The execution worktree was dirty;
-its recorded source-file hashes identify the measured implementation. The results were not
-measured against the later upstream rebase or this evaluation-only refactor.
-
-From the repository root, with the project's Python dependencies installed:
-
-```bash
-python -m research.benchmarks.zh_reliability.aggregate_massive \
- --artifact research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json \
- --report /tmp/massive-alarm-result.md
-```
-
-This performs no inference or downloads and needs no CUDA/model checkpoint. It verifies record
-integrity, partition/group/source pairing and original temperatures; refits temperatures from
-calibration logits; rebuilds probabilities and metrics; recomputes 1,000 paired group bootstrap
-draws; and compares all numerical summaries with the saved values. Hashes bind the archived
-artifacts; they are not independent authentication of data provenance. Auditing original text
-and annotation declarations additionally requires the prepared public dataset.
-
-| Condition | Accuracy | Macro F1 | NLL | Brier | ECE |
-|---|---:|---:|---:|---:|---:|
-| E0 original temperatures | 0.532258 | 0.440840 | 1.383747 | 0.730071 | 0.366488 |
-| E1 calibrated temperatures | 0.532258 | 0.440840 | 0.785033 | 0.506898 | 0.197529 |
-
-Accuracy and F1 are unchanged because positive scalar temperature preserves argmax. The NLL
-difference is −0.598713, with a group bootstrap 95% interval [−0.865774, −0.369099]. This
-supports a calibration result on this subset, not general Chinese capability improvement.
-Choice temperature is 3.135504; noul reaches the permitted upper bound of 5.0. Risk/coverage
-does not improve at every coverage.
-
-## Prepare public data
-
-Amazon's MASSIVE 1.1 is human-localized virtual-assistant text, licensed CC BY 4.0
-([pinned publisher notice](https://github.com/alexa/massive/blob/f966f21846043aabef9b0f974fa7970027f43738/NOTICE.md)).
-It is not naturally occurring Chinese customer-service logs. The derived tasks use the original
-`alarm_set`, `alarm_query`, and `alarm_remove` labels: choice selects the operation; noul asks
-whether the intent is setting an alarm. No paraphrases or new semantic labels are generated.
-
-```bash
-# Optional dependency for the small Parquet download route only.
-uv pip install --python .venv/bin/python pyarrow
-.venv/bin/python -m research.benchmarks.zh_reliability.prepare_massive \
- --download-parquet --run-dir runs/massive-alarm-data
-.venv/bin/python -m research.benchmarks.zh_reliability import-data \
- --data runs/massive-alarm-data/data.jsonl --formal \
- --split-manifest runs/massive-alarm-data/split_manifest.json \
- --run-dir runs/massive-import
-```
-
-The Parquet route downloads six Chinese/English files, about 2 MB, from the publisher's
-`AmazonScience/massive` repository at conversion revision
-`ed58ac423a2f4121720918bf5301577edce4ffd3`. It executes no dataset loading script. Embedded
-ClassLabel indices are decoded, and column-oriented annotation lists are transposed without
-changing text or labels. Saved manifests identify transport, decoding rules, reader version,
-file hashes and pinned attribution/legal text. Decoded source-row hashes are not archive-byte
-hashes. Alternatively, `--download` uses the publisher's approximately 40 MB archive and
-records its hash; only selected regular members are read. `--source-dir PATH` converts an
-already downloaded, hash-checked source directory offline. Every output directory must be new.
-
-The fixed admission policy requires at least three distinct upstream reviewers whose intent
-scores all equal 1. It does not imply unanimous grammar/spelling/slot approval. Provenance
-retains these upstream judgments and `human_verified=false`; no independent local review is
-claimed. Other records remain in `diagnostic.jsonl`, with reasons. No model scores determine
-admission, grouping or labels.
-
-Connected groups link identical source IDs, normalized Chinese text and delexicalized English
-source templates. Cross-partition groups keep the highest-priority original partition
-(test > dev > train); lower-priority members are quarantined. Conflicting-label groups are
-quarantined. Calibration uses 20% of the remaining training groups with seed `20260924`.
-Grouping is heuristic and can miss semantic dependencies.
-
-The retained 471 source utterances generate 942 correlated decisions. Fixed partition decision
-counts are train 522, dev 106, calibration 128 and test 186, with 247/53/62/89 groups.
-**This evaluation consumes only calibration and test**; the other source partitions remain
-reserved. Test has 93 utterances, each represented by both question types.
-
-## Run a new evaluation
-
-Use a local copy of `convaiinnovations/laya-multilingual` revision
-`e4e9ddf21a7b1903b7acffd8814ad4307bf63a67`, including the recorded `zh_lineage.json` identifying
-that original revision. The script verifies fixed data/group counts and records paths/hashes
-before predictions. Set `LAYA_ZH_PYTHON` if the interpreter is not `.venv/bin/python`.
-
-On a clean clone, explicitly download that pinned revision and record its original-checkpoint
-lineage. This downloads only runtime files to the Hugging Face cache, copies them into a new
-local model directory, and performs no inference. It does not fall back to `main`:
-
-```bash
-.venv/bin/python - <<'PY_DOWNLOAD'
-from pathlib import Path
-from research.benchmarks.zh_reliability.data import atomic_json
-from research.benchmarks.zh_reliability.runtime import resolve_model
-
-revision = 'e4e9ddf21a7b1903b7acffd8814ad4307bf63a67'
-work = Path('runs/massive-model-download')
-work.mkdir(parents=True) # Refuse an existing output directory.
-model, identity = resolve_model('convaiinnovations/laya-multilingual', revision, False, work)
-if identity['revision'] != revision:
- raise ValueError('download resolved to an unexpected revision')
-atomic_json(model / 'zh_lineage.json', {
- 'model_id': identity['model_id'], 'base_revision': revision,
- 'trained': False, 'weights_hash': identity['files']['model.safetensors']})
-print(model)
-PY_DOWNLOAD
-```
-
-The resulting path is `runs/massive-model-download/model_input`:
-
-```bash
-bash research/benchmarks/zh_reliability/reproduce_massive.sh \
- runs/new-massive-calibration runs/massive-model-download/model_input \
- runs/massive-alarm-data/data.jsonl runs/massive-alarm-data/split_manifest.json
-```
-
-The fixed recipe runs only `calibrate`, baseline `eval`, calibrated `eval`, and CPU aggregation.
-It defaults to process-visible `cuda:0`, FP16, max_len=512, and head_max_len=192. Set
-`LAYA_ZH_DEVICE` to a visible device selector, for example `LAYA_ZH_DEVICE="RTX 2080 Ti"`,
-to use another device. The selected value is recorded before predictions in the generated
-protocol; the historical measurements used RTX 2080 Ti. Selectors refer to visible CUDA devices; no device mask or driver is changed. Downloading models is
-not implicit. A failed stage returns nonzero and records its stage in a FAILED manifest.
-Run directories hold local predictions, environment, split, calibration and model identities;
-private tokenizer copies protect the input checkpoint from compatibility patches.
-
-Standalone commands are available with `python -m research.benchmarks.zh_reliability --help`.
-`eval` and `calibrate` require consistent data, `--formal`, `--split-manifest`, model, dtype and
-length settings. Calibration defaults to at least 30 samples per type. Artifact validation
-rejects a different model/data/split or unapproved smoke-only fit. A new completed recipe can
-be checked and its report rebuilt without GPU inference:
-
-```bash
-python -m research.benchmarks.zh_reliability.aggregate_massive runs/new-massive-calibration
-```
-
-The short agent-authored examples in `fixtures.jsonl` remain unverified regression diagnostics;
-passing fixture tests is not a measured public-data result.
-
-## Definitions and limits
-
-- Only valid decision logits are used, never action outputs or rounded public probabilities.
- Noul option order is `[false,true]`; confidence is max(p). Existing temperatures are not
- applied twice. The bounded fit minimizes calibration NLL in the runtime range [0.5,5.0].
-- NLL uses float64 log-softmax. Brier is the mean sum of squared class errors. Macro F1 averages
- task-local macro F1 over the full option set. ECE has 15 equal bins, the last including 1.
- Risk/coverage breaks confidence ties by lexical sample ID. Type/task/truncation slices and
- official-Agent probability parity are retained by standalone evaluation.
-- Bootstrap resamples the 89 test groups together across both conditions and question types.
- Its 1,000 draws use seed `20260924`. Intervals condition on this checkpoint and fitted
- temperatures, omit calibration uncertainty, and make no corrected multiple-testing claim.
-- This is one task subset and one split/seed. Checkpoint exposure to MASSIVE/SLURP is unknown.
- It does not establish full MASSIVE performance, general Chinese improvement, or unchanged
- behavior in other domains. Action outputs are outside scope.
-- Historical test sequences were at most 60 tokens with no truncation. max_len=512 is a limit,
- not a 512-token stress test. Optional timings synchronize CUDA, distinguish loading and
- inference memory, and report single-decision batch p50/p95; GPUs are not exclusively reserved.
-
-Offline checks use `python -m unittest discover -s tests -p 'test_zh_*.py' -v`. The archive
-recomputation command above additionally verifies the submitted numerical evidence. Repository
-Ruff and compileall gates apply; no hosted inference service or new runtime dependency is added.
diff --git a/research/benchmarks/zh_reliability/__init__.py b/research/benchmarks/zh_reliability/__init__.py
deleted file mode 100644
index b634e621..00000000
--- a/research/benchmarks/zh_reliability/__init__.py
+++ /dev/null
@@ -1 +0,0 @@
-"""Opt-in local Chinese decision evaluation and calibration; not a runtime API."""
diff --git a/research/benchmarks/zh_reliability/__main__.py b/research/benchmarks/zh_reliability/__main__.py
deleted file mode 100644
index c7568bf7..00000000
--- a/research/benchmarks/zh_reliability/__main__.py
+++ /dev/null
@@ -1,297 +0,0 @@
-"""Reproducible opt-in experiments: python -m research.benchmarks.zh_reliability --help."""
-import argparse
-from datetime import datetime, timezone
-import json
-from pathlib import Path
-import sys
-import traceback
-import uuid
-
-from .data import FIXTURES, SPLITS, atomic_json, partition, read_data
-from .runtime import environment, select_device
-
-
-def parser():
- p = argparse.ArgumentParser(description=__doc__)
- sub = p.add_subparsers(dest='command', required=True)
- for name in ('check-env', 'import-data', 'smoke', 'eval', 'calibrate', 'report'):
- s = sub.add_parser(name)
- s.add_argument('--run-dir', type=Path, help='fresh output directory; existing runs are never overwritten')
- s.add_argument('--data', type=Path, default=FIXTURES)
- s.add_argument('--formal', action='store_true', help='require licensed data, documented human annotation and fixed splits')
- s.add_argument('--split-manifest', type=Path)
- s.add_argument('--seed', type=int, default=20260924)
- if name in ('smoke', 'eval', 'calibrate'):
- s.add_argument('--model', default='convaiinnovations/laya-multilingual')
- s.add_argument('--revision', default='main', help='resolved to immutable HF SHA before download')
- s.add_argument('--download', action='store_true', help='allow target-model-only download; default offline')
- s.add_argument('--device', default='cuda:0', help='unique visible GPU name, UUID or process cuda:index')
- s.add_argument('--dtype', choices=('fp16', 'fp32'), default='fp16')
- s.add_argument('--max-len', type=int, default=256 if name == 'smoke' else 512)
- s.add_argument('--head-max-len', type=int, default=192)
- if name in ('eval', 'smoke'):
- s.add_argument('--warmup', type=int, default=10)
- s.add_argument('--batches', type=int, default=30)
- if name == 'eval':
- s.add_argument('--split', choices=SPLITS, default='test')
- s.add_argument('--calibration', type=Path)
- s.add_argument('--allow-smoke-calibration', action='store_true')
- s.add_argument('--measure', action='store_true')
- if name in ('calibrate', 'smoke'):
- s.add_argument('--minimum', type=int, default=1 if name == 'smoke' else 30)
- if name == 'calibrate':
- s.add_argument('--smoke-only', action='store_true')
- if name == 'smoke':
- s.add_argument('--gpu', action='store_true', help='opt in to actual Laya inference, calibration and precision measurements')
- if name == 'report':
- s.add_argument('--source', type=Path, required=True, help='existing result directory (read only)')
- return p
-
-
-def write_predictions(path, rows):
- # Runs are private until manifest finalization; each row retains raw unrounded decision logits.
- with Path(path).open('w', encoding='utf-8') as f:
- for row in rows:
- f.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + '\n')
-
-
-def report(run, destination=None):
- run = Path(run)
- manifest = json.loads((run / 'manifest.json').read_text())
- metrics = json.loads((run / 'metrics.json').read_text())
- text = '# Chinese decision reliability run\n\n'
- text += f"Status: **{manifest['status']}**. Run: `{run.name}`.\n\n"
- data_info = manifest.get('data_provenance', manifest.get('source_manifest', {}).get('data_provenance', {}))
- if 'public_human_annotated' in data_info.get('kinds', []):
- text += 'Public human-localized data with retained upstream annotation evidence; no independent local human review. '
- text += 'This derived task/subset is not the full source benchmark or evidence of general Chinese improvement. '
- elif data_info.get('formal'):
- text += 'Source/annotation declarations and fixed splits were validated; this does not establish representative sampling or independent label correctness. '
- else:
- text += 'Agent-authored fixtures are unverified diagnostics, not a standard evaluation or evidence of general Chinese improvement. '
- text += 'Action outputs are outside this evaluation. No test-set temperature selection.\n\n'
- text += 'Full slices, per-sample evidence and configuration: `metrics.json`, `predictions.jsonl`, '
- text += '`split_manifest.json`, `environment.json`, `manifest.json`.\n\n'
- conditions = [(k, v['overall']) for k, v in metrics.items() if isinstance(v, dict) and 'overall' in v]
- if 'overall' in metrics:
- conditions = [('evaluated split', metrics['overall'])]
- if conditions:
- text += '| Condition | n | Accuracy | Task macro F1 | NLL | Brier | ECE |\n'
- text += '|---|---:|---:|---:|---:|---:|---:|\n'
- for k, m in conditions:
- values = [str(m['n'])] + [f"{m[x]:.6f}" if m[x] is not None else 'null'
- for x in ('accuracy', 'macro_f1', 'nll', 'brier', 'ece')]
- text += '| ' + k + ' | ' + ' | '.join(values) + ' |\n'
- text += '\n'
- timing = [(k, v) for k, v in metrics.items() if isinstance(v, dict) and 'end_to_end' in v]
- if timing:
- text += '| Device / dtype | E2E batch p50 ms | E2E batch p95 ms | state/s | Model p50 ms | Peak allocated MiB |\n'
- text += '|---|---:|---:|---:|---:|---:|\n'
- for k, v in timing:
- e, m = v['end_to_end'], v['model_stage']
- text += (f"| {k} | {e['p50_batch_ms']:.3f} | {e['p95_batch_ms']:.3f} | "
- f"{e['state_per_s']:.2f} | {m['p50_batch_ms']:.3f} | {e['max_memory_allocated']/2**20:.1f} |\n")
- text += '\nLatency is per batch (one state/decision); model stage excludes H2D/D2H.\n\n'
- details = {k: v for k, v in metrics.items() if k not in {x[0] for x in conditions + timing}}
- text += '## Verification and numerical diagnostics\n\n```json\n'
- text += json.dumps(details, ensure_ascii=False, indent=2, allow_nan=False) + '\n```\n'
- text += '\n## Execution\n\n```json\n' + json.dumps(manifest, ensure_ascii=False, indent=2) + '\n```\n'
- target = Path(destination) if destination else run / 'report.md'
- target.write_text(text, encoding='utf-8')
-
-
-def load_adapter(args, env, run):
- import torch
- from laya.agent import Agent
- from .runtime import Adapter, ensure_device, resolve_model
- device = select_device(args.device, env)
- model_input, identity = resolve_model(args.model, args.revision, not args.download, run)
- torch.cuda.synchronize(device)
- torch.cuda.reset_peak_memory_stats(device)
- agent = Agent(str(model_input), device=device, fast=False, compile=False)
- ensure_device(agent, device)
- atomic_json(Path(run) / 'loading_memory.json', {'device': device,
- 'max_memory_allocated': torch.cuda.max_memory_allocated(device),
- 'max_memory_reserved': torch.cuda.max_memory_reserved(device), 'scope': 'model loading only'})
- return Adapter(agent, identity, args.max_len, args.head_max_len, args.dtype), model_input
-
-
-def evaluate(adapter, rows, split, calibration=None, smoke=False, selection='test'):
- import copy
- from .calibration import validate_calibration
- from .metrics import summarize
- adapter.agent.lang_temperatures = (validate_calibration(calibration, adapter.identity, split, smoke)
- if calibration else copy.deepcopy(adapter.original['lang_temperatures']))
- condition = 'E1' if calibration else 'E0'
- selected = [r for r in rows if split['assignments'][r['id']] == selection]
- predictions = adapter.predict(selected, condition, split)
- return predictions, summarize(predictions)
-
-
-def gpu_smoke(args, env, run, rows, split, manifest):
- import copy
- import numpy as np
- import torch
- from .calibration import calibrate
- from .metrics import paired_bootstrap
- from .runtime import measure
- device = select_device(args.device, env)
- results = {'devices': {args.device: {'status': 'AVAILABLE', 'device': device}}}
- manifest['gpu_experiment'] = 'PARTIAL'
- base_dir = run / 'base'
- base_dir.mkdir()
- adapter, _ = load_adapter(args, env, base_dir)
- manifest['model'] = adapter.identity
- subset = [
- r for kind in ('choice', 'noul') for r in [x for x in rows if x['question']['type'] == kind][:4]]
- results['official_parity'] = adapter.parity(subset)
- cal_rows = [r for r in rows if split['assignments'][r['id']] == 'calibration']
- base_preds, results['E0'] = evaluate(adapter, rows, split)
- cal_preds = adapter.predict(cal_rows, 'E0', split)
- artifact = calibrate(cal_preds, adapter.identity, split, adapter.original,
- run / 'calibration.json', args.minimum, True)
- results['calibration_E1'] = artifact['fitted']
- if any(x['status'] == 'SKIPPED' for x in artifact['fitted'].values()):
- manifest['status'] = 'PARTIAL'
- fitted_preds, results['E1'] = evaluate(adapter, rows, split, artifact, True)
- all_predictions = base_preds + fitted_preds + cal_preds
- results['E1-E0'] = (paired_bootstrap(base_preds, fitted_preds) if args.formal
- else {'status': 'SKIPPED', 'reason': 'unverified fixtures cannot support population inference'})
- manifest['completed'].append('E0/E1')
- write_predictions(run / 'predictions.jsonl', all_predictions)
- atomic_json(run / 'metrics.json', results)
- atomic_json(run / 'manifest.json', manifest)
- # Precision comparisons use the original checkpoint temperatures and fixed samples.
- adapter.agent.lang_temperatures = copy.deepcopy(adapter.original['lang_temperatures'])
- numeric = {}
- for dtype in ('fp16', 'fp32'):
- key = args.device + '/' + dtype
- adapter.dtype = dtype
- adapter.agent.amp_enabled = dtype == 'fp16'
- adapter.agent.dtype = torch.float16 if dtype == 'fp16' else torch.float32
- numeric[dtype] = adapter.predict(subset, 'numeric-' + dtype, split)
- results[key] = measure(adapter, subset, args.warmup, args.batches)
- all_predictions += numeric[dtype]
- write_predictions(run / 'predictions.jsonl', all_predictions)
- atomic_json(run / 'metrics.json', results)
- delta = np.concatenate([np.abs(np.array(x['probabilities']) - y['probabilities'])
- for x, y in zip(numeric['fp16'], numeric['fp32'])])
- results['numerical_comparisons'] = {args.device + '/fp16 vs ' + args.device + '/fp32': {
- 'n': len(subset), 'argmax_agreement': float(np.mean([x['predicted'] == y['predicted']
- for x, y in zip(numeric['fp16'], numeric['fp32'])])),
- 'probability_abs_mean': float(delta.mean()), 'probability_abs_max': float(delta.max()),
- 'probability_abs_p50_p95_p99': np.quantile(delta, [.5, .95, .99]).tolist()}}
- manifest['completed'].append('FP16/FP32 measurements on requested device')
- return results
-
-
-def main(argv=None):
- args = parser().parse_args(argv)
- run = args.run_dir or Path('runs') / (datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ') + '-' + uuid.uuid4().hex[:8])
- if run.exists():
- print('refusing existing run directory: ' + str(run), file=sys.stderr)
- return 2
- run.mkdir(parents=True)
- manifest = {'run_id': run.name, 'status': 'RUNNING', 'command': args.command, 'completed': [],
- 'config': {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()},
- 'smoke_only': args.command == 'smoke' or getattr(args, 'smoke_only', False),
- 'gpu_experiment': 'NOT_RUN'}
- atomic_json(run / 'manifest.json', manifest)
- atomic_json(run / 'metrics.json', {'status': 'NOT_RUN'})
- write_predictions(run / 'predictions.jsonl', [])
- stage = 'environment'
- try:
- env = environment()
- atomic_json(run / 'environment.json', env)
- if args.command == 'report':
- import shutil
- stage = 'report'
- manifest['source_manifest'] = json.loads((args.source / 'manifest.json').read_text())
- for name in ('metrics.json', 'predictions.jsonl', 'split_manifest.json'):
- shutil.copyfile(args.source / name, run / name)
- shutil.copyfile(args.source / 'environment.json', run / 'source_environment.json')
- manifest['completed'].append('report generated from ' + str(args.source))
- else:
- stage = 'data'
- rows = read_data(args.data, args.formal)
- manifest['data_provenance'] = {
- 'formal': args.formal,
- 'kinds': sorted({r['provenance']['kind'] for r in rows}),
- 'licenses': sorted({r['provenance'].get('license', 'unspecified') for r in rows}),
- 'human_verified_rows': sum(r['provenance']['human_verified'] for r in rows),
- 'upstream_review_rows': sum('upstream_annotation' in r['provenance'] for r in rows)}
- split = partition(rows, args.split_manifest, args.seed, args.formal)
- atomic_json(run / 'split_manifest.json', split)
- manifest['completed'].append('data/schema/group partitions')
- atomic_json(run / 'metrics.json', {'data_counts': split['counts'], 'groups': split['group_counts']})
- if args.command == 'check-env':
- manifest['completed'].append('CUDA execution probes')
- if env['gpu_status'] == 'FAILED':
- raise RuntimeError('CUDA execution probe failed; see environment.json')
- elif args.command == 'smoke' and not args.gpu:
- import subprocess
- stage = 'offline tests'
- completed = subprocess.run([sys.executable, '-m', 'unittest', 'discover', '-s', 'tests',
- '-p', 'test_zh_*.py', '-v'], capture_output=True, text=True)
- (run / 'offline-tests.log').write_text(completed.stdout + completed.stderr)
- if completed.returncode:
- raise RuntimeError('offline tests failed; see offline-tests.log')
- atomic_json(run / 'metrics.json', {'offline_tests': 'SUCCESS', 'gpu': 'NOT_RUN',
- 'reason': 'default smoke is offline; opt in using --gpu', 'data_counts': split['counts']})
- manifest['completed'].append('offline tests (tiny/mock inference only)')
- manifest['status'] = 'PARTIAL'
- elif args.command == 'smoke':
- stage = 'GPU smoke'
- if not env['gpus']:
- manifest['status'] = 'SKIPPED'
- manifest['reason'] = 'CUDA unavailable; run default offline smoke'
- else:
- results = gpu_smoke(args, env, run, rows, split, manifest)
- atomic_json(run / 'metrics.json', results)
- manifest['gpu_experiment'] = ('SUCCESS' if manifest['status'] == 'RUNNING'
- else manifest['status'])
- elif args.command not in ('check-env', 'import-data'):
- stage = 'model loading'
- adapter, _ = load_adapter(args, env, run)
- manifest['model'] = adapter.identity
- stage = args.command
- if args.command == 'eval':
- calibration = json.loads(args.calibration.read_text()) if args.calibration else None
- preds, metrics = evaluate(adapter, rows, split, calibration, args.allow_smoke_calibration, args.split)
- metrics['official_parity'] = adapter.parity([r for r in rows if split['assignments'][r['id']] == args.split][:4])
- if args.measure:
- from .runtime import measure
- metrics['measurement'] = measure(adapter, [r for r in rows if split['assignments'][r['id']] == args.split],
- args.warmup, args.batches)
- write_predictions(run / 'predictions.jsonl', preds)
- elif args.command == 'calibrate':
- from .calibration import calibrate
- preds, metrics = evaluate(adapter, rows, split, selection='calibration')
- artifact = calibrate(preds, adapter.identity, split, adapter.original, run / 'calibration.json',
- args.minimum, args.smoke_only)
- metrics['fitted'] = artifact['fitted']
- if any(x['status'] == 'SKIPPED' for x in artifact['fitted'].values()):
- manifest['status'] = 'PARTIAL'
- write_predictions(run / 'predictions.jsonl', preds)
- atomic_json(run / 'metrics.json', metrics)
- manifest['completed'].append(args.command)
- manifest['gpu_experiment'] = 'SUCCESS'
- if manifest['status'] == 'RUNNING':
- manifest['status'] = 'SUCCESS'
- atomic_json(run / 'manifest.json', manifest)
- report(run)
- print(str(run.resolve()))
- return 0
- except Exception as e:
- if manifest['gpu_experiment'] != 'NOT_RUN':
- manifest['gpu_experiment'] = 'FAILED'
- manifest.update(status='FAILED', error={'stage': stage, 'type': type(e).__name__, 'reason': str(e)})
- atomic_json(run / 'manifest.json', manifest)
- (run / 'error.log').write_text(traceback.format_exc())
- report(run)
- print(f'{stage}: {e}; diagnostics: {run}', file=sys.stderr)
- return 1
-
-
-if __name__ == '__main__':
- raise SystemExit(main())
diff --git a/research/benchmarks/zh_reliability/aggregate_massive.py b/research/benchmarks/zh_reliability/aggregate_massive.py
deleted file mode 100644
index afadf689..00000000
--- a/research/benchmarks/zh_reliability/aggregate_massive.py
+++ /dev/null
@@ -1,320 +0,0 @@
-"""Audit baseline/calibrated runs, or recompute the compact historical evidence on CPU."""
-import argparse
-from collections import Counter
-from datetime import datetime, timezone
-import json
-import math
-from pathlib import Path
-import sys
-import traceback
-
-sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
-from research.benchmarks.zh_reliability.calibration import fit_temperature, validate_calibration
-from research.benchmarks.zh_reliability.data import atomic_json, digest, file_hash, partition, read_data
-from research.benchmarks.zh_reliability.metrics import paired_bootstrap, probability_record, summarize
-
-CHILDREN = ('base-calibration', 'E0', 'E1')
-CONDITIONS = ('E0', 'E1')
-SCHEMA = 'massive-alarm-calibration-evidence-v1'
-
-
-def require(value, message):
- if not value:
- raise ValueError(message)
-
-
-def read_json(path):
- return json.loads(Path(path).read_text(encoding='utf-8'))
-
-
-def read_jsonl(path):
- return [json.loads(line) for line in Path(path).read_text(encoding='utf-8').splitlines() if line.strip()]
-
-
-def numerically_equal(a, b):
- if isinstance(a, dict) and isinstance(b, dict):
- return a.keys() == b.keys() and all(numerically_equal(a[k], b[k]) for k in a)
- if isinstance(a, list) and isinstance(b, list):
- return len(a) == len(b) and all(numerically_equal(x, y) for x, y in zip(a, b))
- if isinstance(a, float) and isinstance(b, (int, float)):
- return math.isclose(a, b, rel_tol=1e-10, abs_tol=1e-10)
- return a == b
-
-
-def evidence_prediction(record, condition):
- """Rebuild probabilities without source text, a tokenizer, model weights or CUDA."""
- row = {'id': record['id'], 'group_id': record['group_id'], 'task_id': record['task_id'],
- 'question': {'type': record['type'], 'criteria': dict.fromkeys(record['option_order'], '')},
- 'label': record['gold'], 'tags': ['public_localized', 'alarm']}
- logits = record['raw_logits'] if condition == 'E0' else record['calibrated_raw_logits']
- temperature = record['temperature'] if condition == 'E0' else record['calibrated_temperature']
- result = probability_record(row, logits, temperature)
- result.update(sample_hash=record['sample_hash'], split=record['split'],
- tokens={'input_tokens': record['input_tokens'], 'truncated': record['truncated']})
- return result
-
-
-def compact_summary(rows):
- result = summarize(rows)
- return {key: result[key] for key in ('overall', 'by_type', 'definitions')}
-
-
-def recompute(artifact, bootstrap=True):
- """Recompute evidence and validate pairing/fit; do not trust stored summary values."""
- require(artifact['schema'] == SCHEMA, 'unsupported evidence schema')
- records = artifact['records']
- model = artifact['provenance']['model']
- require(model['content_hash'] == digest(model['files']), 'model identity hash mismatch')
- require(model['model_id'] == 'convaiinnovations/laya-multilingual'
- and model['revision'] == 'e4e9ddf21a7b1903b7acffd8814ad4307bf63a67'
- and model['trained'] is False, 'expected pinned original multilingual checkpoint')
- require(artifact['calibration']['minimum_per_type'] == 30 and artifact['calibration']['bounds'] == [.5, 5.],
- 'calibration protocol mismatch')
- from laya.common import QTYPES, temp_bucket
- original = artifact['calibration']['original']
- temperature_config = original['lang_temperatures'].get('zh', original)
- require(artifact['records_sha256'] == digest(records), 'decision record hash mismatch')
- require(len({r['id'] for r in records}) == len(records), 'duplicate decision IDs')
- require(all(r['split'] in ('calibration', 'test') for r in records), 'unexpected evidence split')
- groups = {}
- source_types = {}
- source_bindings = {}
- for rec in records:
- require(rec['type'] in ('choice', 'noul'), 'unsupported question type')
- require(rec['gold'] in rec['option_order'] and len(set(rec['option_order'])) == len(rec['option_order']),
- 'invalid gold/options')
- require(len(rec['raw_logits']) == len(rec['option_order']), 'logit/options mismatch')
- require(len(rec['sample_hash']) == 64 and len(rec['source_row_hash']) == 64, 'invalid source hash')
- require(rec['group_id'] not in groups or groups[rec['group_id']] == rec['split'], 'group leakage')
- groups[rec['group_id']] = rec['split']
- source = (rec['split'], rec['source_id'])
- binding = (rec['split'], rec['group_id'], rec['source_row_hash'])
- require(rec['source_id'] not in source_bindings or source_bindings[rec['source_id']] == binding,
- 'source utterance split/group/hash mismatch')
- source_bindings[rec['source_id']] = binding
- qt = QTYPES[rec['type']]
- expected_temperature = temperature_config['temperature_by_options'].get(
- temp_bucket(qt, len(rec['option_order'])), temperature_config['temperature'][qt])
- require(rec['temperature'] == expected_temperature, 'baseline temperature mismatch')
- source_types.setdefault(source, []).append(rec['type'])
- if rec['split'] == 'test':
- require(rec['calibrated_raw_logits'] == rec['raw_logits'], 'calibration changed raw decision logits')
- require(all(sorted(types) == ['choice', 'noul'] for types in source_types.values()),
- 'source utterances must keep both question types together')
- counts = Counter(r['split'] for r in records)
- group_counts = Counter(groups.values())
- require(counts == {'calibration': 128, 'test': 186}, 'fixed subset decision counts mismatch')
- require(group_counts == {'calibration': 62, 'test': 89}, 'fixed subset group counts mismatch')
- calibration = [evidence_prediction(r, 'E0') for r in records if r['split'] == 'calibration']
- fitted = {}
- for kind in ('choice', 'noul'):
- selected = [r for r in calibration if r['question']['type'] == kind]
- require(len(selected) == 64, 'calibration type count mismatch')
- fitted[kind] = fit_temperature(selected, minimum=30)
- require(numerically_equal(fitted[kind], artifact['calibration']['fitted'][kind]),
- 'fitted temperature/objective mismatch: ' + kind)
- test = [r for r in records if r['split'] == 'test']
- for rec in test:
- require(rec['calibrated_temperature'] == artifact['calibration']['fitted'][rec['type']]['temperature'],
- 'test calibration temperature mismatch')
- predictions = {c: [evidence_prediction(r, c) for r in test] for c in CONDITIONS}
- require(all(a['predicted'] == b['predicted'] for a, b in zip(predictions['E0'], predictions['E1'])),
- 'positive scalar temperature changed argmax')
- result = {c: compact_summary(predictions[c]) for c in CONDITIONS}
- result['fitted'] = fitted
- result['counts'] = {'calibration_decisions': 128, 'calibration_groups': 62,
- 'test_decisions': 186, 'test_groups': 89, 'test_source_utterances': 93}
- if bootstrap:
- config = artifact['analysis']['paired_group_bootstrap']
- require(config == {'repeats': 1000, 'seed': 20260924, 'min_groups': 20}, 'bootstrap protocol mismatch')
- result['paired'] = {'E1-E0': paired_bootstrap(predictions['E0'], predictions['E1'], **config)}
- return result
-
-
-def verify_artifact(artifact, bootstrap=True):
- """Return recomputed results; reject changed records, fit, pairing or stored metrics."""
- result = recompute(artifact, bootstrap)
- for key, value in result.items():
- require(numerically_equal(value, artifact['summary'][key]), 'stored summary mismatch: ' + key)
- return result
-
-
-def render_report(artifact, result):
- provenance = artifact['provenance']
- lines = ['# MASSIVE Chinese alarm calibration evaluation', '',
- f"Measurement date: **{provenance['measurement_date']}**. "
- f"Execution base: `{provenance['execution_base_commit']}`; "
- f"contribution: `{provenance['contribution_commit']}`.", '',
- provenance['version_note'], '',
- 'E0 and E1 use the same multilingual checkpoint and raw decision logits. E1 fits one '
- 'temperature per question type on calibration data. No model weights are updated.', '',
- '| Condition | Slice | n | Accuracy | Macro F1 | NLL | Brier | ECE |',
- '|---|---|---:|---:|---:|---:|---:|---:|']
- for condition in CONDITIONS:
- for kind, scores in [('overall', result[condition]['overall']), *result[condition]['by_type'].items()]:
- values = ' | '.join(f'{scores[k]:.6f}' for k in ('accuracy', 'macro_f1', 'nll', 'brier', 'ece'))
- lines.append(f"| {condition} | {kind} | {scores['n']} | {values} |")
- lines += ['', 'Positive scalar temperature preserves argmax: accuracy and F1 are unchanged.', '',
- '| Type | Calibration n | Temperature | Bound reached |', '|---|---:|---:|---|']
- for kind, fit in result['fitted'].items():
- lines.append(f"| {kind} | {fit['n']} | {fit['temperature']:.6f} | {fit['boundary_hit']} |")
- if 'paired' in result:
- lines += ['', 'Paired differences are E1 minus E0; negative NLL, Brier and ECE differences are better.', '',
- '| Metric | Difference | 95% group bootstrap interval |', '|---|---:|---|']
- for name, value in result['paired']['E1-E0']['after_minus_before'].items():
- lo, hi = value['ci95']
- lines.append(f"| {name} | {value['delta']:+.6f} | [{lo:+.6f}, {hi:+.6f}] |")
- lines += ['', '## Scope and limits', '',
- '- Test: 93 source utterances, 89 heuristic groups, 186 correlated decisions. Calibration: '
- '64 source utterances, 62 groups, 128 decisions. The paired question types are not independent samples.',
- '- This is a derived MASSIVE 1.1 alarm subset with upstream human intent-review evidence, '
- 'not the full benchmark, naturally occurring customer-service logs, or independent local human review.',
- '- One fixed split/seed; 1,000 group bootstrap draws. Intervals condition on the fixed checkpoint '
- 'and fitted temperatures, omit calibration uncertainty, and are descriptive without multiplicity correction.',
- '- Unknown checkpoint exposure to MASSIVE/SLURP; grouping may miss semantic dependencies. '
- 'These results do not establish general Chinese improvement or unchanged behavior on other tasks.',
- '- The baseline noul fit reaches T=5, the permitted upper bound. Calibration settings were fixed '
- 'without choosing on test. Risk/coverage does not improve at every coverage.',
- '- Configured max_len=512, head_max_len=192, FP16. Historical test inputs were at most 60 tokens; '
- 'this is not a 512-token stress measurement.',
- '- The compact evidence omits source text and retains source IDs/hashes. It can recompute numerical '
- 'results and check pairing; full source-schema/annotation checks require the prepared public dataset.', '']
- return '\n'.join(lines)
-
-
-def verify_run_predictions(predictions, rows, split, model, selection, condition):
- expected = {r['id']: r for r in rows if split['assignments'][r['id']] == selection}
- require(len(predictions) == len(expected) == len({r['id'] for r in predictions}), 'prediction count/duplicate mismatch')
- require({r['id'] for r in predictions} == set(expected), 'prediction membership mismatch')
- for rec in predictions:
- row = expected[rec['id']]
- require(rec['split'] == selection and rec['condition'] == condition, 'prediction partition/condition mismatch')
- require(rec['model'] == model and rec['sample_hash'] == digest(row) == split['row_hashes'][rec['id']],
- 'prediction model/source mismatch')
- require(all(rec.get(k) == v for k, v in row.items()), 'prediction source row changed')
- rebuilt = probability_record(row, rec['raw_logits'], rec['temperature'])
- require(all(numerically_equal(rec[k], rebuilt[k]) for k in
- ('option_order', 'gold', 'predicted', 'probabilities', 'log_probabilities', 'confidence', 'p_true')),
- 'probability reconstruction mismatch')
-
-
-def collect_run(root):
- """Read only the three evaluation children; never modify the supplied run."""
- root = Path(root)
- protocol = read_json(root / 'protocol.json')
- manifest = read_json(root / 'manifest.json')
- require(file_hash(root / 'protocol.json') == manifest['protocol_sha256'], 'protocol hash mismatch')
- rows = read_data(protocol['data'], formal=True)
- require(file_hash(protocol['data']) == protocol['data_file_sha256'], 'source file hash mismatch')
- require(file_hash(protocol['split_manifest']) == protocol['split_file_sha256'], 'split file hash mismatch')
- split = partition(rows, protocol['split_manifest'], 20260924, True)
- children, predictions, hashes = {}, {}, {}
- for name in CHILDREN:
- directory = root / name
- child = read_json(directory / 'manifest.json')
- require(child['status'] == 'SUCCESS' and child['gpu_experiment'] == 'SUCCESS' and not child['smoke_only'],
- name + ': unsuccessful/formally ineligible run')
- expected = {'formal': True, 'seed': 20260924, 'max_len': 512, 'head_max_len': 192, 'dtype': 'fp16'}
- require(all(child['config'].get(k) == v for k, v in expected.items()), name + ': config mismatch')
- require(child['command'] == ('calibrate' if name == 'base-calibration' else 'eval'), 'wrong command')
- require(read_json(directory / 'split_manifest.json') == split, name + ': split mismatch')
- require(read_json(directory / 'model_identity.json') == child['model'], name + ': model metadata mismatch')
- children[name] = child
- predictions[name] = read_jsonl(directory / 'predictions.jsonl')
- hashes.update({name + '/' + file: file_hash(directory / file) for file in
- ('manifest.json', 'predictions.jsonl', 'metrics.json', 'split_manifest.json',
- 'model_identity.json', 'environment.json')})
- model = children['E0']['model']
- require(all(child['model'] == model for child in children.values()), 'model differs between conditions')
- require(model['content_hash'] == digest(model['files']), 'model identity content hash mismatch')
- require(model['trained'] is False, 'expected original checkpoint')
- for name in CHILDREN:
- verify_run_predictions(predictions[name], rows, split, model,
- 'calibration' if name == 'base-calibration' else 'test',
- 'E0' if name == 'base-calibration' else name)
- calibration = read_json(root / 'base-calibration/calibration.json')
- validate_calibration(calibration, model, split)
- require(calibration['minimum_per_type'] == 30 and calibration['smoke_only'] is False, 'invalid calibration minimum/mode')
- require(calibration['calibration_ids'] == [r['id'] for r in predictions['base-calibration']], 'calibration IDs differ')
- hashes['base-calibration/calibration.json'] = file_hash(root / 'base-calibration/calibration.json')
- after = {r['id']: r for r in predictions['E1']}
- records = []
- for rec in predictions['base-calibration'] + predictions['E0']:
- upstream = rec['provenance']['upstream_annotation']
- compact = {key: rec[key] for key in ('id', 'group_id', 'task_id', 'split', 'sample_hash', 'option_order',
- 'gold', 'raw_logits', 'temperature')}
- compact.update(type=rec['question']['type'], source_id=upstream['source_id'],
- source_row_hash=upstream['source_row_hash'], input_tokens=rec['tokens']['input_tokens'],
- truncated=rec['tokens']['truncated'])
- if rec['split'] == 'test':
- paired = after[rec['id']]
- require(paired['gold'] == rec['gold'] and paired['group_id'] == rec['group_id'], 'pairing changed')
- compact.update(calibrated_raw_logits=paired['raw_logits'], calibrated_temperature=paired['temperature'])
- records.append(compact)
- environment = read_json(root / 'E0/environment.json')
- safe_environment = {k: environment[k] for k in ('python', 'os', 'dependencies', 'cuda', 'cudnn')}
- source_hashes = {Path(k).name: v for k, v in environment.get('source_hashes', {}).items()
- if Path(k).name in ('__main__.py', 'data.py', 'metrics.py', 'calibration.py', 'runtime.py', 'prepare_massive.py')}
- artifact = {'schema': SCHEMA, 'provenance': {
- 'measurement_date': protocol['created_utc'][:10], 'execution_base_commit': environment['commit'],
- 'contribution_commit': 'unrecorded', 'version_note': 'Source version recorded by the execution environment; '
- 'recomputing this artifact does not rerun model inference.', 'execution_worktree_dirty': environment['dirty'],
- 'execution_source_hashes': source_hashes, 'original_file_sha256': hashes,
- 'dataset': 'MASSIVE 1.1 zh-CN alarm subset', 'dataset_license': 'CC-BY-4.0',
- 'publisher_notice': 'https://github.com/alexa/massive/blob/f966f21846043aabef9b0f974fa7970027f43738/NOTICE.md',
- 'data_hash': split['data_hash'], 'split_hash': split['split_hash'],
- 'data_file_sha256': protocol['data_file_sha256'], 'split_file_sha256': protocol['split_file_sha256'],
- 'model': model, 'environment': safe_environment,
- 'evaluation_config': {k: children['E0']['config'][k] for k in
- ('seed', 'max_len', 'head_max_len', 'dtype', 'device')}},
- 'analysis': {'paired_group_bootstrap': {'repeats': 1000, 'seed': 20260924, 'min_groups': 20}},
- 'calibration': {'minimum_per_type': 30, 'bounds': [.5, 5.], 'fitted': calibration['fitted'],
- 'original': calibration['original']}, 'records_sha256': digest(records), 'records': records}
- artifact['summary'] = recompute(artifact)
- for name in CONDITIONS:
- source = read_json(root / name / 'metrics.json')
- require(numerically_equal(artifact['summary'][name], {k: source[k] for k in artifact['summary'][name]}),
- 'recomputed metrics differ from measured child: ' + name)
- return artifact
-
-
-def main(argv=None):
- parser = argparse.ArgumentParser(description=__doc__)
- parser.add_argument('run_dir', type=Path, nargs='?')
- parser.add_argument('--artifact', type=Path, help='verify compact evidence without model/data downloads or CUDA')
- parser.add_argument('--report', type=Path, help='write Markdown rebuilt from verified values')
- args = parser.parse_args(argv)
- if (args.run_dir is None) == (args.artifact is None):
- parser.error('provide either RUN_DIR or --artifact')
- try:
- artifact = read_json(args.artifact) if args.artifact else collect_run(args.run_dir)
- result = verify_artifact(artifact) if args.artifact else artifact['summary']
- if args.artifact:
- if args.report:
- args.report.write_text(render_report(artifact, result), encoding='utf-8')
- else:
- root = args.run_dir
- atomic_json(root / 'calibration_evidence.json', artifact)
- atomic_json(root / 'metrics.json', result)
- (args.report or root / 'report.md').write_text(render_report(artifact, result), encoding='utf-8')
- manifest = read_json(root / 'manifest.json')
- manifest.update(status='SUCCESS', gpu_experiment='SUCCESS', active_stage='complete', completed=list(CHILDREN),
- completed_utc=datetime.now(timezone.utc).isoformat(),
- evidence_sha256=file_hash(root / 'calibration_evidence.json'))
- atomic_json(root / 'manifest.json', manifest)
- print('SUCCESS: E0/E1 pairing, fitted temperatures, metrics and 1,000 group bootstrap draws verified')
- return 0
- except Exception as error:
- if args.run_dir:
- manifest_path = args.run_dir / 'manifest.json'
- manifest = read_json(manifest_path) if manifest_path.exists() else {}
- manifest.update(status='FAILED', gpu_experiment='FAILED', active_stage='aggregation',
- error={'stage': 'aggregation', 'type': type(error).__name__, 'reason': str(error)})
- atomic_json(manifest_path, manifest)
- (args.run_dir / 'aggregation-error.log').write_text(traceback.format_exc(), encoding='utf-8')
- print('verification failed: ' + str(error), file=sys.stderr)
- return 1
-
-
-if __name__ == '__main__':
- raise SystemExit(main())
diff --git a/research/benchmarks/zh_reliability/calibration.py b/research/benchmarks/zh_reliability/calibration.py
deleted file mode 100644
index b015a44c..00000000
--- a/research/benchmarks/zh_reliability/calibration.py
+++ /dev/null
@@ -1,86 +0,0 @@
-"""Fit bounded scalar temperatures from unscaled calibration logits only."""
-import copy
-import math
-from .data import atomic_json, digest
-from .metrics import log_probs
-
-
-def fit_temperature(rows, minimum=30):
- from laya.common import TEMP_MIN, TEMP_MAX
- if len(rows) < minimum:
- return {'status': 'SKIPPED', 'n': len(rows), 'reason': 'insufficient calibration samples',
- 'temperature': None, 'boundary_hit': False}
- def loss(beta):
- return sum(-log_probs(r['raw_logits'], 1 / beta)[r['option_order'].index(r['gold'])]
- for r in rows) / len(rows)
- # NLL is convex in inverse temperature. Bounded golden-section, no new dependency.
- lo, hi = 1 / TEMP_MAX, 1 / TEMP_MIN
- ratio = (math.sqrt(5) - 1) / 2
- for _ in range(80):
- x, y = hi - ratio * (hi - lo), lo + ratio * (hi - lo)
- if loss(x) < loss(y):
- hi = y
- else:
- lo = x
- candidates = [TEMP_MIN, TEMP_MAX, 1., 1 / ((lo + hi) / 2)]
- t = min(candidates, key=lambda v: (loss(1 / v), abs(v - 1)))
- return {'status': 'SUCCESS', 'n': len(rows), 'temperature': t,
- 'boundary_hit': t in (TEMP_MIN, TEMP_MAX), 'nll_t1': loss(1.), 'nll_fitted': loss(1 / t)}
-
-
-def calibrate(rows, model, split, original, path=None, minimum=30, smoke=False):
- if minimum < 30 and not smoke:
- raise ValueError('minimum < 30 is smoke_only')
- if not rows or any(r['split'] != 'calibration' for r in rows):
- raise ValueError('fit only on calibration partition')
- if len({r['id'] for r in rows}) != len(rows):
- raise ValueError('duplicate calibration predictions')
- for r in rows:
- if (split['assignments'].get(r['id']) != 'calibration' or r.get('model') != model
- or r.get('sample_hash') != split['row_hashes'].get(r['id'])):
- raise ValueError('calibration model/partition mismatch')
- cfg = copy.deepcopy(original)
- base = cfg.get('lang_temperatures', {}).get('zh', {
- 'temperature': cfg['temperature'], 'temperature_by_options': cfg['temperature_by_options']})
- override = copy.deepcopy(base)
- fitted = {}
- for kind, index in [('choice', 0), ('noul', 2)]:
- fitted[kind] = fit_temperature([r for r in rows if r['question']['type'] == kind], minimum)
- if fitted[kind]['status'] == 'SUCCESS':
- override['temperature'][index] = fitted[kind]['temperature']
- override['temperature_by_options'] = {
- k: v for k, v in override['temperature_by_options'].items() if not k.startswith(kind + ':')}
- out = {'model': model, 'data_hash': split['data_hash'], 'split_hash': split['split_hash'],
- 'calibration_hash': digest([split['row_hashes'][r['id']] for r in rows]),
- 'calibration_ids': [r['id'] for r in rows], 'smoke_only': smoke,
- 'minimum_per_type': minimum, 'fitted': fitted, 'original': original,
- 'lang_temperatures': {**cfg.get('lang_temperatures', {}), 'zh': override}}
- if path:
- atomic_json(path, out)
- return out
-
-
-def validate_calibration(artifact, model, split, allow_smoke=False):
- from laya.common import clamp_temperature
- if artifact['model'] != model or artifact['data_hash'] != split['data_hash']:
- raise ValueError('calibration model or data hash mismatch')
- ids = artifact['calibration_ids']
- if (artifact['split_hash'] != split['split_hash'] or len(set(ids)) != len(ids)
- or any(split['assignments'].get(i) != 'calibration' for i in ids)
- or artifact['calibration_hash'] != digest([split['row_hashes'][i] for i in ids])):
- raise ValueError('calibration partition/hash mismatch')
- if artifact['smoke_only'] and not allow_smoke:
- raise ValueError('smoke calibration requires explicit --allow-smoke-calibration')
- for cfg in artifact['lang_temperatures'].values():
- if len(cfg['temperature']) != 3:
- raise ValueError('invalid temperature vector')
- for t in list(cfg['temperature']) + list(cfg['temperature_by_options'].values()):
- if not isinstance(t, (int, float)) or not math.isfinite(t) or clamp_temperature(t) != t:
- raise ValueError('illegal exported temperature')
- for kind, index in [('choice', 0), ('noul', 2)]:
- fit = artifact['fitted'][kind]
- cfg = artifact['lang_temperatures']['zh']
- if fit['status'] == 'SUCCESS' and (cfg['temperature'][index] != fit['temperature'] or
- any(k.startswith(kind + ':') for k in cfg['temperature_by_options'])):
- raise ValueError('fitted temperature shadowed by stale option bucket')
- return artifact['lang_temperatures']
diff --git a/research/benchmarks/zh_reliability/data.py b/research/benchmarks/zh_reliability/data.py
deleted file mode 100644
index 60ad8a55..00000000
--- a/research/benchmarks/zh_reliability/data.py
+++ /dev/null
@@ -1,189 +0,0 @@
-"""Strict fixture/real-data validation and immutable group partitions."""
-import hashlib
-import json
-import os
-from pathlib import Path
-import random
-import re
-import tempfile
-from collections import Counter
-
-SPLITS = ('train', 'dev', 'calibration', 'test')
-FIXTURES = Path(__file__).with_name('fixtures.jsonl')
-
-
-def digest(value):
- # Preserve criteria insertion order: order is part of the actual model input.
- return hashlib.sha256(json.dumps(value, ensure_ascii=False, separators=(',', ':')).encode()).hexdigest()
-
-
-def file_hash(path):
- h = hashlib.sha256()
- with open(path, 'rb') as f:
- for chunk in iter(lambda: f.read(1024 * 1024), b''):
- h.update(chunk)
- return h.hexdigest()
-
-
-def atomic_json(path, value):
- path = Path(path)
- path.parent.mkdir(parents=True, exist_ok=True)
- fd, tmp = tempfile.mkstemp(dir=path.parent, prefix='.' + path.name)
- try:
- with os.fdopen(fd, 'w', encoding='utf-8') as f:
- json.dump(value, f, ensure_ascii=False, indent=2, allow_nan=False)
- f.write('\n')
- os.replace(tmp, path)
- finally:
- if os.path.exists(tmp):
- os.unlink(tmp)
-
-
-def options(row):
- return ['false', 'true'] if row['question']['type'] == 'noul' else list(row['question']['criteria'])
-
-
-def upstream_annotation_verified(provenance):
- """Validate supplied source-review evidence, not a local review of the derived task."""
- if provenance.get('kind') != 'public_human_annotated' or provenance.get('human_verified') is not False:
- return False
- evidence = provenance.get('upstream_annotation')
- if not isinstance(evidence, dict):
- return False
- if (evidence.get('dataset') != 'MASSIVE' or evidence.get('version') != '1.1'
- or not isinstance(evidence.get('source_id'), str) or not evidence['source_id'].strip()
- or not isinstance(provenance.get('source_intent'), str) or not provenance['source_intent'].strip()
- or evidence.get('intent') != provenance['source_intent']):
- return False
- source_hash = evidence.get('source_row_hash')
- evidence_url = evidence.get('evidence_url')
- if (not isinstance(source_hash, str) or not re.fullmatch(r'[0-9a-f]{64}', source_hash)
- or not isinstance(evidence_url, str) or not re.fullmatch(
- r'https://(?:github\.com/alexa/massive/blob/|raw\.githubusercontent\.com/alexa/massive/)'
- r'[0-9a-f]{40}/README\.md(?:#[A-Za-z0-9_-]+)?', evidence_url)):
- return False
- judgments = evidence.get('judgments')
- if not isinstance(judgments, list) or len(judgments) < 3:
- return False
- workers = set()
- for judgment in judgments:
- if (not isinstance(judgment, dict) or not isinstance(judgment.get('worker_id'), str)
- or not judgment['worker_id'].strip() or judgment['worker_id'] in workers
- or type(judgment.get('intent_score')) is not int or judgment['intent_score'] != 1):
- return False
- workers.add(judgment['worker_id'])
- return True
-
-
-def read_data(path, formal=False):
- rows = [json.loads(line) for line in Path(path).read_text(encoding='utf-8').splitlines() if line.strip()]
- if not rows:
- raise ValueError('empty dataset')
- seen, tasks, families, relations = set(), {}, {}, {}
- for r in rows:
- for key in ('id', 'group_id', 'task_id', 'state'):
- if not isinstance(r.get(key), str) or not r[key].strip():
- raise ValueError('nonempty string required: ' + key)
- if r['id'] in seen:
- raise ValueError('duplicate id: ' + r['id'])
- seen.add(r['id'])
- q = r.get('question', {})
- if q.get('type') not in ('choice', 'noul') or not isinstance(q.get('instructions'), str):
- raise ValueError('only choice/noul with string instructions are supported')
- c = q.get('criteria')
- if not isinstance(c, dict) or len(c) < 2 or any(not isinstance(v, str) for v in c.values()):
- raise ValueError('criteria must map at least two stable option IDs to string descriptions')
- if any(not isinstance(k, str) or not k.strip() for k in c):
- raise ValueError('invalid option_id')
- if q['type'] == 'noul' and set(c) != {'false', 'true'}:
- raise ValueError('noul criteria must be [false, true]')
- if 'labels' in q:
- labels = q['labels']
- if (q['type'] != 'noul' or not isinstance(labels, dict) or set(labels) != {'false', 'true'}
- or any(not isinstance(x, str) or not x.strip() for x in labels.values())
- or len(set(labels.values())) != 2):
- raise ValueError('invalid noul display labels')
- if r.get('label') not in options(r):
- raise ValueError('label not in semantic options')
- if not isinstance(r.get('tags'), list) or any(not isinstance(t, str) for t in r['tags']):
- raise ValueError('tags must be a list of strings')
- p = r.get('provenance', {})
- if not isinstance(p.get('kind'), str) or type(p.get('human_verified')) is not bool:
- raise ValueError('provenance kind and human_verified required')
- if p['kind'].startswith('agent_authored') and p['human_verified']:
- raise ValueError('agent fixtures cannot claim human verification')
- upstream_verified = upstream_annotation_verified(p)
- if p['kind'] == 'public_human_annotated' and not upstream_verified:
- raise ValueError('public human annotations require uncontested pinned source-review evidence')
- if formal and (p['kind'].startswith('agent_authored') or not (p['human_verified'] or upstream_verified)
- or any(not isinstance(p.get(k), str) or not p[k].strip()
- for k in ('source', 'license', 'annotation_status'))
- or r.get('split') not in SPLITS):
- raise ValueError('formal data requires source, license, verified annotation and fixed split')
- task = (q['type'], tuple(sorted(c.items())), q['instructions'])
- if r['task_id'] in tasks and tasks[r['task_id']] != task:
- raise ValueError('task_id must have consistent semantic criteria/instructions')
- tasks[r['task_id']] = task
- family = p.get('template_family', r['group_id'])
- if family in families and families[family] != r['group_id']:
- raise ValueError('template family crosses groups')
- families[family] = r['group_id']
- rel = r.get('relation')
- if rel:
- if rel.get('kind') not in ('preserve', 'flip') or not isinstance(rel.get('pair_id'), str):
- raise ValueError('invalid relation')
- relations.setdefault(rel['pair_id'], []).append(r)
- for pair in relations.values():
- if len(pair) != 2 or len({r['group_id'] for r in pair}) != 1:
- raise ValueError('each relation must pair exactly two rows in the same group')
- if len({r['relation']['kind'] for r in pair}) != 1 or len({r['task_id'] for r in pair}) != 1:
- raise ValueError('inconsistent pair')
- if (pair[0]['label'] == pair[1]['label']) != (pair[0]['relation']['kind'] == 'preserve'):
- raise ValueError('relation contradicts labels')
- return rows
-
-
-def partition(rows, path=None, seed=20260924, formal=False):
- hashes = {r['id']: digest(r) for r in rows}
- content_hash = digest(hashes)
- groups = sorted({r['group_id'] for r in rows})
- if path and Path(path).exists():
- out = json.loads(Path(path).read_text())
- if out['data_hash'] != content_hash or out['row_hashes'] != hashes:
- raise ValueError('split manifest data hash mismatch')
- else:
- if formal:
- mapping = {}
- for r in rows:
- if r['group_id'] in mapping and mapping[r['group_id']] != r['split']:
- raise ValueError('fixed partitions split a group')
- mapping[r['group_id']] = r['split']
- else:
- random.Random(seed).shuffle(groups)
- n = len(groups)
- cuts = [round(n * .6), round(n * .7), round(n * .85), n]
- mapping, start = {}, 0
- for split, end in zip(SPLITS, cuts):
- mapping.update({g: split for g in groups[start:end]})
- start = end
- out = {'seed': seed, 'formal': formal, 'data_hash': content_hash, 'row_hashes': hashes,
- 'groups': mapping, 'assignments': {r['id']: mapping[r['group_id']] for r in rows}}
- if set(out['groups']) != set(groups) or set(out['assignments']) != set(hashes):
- raise ValueError('split manifest membership mismatch')
- for r in rows:
- s = out['assignments'][r['id']]
- if s not in SPLITS or s != out['groups'][r['group_id']] or (formal and s != r['split']):
- raise ValueError('group leakage or fixed split mismatch')
- if out['formal'] != formal:
- raise ValueError('split mode mismatch')
- out['counts'] = dict(Counter(out['assignments'].values()))
- out['group_counts'] = dict(Counter(out['groups'].values()))
- split_hash = digest(out['assignments'])
- if 'split_hash' in out and out['split_hash'] != split_hash:
- raise ValueError('split manifest assignment hash mismatch')
- out['split_hash'] = split_hash
- if any(not out['counts'].get(s) for s in SPLITS):
- raise ValueError('all four partitions must be nonempty')
- if path and not Path(path).exists():
- atomic_json(path, out)
- return out
diff --git a/research/benchmarks/zh_reliability/fixtures.jsonl b/research/benchmarks/zh_reliability/fixtures.jsonl
deleted file mode 100644
index dc37971e..00000000
--- a/research/benchmarks/zh_reliability/fixtures.jsonl
+++ /dev/null
@@ -1,80 +0,0 @@
-{"id": "explicit_refund_01_1", "group_id": "explicit_refund_01", "task_id": "explicit_refund", "state": "包裹还在路上,我决定不要了,请给我退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_01"}}
-{"id": "explicit_refund_01_2", "group_id": "explicit_refund_01", "task_id": "explicit_refund", "state": "包裹还在路上,我不想等了,麻烦退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_01"}}
-{"id": "explicit_refund_02_1", "group_id": "explicit_refund_02", "task_id": "explicit_refund", "state": "别退款,我只想知道维修进度。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_02"}}
-{"id": "explicit_refund_02_2", "group_id": "explicit_refund_02", "task_id": "explicit_refund", "state": "维修进度帮我查查,钱先别退。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_02"}}
-{"id": "explicit_refund_03_1", "group_id": "explicit_refund_03", "task_id": "explicit_refund", "state": "如果明天还没发货,我才申请退款,现在先等。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_03"}}
-{"id": "explicit_refund_03_2", "group_id": "explicit_refund_03", "task_id": "explicit_refund", "state": "明天还没发货再退,今天不要操作。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_03"}}
-{"id": "explicit_refund_04_1", "group_id": "explicit_refund_04", "task_id": "explicit_refund", "state": "我不是不想退款,请现在办理。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_04", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_04"}}
-{"id": "explicit_refund_04_2", "group_id": "explicit_refund_04", "task_id": "explicit_refund", "state": "别误会,我确实要把钱退回来。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_04", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_04"}}
-{"id": "explicit_refund_05_1", "group_id": "explicit_refund_05", "task_id": "explicit_refund", "state": "这单不要啦,退下款呗。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_05", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_05"}}
-{"id": "explicit_refund_05_2", "group_id": "explicit_refund_05", "task_id": "explicit_refund", "state": "这单不要啦,不用退款,送给朋友。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_05", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_05"}}
-{"id": "explicit_refund_06_1", "group_id": "explicit_refund_06", "task_id": "explicit_refund", "state": "Please refund 这笔重复扣款,谢谢。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_06", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_06"}}
-{"id": "explicit_refund_06_2", "group_id": "explicit_refund_06", "task_id": "explicit_refund", "state": "Please check 这笔重复扣款,暂时不退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_06", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_06"}}
-{"id": "explicit_refund_07_1", "group_id": "explicit_refund_07", "task_id": "explicit_refund", "state": "天气真热,快递员也辛苦,但买错尺寸了,请退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_07", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_07"}}
-{"id": "explicit_refund_07_2", "group_id": "explicit_refund_07", "task_id": "explicit_refund", "state": "天气真热,快递员也辛苦,尺寸刚好,不用退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_07", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_07"}}
-{"id": "explicit_refund_08_1", "group_id": "explicit_refund_08", "task_id": "explicit_refund", "state": "退款按钮在哪里?我只是看看,还没决定。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_08"}}
-{"id": "explicit_refund_08_2", "group_id": "explicit_refund_08", "task_id": "explicit_refund", "state": "退款按钮在哪里?我决定现在退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_08"}}
-{"id": "explicit_refund_09_1", "group_id": "explicit_refund_09", "task_id": "explicit_refund", "state": "客服说可以退款,但我选择换货,不退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_09", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_09"}}
-{"id": "explicit_refund_09_2", "group_id": "explicit_refund_09", "task_id": "explicit_refund", "state": "客服说可以退款,我选择退钱,不换货。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_09", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_09"}}
-{"id": "explicit_refund_10_1", "group_id": "explicit_refund_10", "task_id": "explicit_refund", "state": "我没有取消退款的打算,继续退吧。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_10", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_10"}}
-{"id": "explicit_refund_10_2", "group_id": "explicit_refund_10", "task_id": "explicit_refund", "state": "我打算取消退款,别继续退了。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_10", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_10"}}
-{"id": "explicit_refund_11_1", "group_id": "explicit_refund_11", "task_id": "explicit_refund", "state": "发票开错了,只重开发票,不涉及退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_11", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_11"}}
-{"id": "explicit_refund_11_2", "group_id": "explicit_refund_11", "task_id": "explicit_refund", "state": "发票不用重开了,这笔费用请直接退回。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_11", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_11"}}
-{"id": "explicit_refund_12_1", "group_id": "explicit_refund_12", "task_id": "explicit_refund", "state": "定金已经交了,活动也取消了,麻烦退定金。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_12"}}
-{"id": "explicit_refund_12_2", "group_id": "explicit_refund_12", "task_id": "explicit_refund", "state": "活动取消了,把我交的定金退回来吧。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["plain"], "relation": {"pair_id": "explicit_refund_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_12"}}
-{"id": "explicit_refund_13_1", "group_id": "explicit_refund_13", "task_id": "explicit_refund", "state": "上次说退款是气话,这次只查物流。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_13"}}
-{"id": "explicit_refund_13_2", "group_id": "explicit_refund_13", "task_id": "explicit_refund", "state": "别按上次的气话退款,现在帮我查物流就好。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_13"}}
-{"id": "explicit_refund_14_1", "group_id": "explicit_refund_14", "task_id": "explicit_refund", "state": "会员自动续费我没用过,申清退款。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["typo"], "relation": {"pair_id": "explicit_refund_14", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_14"}}
-{"id": "explicit_refund_14_2", "group_id": "explicit_refund_14", "task_id": "explicit_refund", "state": "会员续费没用过,申请把费用退回。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["typo"], "relation": {"pair_id": "explicit_refund_14", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_14"}}
-{"id": "explicit_refund_15_1", "group_id": "explicit_refund_15", "task_id": "explicit_refund", "state": "不排除以后退款,但今天没有这个要求。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_15"}}
-{"id": "explicit_refund_15_2", "group_id": "explicit_refund_15", "task_id": "explicit_refund", "state": "以后也许会退款,今天先不退。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["conditional"], "relation": {"pair_id": "explicit_refund_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_15"}}
-{"id": "explicit_refund_16_1", "group_id": "explicit_refund_16", "task_id": "explicit_refund", "state": "你说不能退款,我不同意,仍要求退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_16", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_16"}}
-{"id": "explicit_refund_16_2", "group_id": "explicit_refund_16", "task_id": "explicit_refund", "state": "你说不能退款,我同意了,不再要求退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["negation"], "relation": {"pair_id": "explicit_refund_16", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_16"}}
-{"id": "explicit_refund_17_1", "group_id": "explicit_refund_17", "task_id": "explicit_refund", "state": "订单 A 保留;订单 B 重复买了,请退 B。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_17", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_17"}}
-{"id": "explicit_refund_17_2", "group_id": "explicit_refund_17", "task_id": "explicit_refund", "state": "订单 A 和 B 都保留,重复那件送人。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["distractor"], "relation": {"pair_id": "explicit_refund_17", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_17"}}
-{"id": "explicit_refund_18_1", "group_id": "explicit_refund_18", "task_id": "explicit_refund", "state": "refund status 显示处理中,查一下即可,别新建申请。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_18", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_18"}}
-{"id": "explicit_refund_18_2", "group_id": "explicit_refund_18", "task_id": "explicit_refund", "state": "查一下已有 refund 的进度,不是要再申请。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["mixed_language"], "relation": {"pair_id": "explicit_refund_18", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_18"}}
-{"id": "explicit_refund_19_1", "group_id": "explicit_refund_19", "task_id": "explicit_refund", "state": "不退不是我的意思,我的意思就是退。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_19", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_19"}}
-{"id": "explicit_refund_19_2", "group_id": "explicit_refund_19", "task_id": "explicit_refund", "state": "我的意思就是不退,别理解反了。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["double_negation"], "relation": {"pair_id": "explicit_refund_19", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_19"}}
-{"id": "explicit_refund_20_1", "group_id": "explicit_refund_20", "task_id": "explicit_refund", "state": "东西破了,赔个配件就行,不必退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "false", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_20"}}
-{"id": "explicit_refund_20_2", "group_id": "explicit_refund_20", "task_id": "explicit_refund", "state": "东西破了,不要配件了,请退钱。", "question": {"type": "noul", "instructions": "用户是否明确提出当前退款请求?", "criteria": {"false": "没有提出当前退款请求,或明确表示不退款", "true": "明确提出当前退款请求"}}, "label": "true", "tags": ["colloquial"], "relation": {"pair_id": "explicit_refund_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "explicit_refund_20"}}
-{"id": "ticket_category_01_1", "group_id": "ticket_category_01", "task_id": "ticket_category", "state": "支付成功后网页又扣了一次钱,请核对账单。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_01"}}
-{"id": "ticket_category_01_2", "group_id": "ticket_category_01", "task_id": "ticket_category", "state": "账单上同一笔支付记了两遍,帮忙核对。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_01", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_01"}}
-{"id": "ticket_category_02_1", "group_id": "ticket_category_02", "task_id": "ticket_category", "state": "应用一点登录就闪退,换过网络也一样。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_02"}}
-{"id": "ticket_category_02_2", "group_id": "ticket_category_02", "task_id": "ticket_category", "state": "切了网络仍然登录闪退,看看程序哪里坏了。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_02", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_02"}}
-{"id": "ticket_category_03_1", "group_id": "ticket_category_03", "task_id": "ticket_category", "state": "新买的键盘一周没送到,查一下运输位置。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_03"}}
-{"id": "ticket_category_03_2", "group_id": "ticket_category_03", "task_id": "ticket_category", "state": "键盘寄出一周了还没到,帮我追踪包裹。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["plain", "option_order"], "relation": {"pair_id": "ticket_category_03", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_03"}}
-{"id": "ticket_category_04_1", "group_id": "ticket_category_04", "task_id": "ticket_category", "state": "不是软件坏了,是账单金额不对。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_04", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_04"}}
-{"id": "ticket_category_04_2", "group_id": "ticket_category_04", "task_id": "ticket_category", "state": "不是账单问题,是软件打开就报错。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_04", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_04"}}
-{"id": "ticket_category_05_1", "group_id": "ticket_category_05", "task_id": "ticket_category", "state": "如果包裹今天到就不用查;现在请处理无法启动的程序。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_05", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_05"}}
-{"id": "ticket_category_05_2", "group_id": "ticket_category_05", "task_id": "ticket_category", "state": "程序无法启动请先处理,物流等今天过后再说。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_05", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_05"}}
-{"id": "ticket_category_06_1", "group_id": "ticket_category_06", "task_id": "ticket_category", "state": "不是没收到包裹,我收到了;现在问如何更改昵称。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_06", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_06"}}
-{"id": "ticket_category_06_2", "group_id": "ticket_category_06", "task_id": "ticket_category", "state": "包裹已经签收,和物流无关,昵称怎么改?", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_06", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_06"}}
-{"id": "ticket_category_07_1", "group_id": "ticket_category_07", "task_id": "ticket_category", "state": "俺的快递跑哪去啦,好几天莫动静。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["colloquial", "option_order"], "relation": {"pair_id": "ticket_category_07", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_07"}}
-{"id": "ticket_category_07_2", "group_id": "ticket_category_07", "task_id": "ticket_category", "state": "快递好几天没动静,帮忙找找在哪。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["colloquial", "option_order"], "relation": {"pair_id": "ticket_category_07", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_07"}}
-{"id": "ticket_category_08_1", "group_id": "ticket_category_08", "task_id": "ticket_category", "state": "App 报 error 503,无法加载首页。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_08"}}
-{"id": "ticket_category_08_2", "group_id": "ticket_category_08", "task_id": "ticket_category", "state": "App 正常,只是 invoice 金额算错。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_08", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_08"}}
-{"id": "ticket_category_09_1", "group_id": "ticket_category_09", "task_id": "ticket_category", "state": "昨天喝了咖啡,今天天气不错,顺便问你们周日几点营业?", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_09", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_09"}}
-{"id": "ticket_category_09_2", "group_id": "ticket_category_09", "task_id": "ticket_category", "state": "天气不错,想问周日营业时间。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_09", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_09"}}
-{"id": "ticket_category_10_1", "group_id": "ticket_category_10", "task_id": "ticket_category", "state": "地址写错了,请联系承运方改配送地址。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_10", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_10"}}
-{"id": "ticket_category_10_2", "group_id": "ticket_category_10", "task_id": "ticket_category", "state": "配送地址填错,麻烦帮我联系快递改一下。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_10", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_10"}}
-{"id": "ticket_category_11_1", "group_id": "ticket_category_11", "task_id": "ticket_category", "state": "优惠券抵扣没有显示在最终付款账单中。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_11", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_11"}}
-{"id": "ticket_category_11_2", "group_id": "ticket_category_11", "task_id": "ticket_category", "state": "付款账单没有减去优惠券金额。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_11", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_11"}}
-{"id": "ticket_category_12_1", "group_id": "ticket_category_12", "task_id": "ticket_category", "state": "导出文件时程序卡死,重装也不行。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_12"}}
-{"id": "ticket_category_12_2", "group_id": "ticket_category_12", "task_id": "ticket_category", "state": "重装后导出还是卡死,需要排查程序。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_12", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_12"}}
-{"id": "ticket_category_13_1", "group_id": "ticket_category_13", "task_id": "ticket_category", "state": "我想了解你们公司的招聘渠道。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_13"}}
-{"id": "ticket_category_13_2", "group_id": "ticket_category_13", "task_id": "ticket_category", "state": "请告诉我应聘岗位应该去哪里看。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["option_order"], "relation": {"pair_id": "ticket_category_13", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_13"}}
-{"id": "ticket_category_14_1", "group_id": "ticket_category_14", "task_id": "ticket_category", "state": "快递没丢,不用找;发票税额有误,请核算。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "billing", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_14", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_14"}}
-{"id": "ticket_category_14_2", "group_id": "ticket_category_14", "task_id": "ticket_category", "state": "税额没错,不用算;快递丢了,请查找。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_14", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_14"}}
-{"id": "ticket_category_15_1", "group_id": "ticket_category_15", "task_id": "ticket_category", "state": "包裹若被退回再联系我,目前页面白屏才是问题。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_15"}}
-{"id": "ticket_category_15_2", "group_id": "ticket_category_15", "task_id": "ticket_category", "state": "页面白屏需要修复,包裹退回是还没发生的假设。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["conditional", "option_order"], "relation": {"pair_id": "ticket_category_15", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_15"}}
-{"id": "ticket_category_16_1", "group_id": "ticket_category_16", "task_id": "ticket_category", "state": "不能说系统没有故障,每次保存都会报错。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "technical", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_16", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_16"}}
-{"id": "ticket_category_16_2", "group_id": "ticket_category_16", "task_id": "ticket_category", "state": "系统保存每次报错,确实有故障。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "technical", "tags": ["double_negation", "option_order"], "relation": {"pair_id": "ticket_category_16", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_16"}}
-{"id": "ticket_category_17_1", "group_id": "ticket_category_17", "task_id": "ticket_category", "state": "物流显是签收,可我根本没拿到呀。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["typo", "option_order"], "relation": {"pair_id": "ticket_category_17", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_17"}}
-{"id": "ticket_category_17_2", "group_id": "ticket_category_17", "task_id": "ticket_category", "state": "物流显示已签收,我却没有收到。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "shipping", "tags": ["typo", "option_order"], "relation": {"pair_id": "ticket_category_17", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_17"}}
-{"id": "ticket_category_18_1", "group_id": "ticket_category_18", "task_id": "ticket_category", "state": "Tracking number 查不到,帮忙确认是否发货。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "shipping", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_18", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_18"}}
-{"id": "ticket_category_18_2", "group_id": "ticket_category_18", "task_id": "ticket_category", "state": "Tracking 正常,VAT invoice 少了一项费用。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["mixed_language", "option_order"], "relation": {"pair_id": "ticket_category_18", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_18"}}
-{"id": "ticket_category_19_1", "group_id": "ticket_category_19", "task_id": "ticket_category", "state": "我看了三篇教程也问过同事,仍想知道客服几点下班。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_19", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_19"}}
-{"id": "ticket_category_19_2", "group_id": "ticket_category_19", "task_id": "ticket_category", "state": "教程与同事都没告诉我,客服几点下班?", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "other", "tags": ["distractor", "option_order"], "relation": {"pair_id": "ticket_category_19", "kind": "preserve"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_19"}}
-{"id": "ticket_category_20_1", "group_id": "ticket_category_20", "task_id": "ticket_category", "state": "不用改付款记录,也不用修程序,我是来提个配色建议。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"billing": "账单、扣费或发票问题", "technical": "软件故障或报错", "shipping": "发货、运输或配送问题", "other": "其他咨询或建议"}}, "label": "other", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_20"}}
-{"id": "ticket_category_20_2", "group_id": "ticket_category_20", "task_id": "ticket_category", "state": "配色挺好,不用改,我要反馈付款记录金额错误。", "question": {"type": "choice", "instructions": "用户当前要求处理的问题属于哪一类?", "criteria": {"other": "其他咨询或建议", "shipping": "发货、运输或配送问题", "technical": "软件故障或报错", "billing": "账单、扣费或发票问题"}}, "label": "billing", "tags": ["negation", "option_order"], "relation": {"pair_id": "ticket_category_20", "kind": "flip"}, "provenance": {"kind": "agent_authored_fixture", "human_verified": false, "source": "original diagnostic fixture", "license": "Apache-2.0", "template_family": "ticket_category_20"}}
diff --git a/research/benchmarks/zh_reliability/massive_protocol.json b/research/benchmarks/zh_reliability/massive_protocol.json
deleted file mode 100644
index 43e65b09..00000000
--- a/research/benchmarks/zh_reliability/massive_protocol.json
+++ /dev/null
@@ -1,74 +0,0 @@
-{
- "version": "massive-alarm-calibration-v1",
- "model_revision": "e4e9ddf21a7b1903b7acffd8814ad4307bf63a67",
- "seed": 20260924,
- "max_len": 512,
- "head_max_len": 192,
- "dtype": "fp16",
- "evaluation_device": "RTX 2080 Ti",
- "calibration": {
- "minimum_per_type": 30,
- "n_per_type": 64,
- "bounds": [
- 0.5,
- 5.0
- ],
- "objective": "calibration partition NLL only",
- "smoke_only": false
- },
- "conditions": {
- "E0": "original checkpoint with runtime-clamped original temperatures",
- "E1": "original checkpoint + calibration-fitted temperatures"
- },
- "analysis": {
- "primary_metrics": [
- "nll",
- "accuracy"
- ],
- "secondary_metrics": [
- "macro_f1",
- "brier",
- "ece"
- ],
- "paired_group_bootstrap": {
- "repeats": 1000,
- "seed": 20260924,
- "interval": "percentile 95%",
- "min_groups": 20
- },
- "comparisons": [
- "E1-E0"
- ],
- "interpretation": "Exploratory fixed-checkpoint estimates on 89 heuristic source groups; intervals condition on the fitted temperatures and exclude calibration uncertainty; descriptive intervals without multiplicity correction.",
- "test_access": "Temperatures and evaluation settings locked before E0/E1 test evaluation; no settings selected using test."
- },
- "timing": {
- "conditions": [
- "E0"
- ],
- "warmup_batches": 10,
- "measured_batches": 186,
- "batch_decisions": 1,
- "note": "all 186 test decisions once per timed stage; short utterances under max_len512, not 512-token stress. GPUs not exclusively reserved."
- },
- "limits": [
- "public human-localized alarm subset, not naturally occurring customer-service logs",
- "upstream intent-match evidence, no independent local human review",
- "choice/noul share source text and group; not 942 independent utterances",
- "base model exposure to MASSIVE/SLURP unknown",
- "single seed, single task, no general Chinese improvement claim"
- ],
- "purpose": "Evaluation-only fixed settings. The runner records paths, hashes and time before predictions; train/dev partitions are retained for source split compatibility and are not consumed.",
- "expected_split_counts": {
- "train": 522,
- "dev": 106,
- "calibration": 128,
- "test": 186
- },
- "expected_group_counts": {
- "train": 247,
- "dev": 53,
- "calibration": 62,
- "test": 89
- }
-}
diff --git a/research/benchmarks/zh_reliability/metrics.py b/research/benchmarks/zh_reliability/metrics.py
deleted file mode 100644
index e323b2f5..00000000
--- a/research/benchmarks/zh_reliability/metrics.py
+++ /dev/null
@@ -1,126 +0,0 @@
-"""Metrics use valid decision probabilities, never action/entropy confidence."""
-import math
-from collections import defaultdict
-import numpy as np
-
-
-def log_probs(logits, temperature=1.0, mask=None):
- from laya.common import TEMP_MIN, TEMP_MAX
- if not math.isfinite(temperature) or not TEMP_MIN <= temperature <= TEMP_MAX:
- raise ValueError('temperature outside runtime range')
- z = np.asarray(logits, dtype=np.float64)
- if mask is not None:
- z = z[np.asarray(mask, dtype=bool)]
- if z.ndim != 1 or not len(z) or not np.isfinite(z).all():
- raise ValueError('invalid decision logits')
- z = z / temperature
- z -= z.max()
- return z - np.log(np.exp(z).sum())
-
-
-def probability_record(row, logits, temperature):
- from .data import digest, options
- order = options(row)
- if len(logits) != len(order):
- raise ValueError('logits/options mismatch')
- lp = log_probs(logits, temperature)
- p = np.exp(lp)
- return {**row, 'sample_hash': digest(row), 'option_order': order, 'gold': row['label'], 'raw_logits': list(logits),
- 'temperature': temperature, 'probabilities': p.tolist(), 'log_probabilities': lp.tolist(),
- 'predicted': order[int(p.argmax())], 'confidence': float(p.max()),
- 'p_true': float(p[1]) if row['question']['type'] == 'noul' else None}
-
-
-def quality(rows):
- if not rows:
- return {'n': 0, 'accuracy': None, 'macro_f1': None, 'nll': None, 'brier': None, 'ece': None,
- 'reason': 'empty group'}
- correct = np.array([r['predicted'] == r['gold'] for r in rows], dtype=float)
- conf = np.array([max(r['probabilities']) for r in rows])
- nll, brier = [], []
- task_rows = defaultdict(list)
- for r in rows:
- task_rows[r['task_id']].append(r)
- g = r['option_order'].index(r['gold'])
- nll.append(-r['log_probabilities'][g])
- brier.append(sum((p - (j == g)) ** 2 for j, p in enumerate(r['probabilities'])))
- f1 = []
- for task in task_rows.values():
- labels = sorted({x for r in task for x in r['option_order']})
- scores = []
- for label in labels:
- tp = sum(r['predicted'] == r['gold'] == label for r in task)
- fp = sum(r['predicted'] == label and r['gold'] != label for r in task)
- fn = sum(r['predicted'] != label and r['gold'] == label for r in task)
- scores.append(2 * tp / (2 * tp + fp + fn) if 2 * tp + fp + fn else 0.0)
- f1.append(float(np.mean(scores)))
- bucket = np.minimum(14, (conf * 15).astype(int))
- ece = sum(np.mean(bucket == b) * abs(correct[bucket == b].mean() - conf[bucket == b].mean())
- for b in range(15) if (bucket == b).any())
- # Ties broken lexically by stable ID, independently of labels/correctness.
- ordered = sorted(rows, key=lambda r: (-max(r['probabilities']), r['id']))
- risk = []
- for c in (.25, .5, .75, .9, 1.):
- k = max(1, math.ceil(c * len(rows)))
- risk.append({'target_coverage': c, 'coverage': k / len(rows), 'n': k,
- 'risk': sum(r['predicted'] != r['gold'] for r in ordered[:k]) / k})
- return {'n': len(rows), 'accuracy': float(correct.mean()), 'macro_f1': float(np.mean(f1)),
- 'nll': float(np.mean(nll)), 'brier': float(np.mean(brier)), 'ece': float(ece),
- 'risk_coverage': risk}
-
-
-def summarize(rows):
- result = {'overall': quality(rows), 'definitions': {
- 'macro_f1': 'equal mean of task-local macro F1; all task options, zero for undefined F1',
- 'brier': 'mean sum of squared errors over all valid classes (including both noul classes)',
- 'ece': '15 equal bins [i/15,(i+1)/15), last bin includes 1; confidence=max(p)',
- 'risk_ties': 'descending max(p), then lexical sample ID; ceil target count; diagnostic only'}}
- for name, getter in [('task', lambda r: [r['task_id']]),
- ('type', lambda r: [r['question']['type']]),
- ('option_count', lambda r: [str(len(r['option_order']))]),
- ('phenomenon', lambda r: r['tags']),
- ('truncation', lambda r: [str(r.get('tokens', {}).get('truncated', False))])]:
- groups = defaultdict(list)
- for r in rows:
- for key in getter(r):
- groups[key].append(r)
- if name == 'type':
- for key in ('choice', 'noul'):
- groups.setdefault(key, [])
- result['by_' + name] = {k: quality(v) for k, v in sorted(groups.items())}
- pairs = defaultdict(list)
- for r in rows:
- if r.get('relation'):
- pairs[r['relation']['pair_id']].append(r)
- result['relations'] = {}
- for kind in ('preserve', 'flip'):
- selected = [v for v in pairs.values() if len(v) == 2 and v[0]['relation']['kind'] == kind]
- result['relations'][kind] = {
- 'n_pairs': len(selected),
- 'consistent_or_changed': (sum((v[0]['predicted'] == v[1]['predicted']) == (kind == 'preserve')
- for v in selected) / len(selected)) if selected else None,
- 'both_correct': (sum(all(r['predicted'] == r['gold'] for r in v) for v in selected)
- / len(selected)) if selected else None}
- return result
-
-
-def paired_bootstrap(before, after, repeats=1000, seed=20260924, min_groups=20):
- a, b = {r['id']: r for r in before}, {r['id']: r for r in after}
- if a.keys() != b.keys() or any(a[k]['group_id'] != b[k]['group_id'] or a[k]['gold'] != b[k]['gold'] for k in a):
- raise ValueError('paired bootstrap requires identical samples/groups/labels')
- groups = sorted({r['group_id'] for r in before})
- if len(groups) < min_groups:
- return {'status': 'SKIPPED', 'reason': 'too few independent groups', 'n_groups': len(groups)}
- ids = {g: [k for k in a if a[k]['group_id'] == g] for g in groups}
- rng = np.random.default_rng(seed)
- keys = ('accuracy', 'macro_f1', 'nll', 'brier', 'ece')
- deltas = {k: [] for k in keys}
- for _ in range(repeats):
- selected = [i for g in rng.choice(groups, len(groups), replace=True) for i in ids[g]]
- qa, qb = quality([a[i] for i in selected]), quality([b[i] for i in selected])
- for k in keys:
- deltas[k].append(qb[k] - qa[k])
- qa, qb = quality(before), quality(after)
- return {'status': 'SUCCESS', 'n_groups': len(groups), 'repeats': repeats, 'seed': seed,
- 'after_minus_before': {k: {'delta': qb[k] - qa[k], 'ci95': np.quantile(v, [.025, .975]).tolist()}
- for k, v in deltas.items()}}
diff --git a/research/benchmarks/zh_reliability/prepare_massive.py b/research/benchmarks/zh_reliability/prepare_massive.py
deleted file mode 100644
index e31c180e..00000000
--- a/research/benchmarks/zh_reliability/prepare_massive.py
+++ /dev/null
@@ -1,357 +0,0 @@
-"""Prepare a licensed, source-reviewed MASSIVE 1.1 Chinese alarm subset (no model calls)."""
-import argparse
-from collections import Counter, defaultdict
-from datetime import datetime, timezone
-import json
-from pathlib import Path
-import random
-import re
-import shutil
-import sys
-import tarfile
-import unicodedata
-import urllib.request
-
-from .data import atomic_json, digest, file_hash, partition, read_data
-
-PUBLISHER_REVISION = 'f966f21846043aabef9b0f974fa7970027f43738'
-ARCHIVE_URL = 'https://amazon-massive-nlu-dataset.s3.amazonaws.com/amazon-massive-dataset-1.1.tar.gz'
-EVIDENCE_URL = f'https://github.com/alexa/massive/blob/{PUBLISHER_REVISION}/README.md'
-RAW_DOCS = f'https://raw.githubusercontent.com/alexa/massive/{PUBLISHER_REVISION}/'
-PARQUET_REVISION = 'ed58ac423a2f4121720918bf5301577edce4ffd3'
-PARQUET_BASE = f'https://huggingface.co/datasets/AmazonScience/massive/resolve/{PARQUET_REVISION}/'
-PARQUET_FILES = tuple(f'{locale}-{split}.parquet' for locale in ('zh-CN', 'en-US')
- for split in ('train', 'validation', 'test'))
-# The publisher NOTICE establishes that this license applies to MASSIVE.
-LICENSE_TEXT_URL = 'https://raw.githubusercontent.com/creativecommons/cc-legal-tools-data/b40ad648826d12d38207b111d66a708bfa6c5ce4/docs/licenses/by/4.0/legalcode.txt'
-INTENTS = {'alarm_set': '设置闹钟', 'alarm_query': '查询闹钟', 'alarm_remove': '删除或取消闹钟'}
-SOURCE_FILES = ('zh-CN.jsonl', 'en-US.jsonl', 'DATA_LICENSE', 'README.md', 'NOTICE.md', 'THIRD-PARTY.md', 'LICENSE.txt')
-
-
-def fetch_source(directory):
- directory = Path(directory)
- directory.mkdir()
- archive = directory / 'massive-1.1.tar.gz'
- with urllib.request.urlopen(ARCHIVE_URL, timeout=60) as source, archive.with_suffix('.part').open('wb') as out:
- shutil.copyfileobj(source, out)
- archive.with_suffix('.part').replace(archive)
- with tarfile.open(archive, 'r:gz') as tar:
- for member in tar:
- name = member.name
- target = None
- if name.endswith('/data/zh-CN.jsonl'):
- target = 'zh-CN.jsonl'
- elif name.endswith('/data/en-US.jsonl'):
- target = 'en-US.jsonl'
- elif name.endswith('/LICENSE'):
- target = 'DATA_LICENSE'
- if target:
- if not member.isfile() or (directory / target).exists():
- raise ValueError('unexpected/duplicate archive member: ' + name)
- # Read only exact members; never extract arbitrary archive paths/symlinks.
- with tar.extractfile(member) as src, (directory / target).open('wb') as out:
- shutil.copyfileobj(src, out)
- for name in SOURCE_FILES[3:]:
- with urllib.request.urlopen(RAW_DOCS + name, timeout=30) as src:
- (directory / name).write_bytes(src.read())
- metadata = {'archive_url': ARCHIVE_URL, 'archive_sha256': file_hash(archive),
- 'publisher_revision': PUBLISHER_REVISION,
- 'files': {name: file_hash(directory / name) for name in SOURCE_FILES}}
- atomic_json(directory / 'source_manifest.json', metadata)
- return directory
-
-
-def check_source(directory):
- directory = Path(directory)
- metadata = json.loads((directory / 'source_manifest.json').read_text())
- if metadata['archive_url'] != ARCHIVE_URL or metadata['publisher_revision'] != PUBLISHER_REVISION:
- raise ValueError('unexpected dataset version or documentation revision')
- transport = metadata.get('transport', 'publisher_archive')
- if transport not in ('publisher_archive', 'hf_parquet'):
- raise ValueError('unexpected source transport')
- if transport == 'hf_parquet' and (metadata.get('mirror_revision') != PARQUET_REVISION
- or metadata.get('mirror_base_url') != PARQUET_BASE
- or metadata.get('license_text_url') != LICENSE_TEXT_URL):
- raise ValueError('unexpected parquet source revision or license URL')
- source_files = SOURCE_FILES + (PARQUET_FILES if transport == 'hf_parquet' else ())
- for name in source_files:
- if metadata['files'].get(name) != file_hash(directory / name):
- raise ValueError('source file hash mismatch: ' + name)
- archive = directory / 'massive-1.1.tar.gz'
- if archive.exists() and file_hash(archive) != metadata['archive_sha256']:
- raise ValueError('source archive hash mismatch')
- if 'CC BY 4.0' not in (directory / 'NOTICE.md').read_text():
- raise ValueError('missing publisher dataset-license notice')
- if 'Attribution 4.0 International' not in (directory / 'DATA_LICENSE').read_text():
- raise ValueError('unexpected dataset license')
- return metadata
-
-
-def decode_parquet_row(row, features, locale, split):
- """Decode HF label indices and column-oriented sequences without changing source text."""
- row = dict(row)
- if row.get('locale') != locale or row.get('partition') != ('dev' if split == 'validation' else split):
- raise ValueError('parquet locale/partition mismatch')
- for field in ('scenario', 'intent'):
- feature, value = features[field], row[field]
- names = feature.get('names', [])
- if feature.get('_type') != 'ClassLabel' or type(value) is not int or not 0 <= value < len(names):
- raise ValueError('invalid parquet class label: ' + field)
- row[field] = names[value]
- for field in ('judgments', 'slot_method'):
- value = row.get(field)
- if not isinstance(value, dict) or not value or any(not isinstance(v, list) for v in value.values()):
- raise ValueError('invalid parquet sequence: ' + field)
- if len({len(v) for v in value.values()}) != 1:
- raise ValueError('unequal parquet sequence lengths: ' + field)
- row[field] = [dict(zip(value, values)) for values in zip(*value.values())]
- return row
-
-
-def prepare_parquet_source(directory):
- """Decode six pinned files and record artifact and decoded hashes; no network calls."""
- import pyarrow
- import pyarrow.parquet as pq
- directory = Path(directory)
- for locale in ('zh-CN', 'en-US'):
- rows = []
- for split in ('train', 'validation', 'test'):
- table = pq.read_table(directory / f'{locale}-{split}.parquet')
- features = json.loads(table.schema.metadata[b'huggingface'])['info']['features']
- rows.extend(decode_parquet_row(row, features, locale, split) for row in table.to_pylist())
- write_jsonl(directory / f'{locale}.jsonl', rows)
- metadata = {'archive_url': ARCHIVE_URL, 'publisher_revision': PUBLISHER_REVISION,
- 'transport': 'hf_parquet', 'mirror_revision': PARQUET_REVISION, 'mirror_base_url': PARQUET_BASE,
- 'license_text_url': LICENSE_TEXT_URL, 'pyarrow_version': pyarrow.__version__,
- 'decoding': 'ClassLabel indices use embedded feature names; column-oriented judgments/slot_method are transposed to lists. Row hashes cover decoded JSON, not original archive bytes.',
- 'files': {name: file_hash(directory / name) for name in SOURCE_FILES + PARQUET_FILES}}
- atomic_json(directory / 'source_manifest.json', metadata)
- return directory
-
-
-def fetch_parquet_source(directory):
- directory = Path(directory)
- directory.mkdir()
- downloads = {f'{locale}-{split}.parquet': f'{PARQUET_BASE}{locale}/{split}/0000.parquet'
- for locale in ('zh-CN', 'en-US') for split in ('train', 'validation', 'test')}
- downloads.update({name: RAW_DOCS + name for name in SOURCE_FILES[3:]})
- downloads['DATA_LICENSE'] = LICENSE_TEXT_URL
- for name, url in downloads.items():
- target = directory / name
- temporary = target.with_suffix(target.suffix + '.part')
- with urllib.request.urlopen(url, timeout=60) as src, temporary.open('wb') as out:
- shutil.copyfileobj(src, out)
- temporary.replace(target)
- return prepare_parquet_source(directory)
-
-
-def normalized(text):
- text = unicodedata.normalize('NFKC', text).casefold()
- return ''.join(c for c in text if not c.isspace() and not unicodedata.category(c).startswith('P'))
-
-
-def source_template(row):
- # Slot values in English source annotations must not create independent groups.
- return normalized(re.sub(r'\[\s*([^:\]]+?)\s*:\s*[^\]]*\]', r'[\1]', row['annot_utt']))
-
-
-def records(path):
- rows = [json.loads(line) for line in Path(path).read_text(encoding='utf-8').splitlines() if line.strip()]
- if len({r['id'] for r in rows}) != len(rows):
- raise ValueError('duplicate source ID')
- return rows
-
-
-def confirmed(row):
- judgments = row.get('judgments', [])
- return (isinstance(judgments, list) and len(judgments) >= 3
- and all(isinstance(j, dict) and isinstance(j.get('worker_id'), str) and j['worker_id'].strip()
- and type(j.get('intent_score')) is int and j['intent_score'] == 1 for j in judgments)
- and len({j['worker_id'] for j in judgments}) == len(judgments))
-
-
-def convert(chinese, english, source, seed=20260924, calibration_fraction=.20):
- if not 0 < calibration_fraction < 1:
- raise ValueError('calibration fraction must be between zero and one')
- en = {r['id']: r for r in english}
- selected = [r for r in chinese if r.get('scenario') == 'alarm' and r.get('intent') in INTENTS]
- if not selected or len({r['id'] for r in selected}) != len(selected):
- raise ValueError('empty alarm selection or duplicate source IDs')
- parent = {r['id']: r['id'] for r in selected}
-
- def root(i):
- while parent[i] != i:
- parent[i] = parent[parent[i]]
- i = parent[i]
- return i
-
- anchors, labels = {}, defaultdict(set)
- for r in selected:
- e = en.get(r['id'])
- if (r.get('locale') != 'zh-CN' or r.get('partition') not in ('train', 'dev', 'test')
- or not isinstance(r.get('utt'), str) or not normalized(r['utt']) or not e
- or e.get('locale') != 'en-US' or e['partition'] != r['partition'] or e['intent'] != r['intent']):
- raise ValueError('Chinese/English source mismatch: ' + str(r['id']))
- chinese_key = normalized(r['utt'])
- labels[chinese_key].add(r['intent'])
- for key in (('zh', chinese_key), ('en_template', source_template(e))):
- if key in anchors:
- parent[root(r['id'])] = root(anchors[key])
- else:
- anchors[key] = r['id']
- groups = defaultdict(list)
- for r in selected:
- groups[root(r['id'])].append(r)
- decisions, diagnostic = [], []
- priority = {'train': 0, 'dev': 1, 'test': 2}
- for members in groups.values():
- owner = max((r['partition'] for r in members), key=priority.get)
- group_id = 'massive-alarm-' + digest(sorted(r['id'] for r in members))[:20]
- conflict = any(len(labels[normalized(r['utt'])]) > 1 for r in members)
- for r in members:
- reasons = []
- if r['partition'] != owner:
- reasons.append('shares_source_template_or_text_with_' + owner)
- if conflict:
- reasons.append('conflicting_intents_for_identical_normalized_chinese')
- if not confirmed(r):
- reasons.append('upstream_intent_review_not_unanimous_or_missing')
- if reasons:
- diagnostic.append({'source_id': r['id'], 'group_id': group_id, 'reasons': reasons,
- 'original_partition': r['partition'], 'source_row': r})
- else:
- decisions.append((r, group_id, owner))
- train_groups = sorted({g for _, g, s in decisions if s == 'train'})
- if len(train_groups) < 2:
- raise ValueError('insufficient independent train groups after source-leakage quarantine')
- random.Random(seed).shuffle(train_groups)
- cal_groups = set(train_groups[:max(1, min(len(train_groups) - 1, round(len(train_groups) * calibration_fraction)))])
- output = []
- for r, group_id, original_split in sorted(decisions, key=lambda x: x[0]['id']):
- split = 'calibration' if group_id in cal_groups else original_split
- provenance = {'kind': 'public_human_annotated', 'human_verified': False,
- 'source': ARCHIVE_URL, 'license': 'CC-BY-4.0', 'source_intent': r['intent'],
- 'annotation_status': 'Unanimous upstream human intent-match judgments; derived mapping has no independent local human review',
- 'local_human_review': False, 'template_family': group_id,
- 'source_partition': original_split, 'source_manifest_hash': digest(source),
- 'source_transport': source.get('transport', 'publisher_archive'),
- 'upstream_annotation': {'dataset': 'MASSIVE', 'version': '1.1', 'source_id': r['id'],
- 'intent': r['intent'], 'source_row_hash': digest(r), 'evidence_url': EVIDENCE_URL,
- 'judgments': r['judgments']}}
- if 'archive_sha256' in source:
- provenance['source_archive_sha256'] = source['archive_sha256']
- if source.get('transport') == 'hf_parquet':
- provenance['source_mirror_revision'] = source['mirror_revision']
- common = {'group_id': group_id, 'state': r['utt'], 'split': split,
- 'tags': ['public_localized', 'alarm'], 'provenance': provenance}
- output.append({**common, 'id': 'massive-' + r['id'] + '-choice', 'task_id': 'massive_alarm_choice_v1',
- 'question': {'type': 'choice', 'instructions': '这句话的主要意图是哪一种闹钟操作?', 'criteria': dict(INTENTS)},
- 'label': r['intent']})
- output.append({**common, 'id': 'massive-' + r['id'] + '-noul', 'task_id': 'massive_alarm_set_noul_v1',
- 'question': {'type': 'noul', 'instructions': '这句话的主要意图是否是设置闹钟?',
- 'criteria': {'false': '意图是查询、删除或取消闹钟', 'true': '意图是设置闹钟'}},
- 'label': 'true' if r['intent'] == 'alarm_set' else 'false'})
- summary = {'source_zh_rows': len(chinese), 'selected_alarm_source_rows': len(selected),
- 'retained_source_rows': len(decisions), 'diagnostic_source_rows': len(diagnostic),
- 'source_groups_before_quarantine': len(groups), 'derived_decisions': len(output),
- 'source_partition_counts': dict(Counter(r['partition'] for r in selected)),
- 'excluded_reasons': dict(Counter(reason for r in diagnostic for reason in r['reasons'])),
- 'seed': seed, 'calibration_fraction_of_remaining_train_groups': calibration_fraction,
- 'local_human_review': False, 'scope': 'MASSIVE-derived source-reviewed alarm subset; not official full benchmark or natural customer-service logs',
- 'grouping_limit': 'source IDs, exact normalized Chinese and delexicalized English templates; undisclosed semantic/template relations may remain',
- 'base_model_training_overlap': 'unknown; no unseen-data claim',
- 'by_split': {s: {'decisions': sum(r['split'] == s for r in output),
- 'source_rows': sum(('calibration' if g in cal_groups else t) == s for _, g, t in decisions),
- 'groups': len({r['group_id'] for r in output if r['split'] == s}),
- 'source_intents': dict(Counter(r['label'] for r in output if r['split'] == s and r['question']['type'] == 'choice'))}
- for s in ('train', 'dev', 'calibration', 'test')}}
- return output, diagnostic, summary
-
-
-def write_jsonl(path, rows):
- with Path(path).open('w', encoding='utf-8') as f:
- for row in rows:
- f.write(json.dumps(row, ensure_ascii=False, allow_nan=False) + '\n')
-
-
-def main(argv=None):
- p = argparse.ArgumentParser(description=__doc__)
- p.add_argument('--source-dir', type=Path, help='existing source with source_manifest.json; default offline')
- p.add_argument('--download', action='store_true', help='download publisher release into this new run')
- p.add_argument('--download-parquet', action='store_true', help='download only zh-CN/en-US from pinned publisher HF mirror; requires pyarrow')
- p.add_argument('--run-dir', type=Path, required=True, help='new output directory')
- p.add_argument('--seed', type=int, default=20260924)
- p.add_argument('--calibration-fraction', type=float, default=.20)
- args = p.parse_args(argv)
- if sum((bool(args.source_dir), args.download, args.download_parquet)) != 1:
- p.error('choose exactly one of --source-dir, --download or --download-parquet')
- run = args.run_dir
- if run.exists():
- print('refusing existing output directory: ' + str(run), file=sys.stderr)
- return 2
- run.mkdir(parents=True)
- manifest = {'status': 'RUNNING', 'command': 'prepare-massive', 'gpu_experiment': 'NOT_RUN',
- 'created_utc': datetime.now(timezone.utc).isoformat(), 'completed': [],
- 'converter_sha256': file_hash(__file__),
- 'validation_sha256': file_hash(Path(__file__).with_name('data.py')),
- 'config': {k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items()}}
- atomic_json(run / 'manifest.json', manifest)
- stage = 'source'
- try:
- if args.download:
- source_dir = fetch_source(run / 'source')
- elif args.download_parquet:
- source_dir = fetch_parquet_source(run / 'source')
- else:
- source_dir = args.source_dir
- source = check_source(source_dir)
- atomic_json(run / 'source_manifest.json', source)
- license_dir = run / 'attribution'
- license_dir.mkdir()
- for name in SOURCE_FILES[2:]:
- shutil.copyfile(source_dir / name, license_dir / name)
- (license_dir / 'DERIVATION.md').write_text(
- '# Derived MASSIVE alarm subset\n\n'
- 'MASSIVE Copyright Amazon.com, Inc. or its affiliates. Source: ' + ARCHIVE_URL + '\n\n'
- 'Dataset and underlying SLURP text: CC BY 4.0; see NOTICE.md and DATA_LICENSE.\n\n'
- 'Changes: select three alarm intents with unanimous upstream intent-match reviews; '
- 'quarantine template overlaps and conflicting normalized utterances; derive choice and noul '
- 'questions; split calibration groups from original train. Original Chinese text and '
- 'intent labels remain unchanged. No local human review was performed.\n\n'
- 'Transport and decoding: see ../source_manifest.json. For Parquet, DATA_LICENSE is '
- 'the CC legal text fetched from its recorded official URL, not archive/LICENSE bytes.\n',
- encoding='utf-8')
- manifest['completed'].append('source hashes and publisher attribution')
- stage = 'conversion'
- rows, diagnostics, summary = convert(records(source_dir / 'zh-CN.jsonl'), records(source_dir / 'en-US.jsonl'),
- source, args.seed, args.calibration_fraction)
- write_jsonl(run / 'data.jsonl', rows)
- write_jsonl(run / 'diagnostic.jsonl', diagnostics)
- # Full normal importer validation is a gate, not just converter-specific assertions.
- validated = read_data(run / 'data.jsonl', formal=True)
- split = partition(validated, run / 'split_manifest.json', args.seed, formal=True)
- summary.update(data_hash=split['data_hash'], split_hash=split['split_hash'])
- atomic_json(run / 'metrics.json', summary)
- manifest.update(status='SUCCESS', data_sha256=file_hash(run / 'data.jsonl'),
- diagnostic_sha256=file_hash(run / 'diagnostic.jsonl'))
- manifest['completed'].append('source-reviewed conversion and fixed group partition validation')
- atomic_json(run / 'manifest.json', manifest)
- (run / 'report.md').write_text('# MASSIVE Chinese alarm data preparation\n\n'
- 'Public human-localized data; no local human review or model evaluation has been performed.\n\n'
- 'The original Chinese utterances and publisher intent labels are preserved. '
- 'Choice and noul are deterministic views of the same source and share a group. '
- 'Official test/dev membership is retained for eligible rows; lower-partition template overlaps '
- 'and contested/missing human reviews remain in diagnostic.jsonl. This is a filtered subset, '
- 'not the full MASSIVE benchmark.\n\n```json\n' + json.dumps(summary, ensure_ascii=False, indent=2)
- + '\n```\n\nAttribution: Amazon MASSIVE and SLURP, CC BY 4.0; see attribution/.\n', encoding='utf-8')
- print(str(run.resolve()))
- return 0
- except Exception as e:
- manifest.update(status='FAILED', error={'stage': stage, 'type': type(e).__name__, 'reason': str(e)})
- atomic_json(run / 'manifest.json', manifest)
- print(f'{stage}: {e}; see {run / "manifest.json"}', file=sys.stderr)
- return 1
-
-
-if __name__ == '__main__':
- raise SystemExit(main())
diff --git a/research/benchmarks/zh_reliability/reproduce_massive.sh b/research/benchmarks/zh_reliability/reproduce_massive.sh
deleted file mode 100644
index dae116a1..00000000
--- a/research/benchmarks/zh_reliability/reproduce_massive.sh
+++ /dev/null
@@ -1,77 +0,0 @@
-#!/usr/bin/env bash
-set -Eeuo pipefail
-# Relative paths are interpreted from the repository root. Outputs must be new.
-BENCH_REPO=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../../.." && pwd)
-cd "$BENCH_REPO"
-EXP_DIR=${1:?Usage: bash research/benchmarks/zh_reliability/reproduce_massive.sh OUTPUT_DIR [MODEL_DIR [DATA_JSONL [SPLIT_JSON]]]}
-BENCH_PY=${LAYA_ZH_PYTHON:-.venv/bin/python}
-BENCH_BASE=${2:-${LAYA_ZH_MODEL:-runs/zh-download/e4e9ddf21a7b1903b7acffd8814ad4307bf63a67}}
-BENCH_DATA=${3:-${LAYA_ZH_DATA:-runs/massive-alarm-data/data.jsonl}}
-BENCH_SPLIT=${4:-${LAYA_ZH_SPLIT:-runs/massive-alarm-data/split_manifest.json}}
-BENCH_REVISION=e4e9ddf21a7b1903b7acffd8814ad4307bf63a67
-BENCH_DEVICE=${LAYA_ZH_DEVICE:-cuda:0}
-mkdir -- "$EXP_DIR"
-BENCH_STAGE=initialization
-mark_failed() {
- BENCH_EXIT_CODE=$?
- "$BENCH_PY" - "$EXP_DIR" "$BENCH_STAGE" "$BENCH_EXIT_CODE" <<'PY_FAIL'
-import json
-from pathlib import Path
-import sys
-from research.benchmarks.zh_reliability.data import atomic_json
-root, stage, code = Path(sys.argv[1]), sys.argv[2], int(sys.argv[3])
-path = root / 'manifest.json'
-manifest = json.loads(path.read_text()) if path.exists() else {}
-manifest.update(status='FAILED', gpu_experiment='FAILED', active_stage=stage,
- error={'stage': stage, 'exit_code': code})
-atomic_json(path, manifest)
-PY_FAIL
- return "$BENCH_EXIT_CODE"
-}
-trap mark_failed ERR
-"$BENCH_PY" - "$EXP_DIR" "$BENCH_BASE" "$BENCH_DATA" "$BENCH_SPLIT" "$BENCH_DEVICE" <<'PY_INIT'
-from datetime import datetime, timezone
-import json
-from pathlib import Path
-import sys
-from research.benchmarks.zh_reliability.data import atomic_json, file_hash, partition, read_data
-
-root, model, data, split_path = (Path(value).resolve() for value in sys.argv[1:5])
-template = Path('research/benchmarks/zh_reliability/massive_protocol.json')
-protocol = json.loads(template.read_text(encoding='utf-8'))
-rows = read_data(data, formal=True)
-split = partition(rows, split_path, protocol['seed'], formal=True)
-if split['counts'] != protocol['expected_split_counts'] or split['group_counts'] != protocol['expected_group_counts']:
- raise ValueError('data partitions do not match the fixed MASSIVE alarm protocol')
-lineage = json.loads((model / 'zh_lineage.json').read_text(encoding='utf-8'))
-if (lineage.get('model_id') != 'convaiinnovations/laya-multilingual'
- or lineage.get('base_revision') != protocol['model_revision'] or lineage.get('trained') is not False):
- raise ValueError('use the pinned original multilingual checkpoint with its zh_lineage.json')
-protocol['evaluation_device'] = sys.argv[5]
-protocol.update(created_utc=datetime.now(timezone.utc).isoformat(), status='FIXED_BEFORE_MODEL_PREDICTIONS',
- data=str(data), data_file_sha256=file_hash(data), split_manifest=str(split_path),
- split_file_sha256=file_hash(split_path), base_model=str(model),
- base_model_weights_sha256=file_hash(model / 'model.safetensors'),
- base_model_lineage_sha256=file_hash(model / 'zh_lineage.json'),
- template_sha256=file_hash(template))
-atomic_json(root / 'protocol.json', protocol)
-atomic_json(root / 'manifest.json', {
- 'status': 'RUNNING', 'gpu_experiment': 'RUNNING', 'command': 'formal-massive-calibration-replay',
- 'completed': [], 'protocol_sha256': file_hash(root / 'protocol.json')})
-PY_INIT
-run_benchmark() {
- "$BENCH_PY" -m research.benchmarks.zh_reliability "$@" --data "$BENCH_DATA" --formal \
- --split-manifest "$BENCH_SPLIT" --max-len 512 --head-max-len 192 --dtype fp16 \
- --seed 20260924 --revision "$BENCH_REVISION"
-}
-BENCH_STAGE=base-calibration
-run_benchmark calibrate --model "$BENCH_BASE" --device "$BENCH_DEVICE" --minimum 30 \
- --run-dir "$EXP_DIR/base-calibration"
-BENCH_STAGE=E0
-run_benchmark eval --model "$BENCH_BASE" --device "$BENCH_DEVICE" --split test \
- --measure --warmup 10 --batches 186 --run-dir "$EXP_DIR/E0"
-BENCH_STAGE=E1
-run_benchmark eval --model "$BENCH_BASE" --device "$BENCH_DEVICE" --split test \
- --calibration "$EXP_DIR/base-calibration/calibration.json" --run-dir "$EXP_DIR/E1"
-BENCH_STAGE=aggregation
-"$BENCH_PY" -m research.benchmarks.zh_reliability.aggregate_massive "$EXP_DIR"
diff --git a/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json
deleted file mode 100644
index 35377d4b..00000000
--- a/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.json
+++ /dev/null
@@ -1,776 +0,0 @@
-{
- "schema": "massive-alarm-calibration-evidence-v1",
- "provenance": {
- "measurement_date": "2026-09-24",
- "execution_base_commit": "23a17522aa4942da6cce53a995a275760320b691",
- "contribution_commit": "37cc58465da5cdabddf1ff543234397d666ad546",
- "version_note": "Historical inference was executed on 2026-09-24 from base commit 23a17522aa4942da6cce53a995a275760320b691 with a dirty contribution worktree subsequently recorded in 37cc58465da5cdabddf1ff543234397d666ad546. The recorded execution source-file hashes are authoritative for the measured code. These measurements were not rerun on the later upstream rebase or evaluation-only refactor.",
- "execution_worktree_dirty": true,
- "execution_source_hashes": {
- "__main__.py": "2db327517919084fa3257659a168442024952fd36c649a37869cd460275cf692",
- "calibration.py": "0bd68229d859cf55b0826a43f1a43789edc9cb9c07ee849e2144e6ad747263c7",
- "data.py": "baf8061a097c0993fe31d8c7d69db8ccddd6d714075f3fc717b8ddf12ce11c66",
- "metrics.py": "8c0be190b7c9e1d55ce5ebe7a5b8aff6ab8c8ae24183ecb59f7838cf7867925f",
- "prepare_massive.py": "a83c5f74285b6ccc03851efebbbc9a77bf220d891d78ff8e4faddf9a22944e95",
- "runtime.py": "8c28e4d8113d3fc25ad1ca8f06d1d8254a4334fbeaff23e93c8098db0b25cfa9"
- },
- "original_file_sha256": {
- "base-calibration/manifest.json": "106c40af31bdfbc436ce68c6482ec36cef747566ff6d428f5335cdc2061da0ac",
- "base-calibration/predictions.jsonl": "c763309e5f81a8cc97656c89d8a7ee945914ca601a9c9cc6402ce77235315e35",
- "base-calibration/metrics.json": "07d3ea93222c202d1bc95757fee99c064c9448bc324eaaa759f5d06429426384",
- "base-calibration/split_manifest.json": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",
- "base-calibration/model_identity.json": "144087e7a313e8b144e2dc5296d502750c6f1d2223d45910038ccdddcfcabf19",
- "base-calibration/environment.json": "0b8b2a2d2cc8b2fdeef997c608b92309fce3fe8bdb2fde2b2fdaa1a29dd362b2",
- "E0/manifest.json": "e3c9a4791c3c2f152944d2245e6cc05a8b78e881de1283f90fe50f438722fb78",
- "E0/predictions.jsonl": "6524bdd74d10b513e6b6cb468f1993a8ff1d74abfda69eb767efdd41213c6f96",
- "E0/metrics.json": "b5bd3a28d867745e4f44c25bc1fdf4e395cf328b95b384f7610567f9635bd933",
- "E0/split_manifest.json": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",
- "E0/model_identity.json": "144087e7a313e8b144e2dc5296d502750c6f1d2223d45910038ccdddcfcabf19",
- "E0/environment.json": "0b8b2a2d2cc8b2fdeef997c608b92309fce3fe8bdb2fde2b2fdaa1a29dd362b2",
- "E1/manifest.json": "b60a71273d9f3a5d48debe5040107a36af045d8ed7447612e3b576242bf6f284",
- "E1/predictions.jsonl": "393009f41ba9e9d44f0ad88926b85f95896f40c922d1bfbf24759669a162b763",
- "E1/metrics.json": "b6d6cc8bfe1a3376648b17d79ac25b7706a0c98ca586a2c3b6e6d5a891cf7451",
- "E1/split_manifest.json": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",
- "E1/model_identity.json": "144087e7a313e8b144e2dc5296d502750c6f1d2223d45910038ccdddcfcabf19",
- "E1/environment.json": "0b8b2a2d2cc8b2fdeef997c608b92309fce3fe8bdb2fde2b2fdaa1a29dd362b2",
- "base-calibration/calibration.json": "b5e8b409cb5fb340a4911109327f166c85437307a5684235509ca0c1ada92ae2"
- },
- "dataset": "MASSIVE 1.1 zh-CN alarm subset",
- "dataset_license": "CC-BY-4.0",
- "publisher_notice": "https://github.com/alexa/massive/blob/f966f21846043aabef9b0f974fa7970027f43738/NOTICE.md",
- "data_hash": "dd800e8bc62366cbcf06907e63a1cf225c9aa1add35aa9892fe567e07d470b16",
- "split_hash": "9abde4aa5ecbc90de2929617b72f380ab18f51b5bac38659f7c5641c7739c707",
- "data_file_sha256": "e97c52a226d202a6a3a654d6645b7caff5b3a48952258e8d67149778faa4939a",
- "split_file_sha256": "65bf8caeaf296e36e6a7951b1d36b025fe9297a410776492a3a9a02332551486",
- "model": {
- "model_id": "convaiinnovations/laya-multilingual",
- "revision": "e4e9ddf21a7b1903b7acffd8814ad4307bf63a67",
- "content_hash": "27d6023d6a2033c2c8e504d88ede242c0f7faf7ffd85781540d5fb935bbde855",
- "files": {
- "model.safetensors": "9d628fd971b700382ac6f65920a86f149777b2e748e0c955fb3b19695aa8f204",
- "rl_agent_config.json": "25061739243b617ad88d1219ba6f8a9c86c5881ca28df024fa2d9b3b2fcc30c6",
- "encoder/config.json": "83f6916d13ef0f556ac461f28308dc2bffa7ebeadee8ec9e2db5812020ea5bb4",
- "tokenizer/tokenizer.json": "609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f",
- "tokenizer/tokenizer_config.json": "424b69444bf7b5809dc2cd2e36d0bd71b8055124dd24274d6db3c655d38205e7"
- },
- "trained": false
- },
- "environment": {
- "python": "3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]",
- "os": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.39",
- "dependencies": {
- "torch": "2.7.1+cu126",
- "transformers": "4.57.6",
- "numpy": "2.5.3",
- "safetensors": "0.8.0",
- "huggingface_hub": "0.36.2",
- "tokenizers": "0.22.2"
- },
- "cuda": "12.6",
- "cudnn": 90501
- },
- "evaluation_config": {
- "seed": 20260924,
- "max_len": 512,
- "head_max_len": 192,
- "dtype": "fp16",
- "device": "RTX 2080 Ti"
- }
- },
- "analysis": {
- "paired_group_bootstrap": {
- "repeats": 1000,
- "seed": 20260924,
- "min_groups": 20
- }
- },
- "calibration": {
- "minimum_per_type": 30,
- "bounds": [
- 0.5,
- 5.0
- ],
- "fitted": {
- "choice": {
- "status": "SUCCESS",
- "n": 64,
- "temperature": 3.1355035588563744,
- "boundary_hit": false,
- "nll_t1": 1.21606373270805,
- "nll_fitted": 0.771914452059509
- },
- "noul": {
- "status": "SUCCESS",
- "n": 64,
- "temperature": 5.0,
- "boundary_hit": true,
- "nll_t1": 1.5203078336357765,
- "nll_fitted": 0.7411119749553794
- }
- },
- "original": {
- "temperature_raw": [
- 1.0,
- 1.0,
- 1.0
- ],
- "temperature_by_options_raw": {},
- "temperature": [
- 1.0,
- 1.0,
- 1.0
- ],
- "temperature_by_options": {},
- "lang_temperatures": {}
- }
- },
- "records_sha256": "570f24fe7110855ac974d12903e45579c496d853625a45e916e20f2381e28852",
- "summary": {
- "E0": {
- "overall": {
- "n": 186,
- "accuracy": 0.532258064516129,
- "macro_f1": 0.4408400882085093,
- "nll": 1.3837465414314734,
- "brier": 0.7300712607122152,
- "ece": 0.36648789161804096,
- "risk_coverage": [
- {
- "target_coverage": 0.25,
- "coverage": 0.25268817204301075,
- "n": 47,
- "risk": 0.1702127659574468
- },
- {
- "target_coverage": 0.5,
- "coverage": 0.5,
- "n": 93,
- "risk": 0.27956989247311825
- },
- {
- "target_coverage": 0.75,
- "coverage": 0.7526881720430108,
- "n": 140,
- "risk": 0.4
- },
- {
- "target_coverage": 0.9,
- "coverage": 0.9032258064516129,
- "n": 168,
- "risk": 0.43452380952380953
- },
- {
- "target_coverage": 1.0,
- "coverage": 1.0,
- "n": 186,
- "risk": 0.46774193548387094
- }
- ]
- },
- "by_type": {
- "choice": {
- "n": 93,
- "accuracy": 0.6344086021505376,
- "macro_f1": 0.5809282967177705,
- "nll": 1.2960431695747896,
- "brier": 0.5208253347319968,
- "ece": 0.2534260594442905,
- "risk_coverage": [
- {
- "target_coverage": 0.25,
- "coverage": 0.25806451612903225,
- "n": 24,
- "risk": 0.2916666666666667
- },
- {
- "target_coverage": 0.5,
- "coverage": 0.5053763440860215,
- "n": 47,
- "risk": 0.19148936170212766
- },
- {
- "target_coverage": 0.75,
- "coverage": 0.7526881720430108,
- "n": 70,
- "risk": 0.2571428571428571
- },
- {
- "target_coverage": 0.9,
- "coverage": 0.9032258064516129,
- "n": 84,
- "risk": 0.3333333333333333
- },
- {
- "target_coverage": 1.0,
- "coverage": 1.0,
- "n": 93,
- "risk": 0.3655913978494624
- }
- ]
- },
- "noul": {
- "n": 93,
- "accuracy": 0.43010752688172044,
- "macro_f1": 0.3007518796992481,
- "nll": 1.4714499132881573,
- "brier": 0.9393171866924332,
- "ece": 0.49461061636621556,
- "risk_coverage": [
- {
- "target_coverage": 0.25,
- "coverage": 0.25806451612903225,
- "n": 24,
- "risk": 0.20833333333333334
- },
- {
- "target_coverage": 0.5,
- "coverage": 0.5053763440860215,
- "n": 47,
- "risk": 0.3829787234042553
- },
- {
- "target_coverage": 0.75,
- "coverage": 0.7526881720430108,
- "n": 70,
- "risk": 0.4857142857142857
- },
- {
- "target_coverage": 0.9,
- "coverage": 0.9032258064516129,
- "n": 84,
- "risk": 0.5357142857142857
- },
- {
- "target_coverage": 1.0,
- "coverage": 1.0,
- "n": 93,
- "risk": 0.5698924731182796
- }
- ]
- }
- },
- "definitions": {
- "macro_f1": "equal mean of task-local macro F1; all task options, zero for undefined F1",
- "brier": "mean sum of squared errors over all valid classes (including both noul classes)",
- "ece": "15 equal bins [i/15,(i+1)/15), last bin includes 1; confidence=max(p)",
- "risk_ties": "descending max(p), then lexical sample ID; ceil target count; diagnostic only"
- }
- },
- "E1": {
- "overall": {
- "n": 186,
- "accuracy": 0.532258064516129,
- "macro_f1": 0.4408400882085093,
- "nll": 0.7850334112663782,
- "brier": 0.5068983072913132,
- "ece": 0.1975294208240641,
- "risk_coverage": [
- {
- "target_coverage": 0.25,
- "coverage": 0.25268817204301075,
- "n": 47,
- "risk": 0.19148936170212766
- },
- {
- "target_coverage": 0.5,
- "coverage": 0.5,
- "n": 93,
- "risk": 0.2903225806451613
- },
- {
- "target_coverage": 0.75,
- "coverage": 0.7526881720430108,
- "n": 140,
- "risk": 0.40714285714285714
- },
- {
- "target_coverage": 0.9,
- "coverage": 0.9032258064516129,
- "n": 168,
- "risk": 0.43452380952380953
- },
- {
- "target_coverage": 1.0,
- "coverage": 1.0,
- "n": 186,
- "risk": 0.46774193548387094
- }
- ]
- },
- "by_type": {
- "choice": {
- "n": 93,
- "accuracy": 0.6344086021505376,
- "macro_f1": 0.5809282967177705,
- "nll": 0.8372707851828709,
- "brier": 0.4740320871109113,
- "ece": 0.16990508653316772,
- "risk_coverage": [
- {
- "target_coverage": 0.25,
- "coverage": 0.25806451612903225,
- "n": 24,
- "risk": 0.2916666666666667
- },
- {
- "target_coverage": 0.5,
- "coverage": 0.5053763440860215,
- "n": 47,
- "risk": 0.19148936170212766
- },
- {
- "target_coverage": 0.75,
- "coverage": 0.7526881720430108,
- "n": 70,
- "risk": 0.24285714285714285
- },
- {
- "target_coverage": 0.9,
- "coverage": 0.9032258064516129,
- "n": 84,
- "risk": 0.32142857142857145
- },
- {
- "target_coverage": 1.0,
- "coverage": 1.0,
- "n": 93,
- "risk": 0.3655913978494624
- }
- ]
- },
- "noul": {
- "n": 93,
- "accuracy": 0.43010752688172044,
- "macro_f1": 0.3007518796992481,
- "nll": 0.7327960373498855,
- "brier": 0.5397645274717153,
- "ece": 0.2896348457994376,
- "risk_coverage": [
- {
- "target_coverage": 0.25,
- "coverage": 0.25806451612903225,
- "n": 24,
- "risk": 0.20833333333333334
- },
- {
- "target_coverage": 0.5,
- "coverage": 0.5053763440860215,
- "n": 47,
- "risk": 0.3829787234042553
- },
- {
- "target_coverage": 0.75,
- "coverage": 0.7526881720430108,
- "n": 70,
- "risk": 0.4857142857142857
- },
- {
- "target_coverage": 0.9,
- "coverage": 0.9032258064516129,
- "n": 84,
- "risk": 0.5357142857142857
- },
- {
- "target_coverage": 1.0,
- "coverage": 1.0,
- "n": 93,
- "risk": 0.5698924731182796
- }
- ]
- }
- },
- "definitions": {
- "macro_f1": "equal mean of task-local macro F1; all task options, zero for undefined F1",
- "brier": "mean sum of squared errors over all valid classes (including both noul classes)",
- "ece": "15 equal bins [i/15,(i+1)/15), last bin includes 1; confidence=max(p)",
- "risk_ties": "descending max(p), then lexical sample ID; ceil target count; diagnostic only"
- }
- },
- "fitted": {
- "choice": {
- "status": "SUCCESS",
- "n": 64,
- "temperature": 3.1355035588563744,
- "boundary_hit": false,
- "nll_t1": 1.21606373270805,
- "nll_fitted": 0.771914452059509
- },
- "noul": {
- "status": "SUCCESS",
- "n": 64,
- "temperature": 5.0,
- "boundary_hit": true,
- "nll_t1": 1.5203078336357765,
- "nll_fitted": 0.7411119749553794
- }
- },
- "counts": {
- "calibration_decisions": 128,
- "calibration_groups": 62,
- "test_decisions": 186,
- "test_groups": 89,
- "test_source_utterances": 93
- },
- "paired": {
- "E1-E0": {
- "status": "SUCCESS",
- "n_groups": 89,
- "repeats": 1000,
- "seed": 20260924,
- "after_minus_before": {
- "accuracy": {
- "delta": 0.0,
- "ci95": [
- 0.0,
- 0.0
- ]
- },
- "macro_f1": {
- "delta": 0.0,
- "ci95": [
- 0.0,
- 0.0
- ]
- },
- "nll": {
- "delta": -0.5987131301650952,
- "ci95": [
- -0.8657735567616283,
- -0.3690991941162156
- ]
- },
- "brier": {
- "delta": -0.22317295342090193,
- "ci95": [
- -0.2988262606403422,
- -0.15288959035634678
- ]
- },
- "ece": {
- "delta": -0.16895847079397686,
- "ci95": [
- -0.22499543687347137,
- -0.09854721867871374
- ]
- }
- }
- }
- }
- },
- "records": [
- {"id":"massive-1003-choice","group_id":"massive-alarm-9c31991bfab87db64771","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"e6fa805478662b9dbcc6c13d66010516a205998b1e55b8fa88718b9a5ca94f46","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.212890625,-3.9296875,1.6923828125],"temperature":1.0,"type":"choice","source_id":"1003","source_row_hash":"c139b76204b21a35e4e183738cdcebd381ba62ffd54b666f3265e945432e67a4","input_tokens":52,"truncated":false},
- {"id":"massive-1003-noul","group_id":"massive-alarm-9c31991bfab87db64771","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"6f5566a71b373d478175fad317696441493eb97aed759b090d2a2c48545ee861","option_order":["false","true"],"gold":"false","raw_logits":[-1.9501953125,0.350830078125],"temperature":1.0,"type":"noul","source_id":"1003","source_row_hash":"c139b76204b21a35e4e183738cdcebd381ba62ffd54b666f3265e945432e67a4","input_tokens":47,"truncated":false},
- {"id":"massive-1037-choice","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"bb5e6d977cbf55ef5e40bb73ea9baf10960578c334f3d536be4306644facb148","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.31640625,-1.30078125,-2.2265625],"temperature":1.0,"type":"choice","source_id":"1037","source_row_hash":"cc358f7ff6234a620435ded52f9c6458c9725655acb4e04af4631fd08e5a5204","input_tokens":54,"truncated":false},
- {"id":"massive-1037-noul","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"3e21af49ed7e7d8892eab2ba9ab022e2e3588c8b688c2341ba43c4b28c8864f9","option_order":["false","true"],"gold":"false","raw_logits":[-2.087890625,0.63623046875],"temperature":1.0,"type":"noul","source_id":"1037","source_row_hash":"cc358f7ff6234a620435ded52f9c6458c9725655acb4e04af4631fd08e5a5204","input_tokens":49,"truncated":false},
- {"id":"massive-1263-choice","group_id":"massive-alarm-63e247661f9a8287cd19","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"08d4385e1c618bb738acb9a5a596d85083a795736d25f99d24a4109c26bce6e1","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.66015625,1.666015625,-2.01171875],"temperature":1.0,"type":"choice","source_id":"1263","source_row_hash":"4c56c9a24f86097f01329c4a62df6b539d96202d275ca101ee455c86e1997ca1","input_tokens":51,"truncated":false},
- {"id":"massive-1263-noul","group_id":"massive-alarm-63e247661f9a8287cd19","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"6754f8f7870ad226a15315003e5c8c2d81894182d894561530bf9df030f82f52","option_order":["false","true"],"gold":"false","raw_logits":[-1.7666015625,0.31396484375],"temperature":1.0,"type":"noul","source_id":"1263","source_row_hash":"4c56c9a24f86097f01329c4a62df6b539d96202d275ca101ee455c86e1997ca1","input_tokens":46,"truncated":false},
- {"id":"massive-1436-choice","group_id":"massive-alarm-c583aafff75bc3841fef","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3424e9c88433013f65faaf5bd116ec0d597ba22d08776672379224d576f843f6","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[5.23828125,-2.333984375,-2.853515625],"temperature":1.0,"type":"choice","source_id":"1436","source_row_hash":"94eb7e687527f8f71b3feca63c7dbbd70e8a983b83278b2da3c6c151e6c788d7","input_tokens":52,"truncated":false},
- {"id":"massive-1436-noul","group_id":"massive-alarm-c583aafff75bc3841fef","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"3a9863e2c88dffd907da9f6044347c9f61de6505d1a9dce9417f406054ad0b2e","option_order":["false","true"],"gold":"false","raw_logits":[-1.951171875,1.1201171875],"temperature":1.0,"type":"noul","source_id":"1436","source_row_hash":"94eb7e687527f8f71b3feca63c7dbbd70e8a983b83278b2da3c6c151e6c788d7","input_tokens":47,"truncated":false},
- {"id":"massive-1604-choice","group_id":"massive-alarm-7aa04ea17df3cdddb3fd","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"c544d289a669972f9dd9ffc0abfbabbd6242e07c0777812794fa4cd915f376dd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.869140625,0.30810546875,-0.5908203125],"temperature":1.0,"type":"choice","source_id":"1604","source_row_hash":"11a0e6002c5740f2e54e2854cfc01f4b30c2d0d7e65eccfa97155aa39c654f4b","input_tokens":52,"truncated":false},
- {"id":"massive-1604-noul","group_id":"massive-alarm-7aa04ea17df3cdddb3fd","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"07d97cb090365c59b005f4f668e2e3fff1c297a02363218def7e349191debaf7","option_order":["false","true"],"gold":"false","raw_logits":[-1.8564453125,0.95654296875],"temperature":1.0,"type":"noul","source_id":"1604","source_row_hash":"11a0e6002c5740f2e54e2854cfc01f4b30c2d0d7e65eccfa97155aa39c654f4b","input_tokens":47,"truncated":false},
- {"id":"massive-1657-choice","group_id":"massive-alarm-8bd33166b8b9b2f13b07","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d7486a4cb8953ccc3bb3ddd9179299768df6d34c1e2e880e766e75efedfd7be5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.91015625,-1.087890625,-3.57421875],"temperature":1.0,"type":"choice","source_id":"1657","source_row_hash":"ef2da490978458ee2ef0ee0a0d3b0133c56588c1e26696bccf3e2aa4c8259058","input_tokens":53,"truncated":false},
- {"id":"massive-1657-noul","group_id":"massive-alarm-8bd33166b8b9b2f13b07","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"a560e880161b61f771ac3a6338ff7e841df824be6b274d90320b5785ce6d72b3","option_order":["false","true"],"gold":"false","raw_logits":[-1.30859375,1.0390625],"temperature":1.0,"type":"noul","source_id":"1657","source_row_hash":"ef2da490978458ee2ef0ee0a0d3b0133c56588c1e26696bccf3e2aa4c8259058","input_tokens":48,"truncated":false},
- {"id":"massive-1733-choice","group_id":"massive-alarm-57e77352e57177a49115","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a7c5c35f649bc0b1446ed1c39de844fbbe2bfedb8762971db11bff47397c568d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.76171875,-1.87890625,-0.54833984375],"temperature":1.0,"type":"choice","source_id":"1733","source_row_hash":"a1f5a6d541d3e0f93903c8eb255a58ef9734fe6e55a7407e4015d4ebe61f872a","input_tokens":50,"truncated":false},
- {"id":"massive-1733-noul","group_id":"massive-alarm-57e77352e57177a49115","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"444a0c395aed016e36c3ee3c22e2f4668c4d0012de718db84987c75b48deba4a","option_order":["false","true"],"gold":"false","raw_logits":[-0.98388671875,0.47119140625],"temperature":1.0,"type":"noul","source_id":"1733","source_row_hash":"a1f5a6d541d3e0f93903c8eb255a58ef9734fe6e55a7407e4015d4ebe61f872a","input_tokens":45,"truncated":false},
- {"id":"massive-233-choice","group_id":"massive-alarm-a76f37cef918fa476dce","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1edd77d48a208990431178b3f9ff0e81bba90fc424f207edfe4901049150c5c6","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.35546875,-3.86328125,-3.658203125],"temperature":1.0,"type":"choice","source_id":"233","source_row_hash":"d4a1830096b5b362cb2c8979d9bee17ce35e11662498301fcf7d424aba32f005","input_tokens":53,"truncated":false},
- {"id":"massive-233-noul","group_id":"massive-alarm-a76f37cef918fa476dce","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"bbac79120c3ff2705b203e60febc97ce48a4593d3ed949b4e48d407694ac163a","option_order":["false","true"],"gold":"true","raw_logits":[-2.30078125,1.5107421875],"temperature":1.0,"type":"noul","source_id":"233","source_row_hash":"d4a1830096b5b362cb2c8979d9bee17ce35e11662498301fcf7d424aba32f005","input_tokens":48,"truncated":false},
- {"id":"massive-2335-choice","group_id":"massive-alarm-2e28becb2c156092a49a","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1b380ecf765e790d39243a04e05a6a1a4e89dc1dcd3153c1351181a4f7f04dea","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.982421875,-4.1015625,-3.669921875],"temperature":1.0,"type":"choice","source_id":"2335","source_row_hash":"714a2094c1a038fba50aa33f971e26771226ce7e9e04ebb9d2c7efcec38d0310","input_tokens":56,"truncated":false},
- {"id":"massive-2335-noul","group_id":"massive-alarm-2e28becb2c156092a49a","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"2ab3d560c5bc5dafe905f62e711b23fa7294c813fbb83752e5d86531b0739ce4","option_order":["false","true"],"gold":"true","raw_logits":[-2.322265625,1.3525390625],"temperature":1.0,"type":"noul","source_id":"2335","source_row_hash":"714a2094c1a038fba50aa33f971e26771226ce7e9e04ebb9d2c7efcec38d0310","input_tokens":51,"truncated":false},
- {"id":"massive-238-choice","group_id":"massive-alarm-d0053508597b2d9fa47e","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d7642d01f72c4bf4414827571c80447d99b745af71a1fbb58350a576c27de378","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.7109375,-1.130859375,-1.6572265625],"temperature":1.0,"type":"choice","source_id":"238","source_row_hash":"b05a2c56edee6e6a1f6ad98e6e4db9b0ac80e73346236fb818961f2a8aad75eb","input_tokens":50,"truncated":false},
- {"id":"massive-238-noul","group_id":"massive-alarm-d0053508597b2d9fa47e","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"8901ba86edfac63a84581592485d396272294c4567e25620eecaf2ab26a0efeb","option_order":["false","true"],"gold":"false","raw_logits":[-1.6650390625,1.33203125],"temperature":1.0,"type":"noul","source_id":"238","source_row_hash":"b05a2c56edee6e6a1f6ad98e6e4db9b0ac80e73346236fb818961f2a8aad75eb","input_tokens":45,"truncated":false},
- {"id":"massive-2872-choice","group_id":"massive-alarm-28ce6af777b18d848cc4","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3266abead6625894002caf1ffbd1b89d1302d2cfda49d08401fd6188c04d450e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[5.796875,-4.578125,-4.578125],"temperature":1.0,"type":"choice","source_id":"2872","source_row_hash":"4702426e7fb1d8fa48b86077ff3fb58fa8999c52a880e860661019a7aff3a2ff","input_tokens":50,"truncated":false},
- {"id":"massive-2872-noul","group_id":"massive-alarm-28ce6af777b18d848cc4","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"a90d58891fd2025284bd565a1f8fa39b735090a012bd696054f66d8d81022dfb","option_order":["false","true"],"gold":"false","raw_logits":[-1.7099609375,1.833984375],"temperature":1.0,"type":"noul","source_id":"2872","source_row_hash":"4702426e7fb1d8fa48b86077ff3fb58fa8999c52a880e860661019a7aff3a2ff","input_tokens":45,"truncated":false},
- {"id":"massive-3150-choice","group_id":"massive-alarm-97d6527cc37656846b02","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"090b673551073f1ebfdf857a5bb68d8e62b08acfc82426883ea71e1709f3797e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.951171875,-2.08203125,-0.1639404296875],"temperature":1.0,"type":"choice","source_id":"3150","source_row_hash":"4d419b74038f055d4737cbf88b4723ee2a78b9ce112e8bf29e34e37d457b8022","input_tokens":53,"truncated":false},
- {"id":"massive-3150-noul","group_id":"massive-alarm-97d6527cc37656846b02","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c90f89f70a86da01043d5a5439c184efa63669f3f9a36adace7492118154db62","option_order":["false","true"],"gold":"false","raw_logits":[-2.302734375,1.0859375],"temperature":1.0,"type":"noul","source_id":"3150","source_row_hash":"4d419b74038f055d4737cbf88b4723ee2a78b9ce112e8bf29e34e37d457b8022","input_tokens":48,"truncated":false},
- {"id":"massive-3155-choice","group_id":"massive-alarm-1a05a193b7245896522f","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"bff7c384e9000bf1ca3d85672a1230b2e6b175b679b063b0be917adc3a0b6ae9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.4921875,-2.19140625,-2.087890625],"temperature":1.0,"type":"choice","source_id":"3155","source_row_hash":"ad03657b56afbfa502e69cf2bd0e29a98c83996c41222f4931b15bd39ae5c116","input_tokens":61,"truncated":false},
- {"id":"massive-3155-noul","group_id":"massive-alarm-1a05a193b7245896522f","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"761825a90cafa87fe6dc6a599248c59f4ac2296feb91e87a0b7ba56befc4453a","option_order":["false","true"],"gold":"true","raw_logits":[-2.064453125,1.5185546875],"temperature":1.0,"type":"noul","source_id":"3155","source_row_hash":"ad03657b56afbfa502e69cf2bd0e29a98c83996c41222f4931b15bd39ae5c116","input_tokens":56,"truncated":false},
- {"id":"massive-3180-choice","group_id":"massive-alarm-e4f7f3a520230de4d198","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1be584752d9da34d19148ed46c921d36f618e808f753b1b66487bab510798a0c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.625,-2.34375,-1.4287109375],"temperature":1.0,"type":"choice","source_id":"3180","source_row_hash":"48769eebc7610c81b81ad51c5266ea0166abeb999712ec3304969fda5c1136c6","input_tokens":55,"truncated":false},
- {"id":"massive-3180-noul","group_id":"massive-alarm-e4f7f3a520230de4d198","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"74856b6b33b3577a5dd45f8b2bf787504276b37df6b707eeffe9fae3efca17d7","option_order":["false","true"],"gold":"true","raw_logits":[-0.70849609375,0.75732421875],"temperature":1.0,"type":"noul","source_id":"3180","source_row_hash":"48769eebc7610c81b81ad51c5266ea0166abeb999712ec3304969fda5c1136c6","input_tokens":50,"truncated":false},
- {"id":"massive-3181-choice","group_id":"massive-alarm-513568a33433630f98d8","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"830d07369c5b2e5c39ee664f2fdcbf69533affad202f7d9a653a5494cf4b489e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.8046875,-4.19140625,-3.318359375],"temperature":1.0,"type":"choice","source_id":"3181","source_row_hash":"ac7f0b47957f7510837d1547341e088f0a4d09ed3a9701a61b95a7854d695e0e","input_tokens":55,"truncated":false},
- {"id":"massive-3181-noul","group_id":"massive-alarm-513568a33433630f98d8","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"3defc8360fec5e2ca53cce81386406fafd3a3537186fb44d3bd1e8dc5f2d474b","option_order":["false","true"],"gold":"true","raw_logits":[-1.85546875,1.07421875],"temperature":1.0,"type":"noul","source_id":"3181","source_row_hash":"ac7f0b47957f7510837d1547341e088f0a4d09ed3a9701a61b95a7854d695e0e","input_tokens":50,"truncated":false},
- {"id":"massive-3241-choice","group_id":"massive-alarm-b22b203130c5192a6423","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"f524b17f665cda002061c754e3fd41fd0e740c3018b24ba2d98ae591ceedb7c4","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.1953125,-3.646484375,-3.552734375],"temperature":1.0,"type":"choice","source_id":"3241","source_row_hash":"16f6792d8e8ccd3989e04e04b8148d3be61199c7e43d402bc6eeb08cbe3fd917","input_tokens":53,"truncated":false},
- {"id":"massive-3241-noul","group_id":"massive-alarm-b22b203130c5192a6423","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"fc6c5b8d2f44d273695f46d9a43719cadeb39c36eacba4524af59eb3c7f4767b","option_order":["false","true"],"gold":"true","raw_logits":[-2.30078125,1.1953125],"temperature":1.0,"type":"noul","source_id":"3241","source_row_hash":"16f6792d8e8ccd3989e04e04b8148d3be61199c7e43d402bc6eeb08cbe3fd917","input_tokens":48,"truncated":false},
- {"id":"massive-3271-choice","group_id":"massive-alarm-cc4399c7e1fe9bedaecc","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"b2fb2fd52dd7d6293ffb2ce75dfa9d145afb3da928c3f35cb90089a75a41290c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.455078125,-4.16015625,1.923828125],"temperature":1.0,"type":"choice","source_id":"3271","source_row_hash":"3d4b1654d3459ec3a62cc33209806f071eb0d809cc71d0fb51be19014a8b13d9","input_tokens":51,"truncated":false},
- {"id":"massive-3271-noul","group_id":"massive-alarm-cc4399c7e1fe9bedaecc","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"304b596915cb7afe22d5674667661a63184a832a5d2bd367bc730d199d2cd0ba","option_order":["false","true"],"gold":"false","raw_logits":[-2.5078125,-0.36181640625],"temperature":1.0,"type":"noul","source_id":"3271","source_row_hash":"3d4b1654d3459ec3a62cc33209806f071eb0d809cc71d0fb51be19014a8b13d9","input_tokens":46,"truncated":false},
- {"id":"massive-3313-choice","group_id":"massive-alarm-9624abc57fb5f9491c3c","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a63a16cad37a46fa428b72311b35fdecf3557e049597fe79ffbc0e23619e92ac","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.78125,-4.37890625,-4.58203125],"temperature":1.0,"type":"choice","source_id":"3313","source_row_hash":"25d5d82e30b677564c58a29305e0946447be07c0985598b902c051ef1fb1ea80","input_tokens":60,"truncated":false},
- {"id":"massive-3313-noul","group_id":"massive-alarm-9624abc57fb5f9491c3c","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"62a66e214884f42fa03ddc5d6e062bb3657c1418478a8e7b625c87853a0ee453","option_order":["false","true"],"gold":"true","raw_logits":[-2.55078125,1.4404296875],"temperature":1.0,"type":"noul","source_id":"3313","source_row_hash":"25d5d82e30b677564c58a29305e0946447be07c0985598b902c051ef1fb1ea80","input_tokens":55,"truncated":false},
- {"id":"massive-3344-choice","group_id":"massive-alarm-2124aad0708d0ec9337e","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"9748505b1055a8e4b9b580e4f065f1fb025a351d0316f2664fe9fb467aa20b01","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.60205078125,-1.8876953125,-0.740234375],"temperature":1.0,"type":"choice","source_id":"3344","source_row_hash":"503f2378950bb6b7542b8fd34d1781fa9a81f3da8e533bc3e5474ee2e90182a6","input_tokens":51,"truncated":false},
- {"id":"massive-3344-noul","group_id":"massive-alarm-2124aad0708d0ec9337e","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"40547d94b9391018d836f7c33cb83a9888d4a349795a9961a9c4d63bd0f9b699","option_order":["false","true"],"gold":"false","raw_logits":[-1.6083984375,0.9453125],"temperature":1.0,"type":"noul","source_id":"3344","source_row_hash":"503f2378950bb6b7542b8fd34d1781fa9a81f3da8e533bc3e5474ee2e90182a6","input_tokens":46,"truncated":false},
- {"id":"massive-3461-choice","group_id":"massive-alarm-ce0cb4819c1987861556","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"73431ea9804730e590ddc48be8d2c8035bb0dbeb45e6c1cfa2b74cddf9289334","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.64453125,-3.767578125,-3.158203125],"temperature":1.0,"type":"choice","source_id":"3461","source_row_hash":"0e09b286b7b15ee40d39a9dc76bc5d58004ddbae3898a25e3673897d633f6fd7","input_tokens":54,"truncated":false},
- {"id":"massive-3461-noul","group_id":"massive-alarm-ce0cb4819c1987861556","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"9c083d9ba471c24cd29e87a06ee702b6f461f167c9e5f0040a36737925acfc5b","option_order":["false","true"],"gold":"true","raw_logits":[-2.619140625,1.3740234375],"temperature":1.0,"type":"noul","source_id":"3461","source_row_hash":"0e09b286b7b15ee40d39a9dc76bc5d58004ddbae3898a25e3673897d633f6fd7","input_tokens":49,"truncated":false},
- {"id":"massive-3537-choice","group_id":"massive-alarm-1b97665270a474c9189b","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"58b2a286faf22ace00cb5b5322c9207f9304330fc3153fe79b6e308492cccfb8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.4990234375,-3.5234375,2.623046875],"temperature":1.0,"type":"choice","source_id":"3537","source_row_hash":"3bb0e1d5d6780212fc57f7c5bdc635e4f3b8af60e8f05a8864c0e0801b159e28","input_tokens":50,"truncated":false},
- {"id":"massive-3537-noul","group_id":"massive-alarm-1b97665270a474c9189b","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"27a1f6fa962a3cd311f0a1ded2227f879f7d3f63afd038278229abc437872012","option_order":["false","true"],"gold":"false","raw_logits":[-2.248046875,0.375244140625],"temperature":1.0,"type":"noul","source_id":"3537","source_row_hash":"3bb0e1d5d6780212fc57f7c5bdc635e4f3b8af60e8f05a8864c0e0801b159e28","input_tokens":45,"truncated":false},
- {"id":"massive-3696-choice","group_id":"massive-alarm-66bb02dcd4dfec9a31da","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"5c799de6dc9b83c02640969e26d2a2b4abd09017c1da008663ae386f27a420dc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.1328125,-3.171875,0.63134765625],"temperature":1.0,"type":"choice","source_id":"3696","source_row_hash":"50b44cd61dd0112bec73547e36d51b18ec64f7f88e01642a88fe65e66aaf8019","input_tokens":57,"truncated":false},
- {"id":"massive-3696-noul","group_id":"massive-alarm-66bb02dcd4dfec9a31da","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"83b6f74e7c01aed19a8a603c7b23d4c58da8a0282bc12b481546ed07d0dae1ca","option_order":["false","true"],"gold":"true","raw_logits":[-2.15625,1.0517578125],"temperature":1.0,"type":"noul","source_id":"3696","source_row_hash":"50b44cd61dd0112bec73547e36d51b18ec64f7f88e01642a88fe65e66aaf8019","input_tokens":52,"truncated":false},
- {"id":"massive-3760-choice","group_id":"massive-alarm-7f77ea61d28ffa0c4760","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a226efece8e4ca552b5c604d2c0e77f1e7ad8ef574882ccdf2abc1353914a530","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.5634765625,-1.9404296875,1.23046875],"temperature":1.0,"type":"choice","source_id":"3760","source_row_hash":"c58ac469fe7402eeee9373c32aed43cde55b7f6a576c78cf3e6cdbdcad81b62f","input_tokens":54,"truncated":false},
- {"id":"massive-3760-noul","group_id":"massive-alarm-7f77ea61d28ffa0c4760","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"a17c30741ef64b9a54cd5fa80638f0d977ace713acc174493612ac98af69da35","option_order":["false","true"],"gold":"true","raw_logits":[-2.052734375,1.4296875],"temperature":1.0,"type":"noul","source_id":"3760","source_row_hash":"c58ac469fe7402eeee9373c32aed43cde55b7f6a576c78cf3e6cdbdcad81b62f","input_tokens":49,"truncated":false},
- {"id":"massive-3948-choice","group_id":"massive-alarm-74196bc28bd235b5ebd4","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2244c648b7fdc8555facdb170e46c5ab59d024a693910e18c7b28b94ac0edb53","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.94140625,-4.47265625,-4.48828125],"temperature":1.0,"type":"choice","source_id":"3948","source_row_hash":"efdef6a7ca0afe5619c4c98377ee543a2c63c2f4a10c19d98cf9308324489779","input_tokens":52,"truncated":false},
- {"id":"massive-3948-noul","group_id":"massive-alarm-74196bc28bd235b5ebd4","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"72dd631c4fbc49146d876163d1ab74c4cb6eeb7076498c11a5a138878ca345bd","option_order":["false","true"],"gold":"true","raw_logits":[-2.017578125,1.310546875],"temperature":1.0,"type":"noul","source_id":"3948","source_row_hash":"efdef6a7ca0afe5619c4c98377ee543a2c63c2f4a10c19d98cf9308324489779","input_tokens":47,"truncated":false},
- {"id":"massive-3959-choice","group_id":"massive-alarm-1e2d1e22e2a448d0d4be","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3d05bb7983f196a0d3e77f5d1eda7f2de7d1eb9f65c25e6a5aa4985decca2ffc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.8291015625,-4.109375,-0.41552734375],"temperature":1.0,"type":"choice","source_id":"3959","source_row_hash":"7539f118e45280ccaa989666ca00e8b1765628ce7a944ffa526b60c3c58b91ab","input_tokens":52,"truncated":false},
- {"id":"massive-3959-noul","group_id":"massive-alarm-1e2d1e22e2a448d0d4be","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"0ca03992c0d8713a6590531ef8364a455a9ee6f7d7116d6a6262e593a496f42e","option_order":["false","true"],"gold":"true","raw_logits":[-1.388671875,1.0908203125],"temperature":1.0,"type":"noul","source_id":"3959","source_row_hash":"7539f118e45280ccaa989666ca00e8b1765628ce7a944ffa526b60c3c58b91ab","input_tokens":47,"truncated":false},
- {"id":"massive-3975-choice","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"e9c4661dc32b87b3f0762cd7a7254309cd7852b13a77514c93a4e4003111fe2f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.0711669921875,-2.5546875,-0.60888671875],"temperature":1.0,"type":"choice","source_id":"3975","source_row_hash":"c8c3d4b20f0116b687d7d684dba6fe8070f18bad470dc44ddbb7000c8c4ebaee","input_tokens":49,"truncated":false},
- {"id":"massive-3975-noul","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"df1f03cd870145a75b4e1403f5ec6464d4217ddf98f0e60c79cebc6dcf18f34c","option_order":["false","true"],"gold":"true","raw_logits":[-2.453125,1.7314453125],"temperature":1.0,"type":"noul","source_id":"3975","source_row_hash":"c8c3d4b20f0116b687d7d684dba6fe8070f18bad470dc44ddbb7000c8c4ebaee","input_tokens":44,"truncated":false},
- {"id":"massive-3985-choice","group_id":"massive-alarm-0eab4c88f4262e142df9","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a0ae587eb80ef7ddb6b9eb42df39de2690cd1df443b346b429870aa844881111","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.611328125,-4.12890625,-4.41015625],"temperature":1.0,"type":"choice","source_id":"3985","source_row_hash":"405084aa291615ce23004c5c83303d8418be167a30ba30ed939eb50f7b2e6b87","input_tokens":53,"truncated":false},
- {"id":"massive-3985-noul","group_id":"massive-alarm-0eab4c88f4262e142df9","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"ea1cfc0c358999cca571b97cb637ab166d0ca3225c812cf9c03f24cc95d8d626","option_order":["false","true"],"gold":"true","raw_logits":[-2.0390625,1.4208984375],"temperature":1.0,"type":"noul","source_id":"3985","source_row_hash":"405084aa291615ce23004c5c83303d8418be167a30ba30ed939eb50f7b2e6b87","input_tokens":48,"truncated":false},
- {"id":"massive-4016-choice","group_id":"massive-alarm-4be456711aa1993110b2","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"505ee765de12c09f3f828b85ca69cd4953e2191d4d8c4ee6a25189a83a760043","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.306640625,-4.37890625,4.83203125],"temperature":1.0,"type":"choice","source_id":"4016","source_row_hash":"eb5665c8fa5ae7c7dc5804205acfba089cd6c2936b540d0c86da0c3ee73ec92c","input_tokens":53,"truncated":false},
- {"id":"massive-4016-noul","group_id":"massive-alarm-4be456711aa1993110b2","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"d140649f39079e246dadf8169a7a97f6b2489328b4734f0c178f3de16660dba5","option_order":["false","true"],"gold":"false","raw_logits":[-1.740234375,0.7607421875],"temperature":1.0,"type":"noul","source_id":"4016","source_row_hash":"eb5665c8fa5ae7c7dc5804205acfba089cd6c2936b540d0c86da0c3ee73ec92c","input_tokens":48,"truncated":false},
- {"id":"massive-4108-choice","group_id":"massive-alarm-ba768fc0bd4d5e4b7d89","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"98479093baf492b301adaa3cc8ba52c0481115d2eca82b13c0b146b35bb9a417","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.076171875,1.240234375,-1.5771484375],"temperature":1.0,"type":"choice","source_id":"4108","source_row_hash":"442cf4064902074bab424f8a13b5da971239324a1ddb14ac9c1461459605890d","input_tokens":51,"truncated":false},
- {"id":"massive-4108-noul","group_id":"massive-alarm-ba768fc0bd4d5e4b7d89","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"85af1497981b267f36416626603b0919ffe18af189c37a8b46e7da2c551c5255","option_order":["false","true"],"gold":"false","raw_logits":[-1.4697265625,1.169921875],"temperature":1.0,"type":"noul","source_id":"4108","source_row_hash":"442cf4064902074bab424f8a13b5da971239324a1ddb14ac9c1461459605890d","input_tokens":46,"truncated":false},
- {"id":"massive-4122-choice","group_id":"massive-alarm-7b84a2782f58c91b2bae","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"acc9bcb43f252751dd15370f7a1f9f35ad22a6b1c088fb3e9b36ede8d965b270","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.61328125,-0.46728515625,-1.58203125],"temperature":1.0,"type":"choice","source_id":"4122","source_row_hash":"844fa5f915442e44ef3076cd0ce2150f380ac58e2b067d29b96f08c100a1d939","input_tokens":50,"truncated":false},
- {"id":"massive-4122-noul","group_id":"massive-alarm-7b84a2782f58c91b2bae","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"907aa9c5faa26b58d904e64f202bcce86d6c4f242c23186d9ee265ae4166ee9d","option_order":["false","true"],"gold":"false","raw_logits":[-1.4423828125,0.82763671875],"temperature":1.0,"type":"noul","source_id":"4122","source_row_hash":"844fa5f915442e44ef3076cd0ce2150f380ac58e2b067d29b96f08c100a1d939","input_tokens":45,"truncated":false},
- {"id":"massive-4144-choice","group_id":"massive-alarm-95d77d103fd5fab659a2","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"3f8abfe30982bff102e8b189574ee14bbf601371a4c033c0ebc000c980e801f7","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[1.3994140625,-2.697265625,-0.350830078125],"temperature":1.0,"type":"choice","source_id":"4144","source_row_hash":"0fd0739e6946a24cc24d8339f712773ba73cdcb18125f66f4b1af0f6c56cb409","input_tokens":54,"truncated":false},
- {"id":"massive-4144-noul","group_id":"massive-alarm-95d77d103fd5fab659a2","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"bb33b9b4e1da7c9029b695525c094b5019ad962957ae1e33e14fe4d50ffc5925","option_order":["false","true"],"gold":"false","raw_logits":[-1.7705078125,0.91015625],"temperature":1.0,"type":"noul","source_id":"4144","source_row_hash":"0fd0739e6946a24cc24d8339f712773ba73cdcb18125f66f4b1af0f6c56cb409","input_tokens":49,"truncated":false},
- {"id":"massive-4200-choice","group_id":"massive-alarm-a56e43f47ec9a9386d07","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"6774a6320d42c290b80ddb390be36ac859ee6f15374b0bd6ff3fefd461823904","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.27734375,-4.15234375,1.6220703125],"temperature":1.0,"type":"choice","source_id":"4200","source_row_hash":"c355f735101213734ce53521f480d2713ee60fde1860e82542bfa9d357596c71","input_tokens":52,"truncated":false},
- {"id":"massive-4200-noul","group_id":"massive-alarm-a56e43f47ec9a9386d07","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c2d79b9b8cf8fe2e19bf991be938ccd39a5638a21c2c00469f612d6d93b32d30","option_order":["false","true"],"gold":"false","raw_logits":[-3.84375,0.4052734375],"temperature":1.0,"type":"noul","source_id":"4200","source_row_hash":"c355f735101213734ce53521f480d2713ee60fde1860e82542bfa9d357596c71","input_tokens":47,"truncated":false},
- {"id":"massive-4509-choice","group_id":"massive-alarm-d4f2a8b9df951f72e77a","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"31ad31c567248f3781353d8e1cc63b681d0d3e9c5bdda2d3fab14afc0a04b41a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.2138671875,-3.05078125,1.6796875],"temperature":1.0,"type":"choice","source_id":"4509","source_row_hash":"bf224c141975e01266dcf5ab02b2f22e5e961fe3acb83f2ff99c3f8f960284b2","input_tokens":51,"truncated":false},
- {"id":"massive-4509-noul","group_id":"massive-alarm-d4f2a8b9df951f72e77a","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"dbe036ea36070f090a10050a3982caa2a9e3a00a5640b7586cef8afc7276731c","option_order":["false","true"],"gold":"false","raw_logits":[-2.4609375,1.2763671875],"temperature":1.0,"type":"noul","source_id":"4509","source_row_hash":"bf224c141975e01266dcf5ab02b2f22e5e961fe3acb83f2ff99c3f8f960284b2","input_tokens":46,"truncated":false},
- {"id":"massive-4530-choice","group_id":"massive-alarm-c0492ef78107c7e23f86","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"4bcb70abe01980ab58b9e301d26da030dae938a5e9ae12f907c932fb5ffa09a8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.2783203125,-0.74560546875,-0.94482421875],"temperature":1.0,"type":"choice","source_id":"4530","source_row_hash":"3dc1bd3fdc52e2c231408ad4a522b21e4518cff1813ef183884a10d582345cee","input_tokens":55,"truncated":false},
- {"id":"massive-4530-noul","group_id":"massive-alarm-c0492ef78107c7e23f86","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"15ca7806e7ca42d6ce4a143c2712cc799fa9ccd81611e366e35693d2927483cb","option_order":["false","true"],"gold":"false","raw_logits":[-1.53125,0.697265625],"temperature":1.0,"type":"noul","source_id":"4530","source_row_hash":"3dc1bd3fdc52e2c231408ad4a522b21e4518cff1813ef183884a10d582345cee","input_tokens":50,"truncated":false},
- {"id":"massive-4585-choice","group_id":"massive-alarm-57ee6f9659a3c6835215","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"8e4c5a399d92bc0fee79a0b7c274ef67eb3dadc5c492cccc34f148aa6184c53d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.625,-3.06640625,-2.103515625],"temperature":1.0,"type":"choice","source_id":"4585","source_row_hash":"29cb3192dcfaf8dd3c91f8560e6956e6d62314b02d2c3a19b339b44e7f94808e","input_tokens":61,"truncated":false},
- {"id":"massive-4585-noul","group_id":"massive-alarm-57ee6f9659a3c6835215","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"5d41960f1abd888d11d48ec0e983393f7efd4ffdad5f9941815404c8ba47ba75","option_order":["false","true"],"gold":"false","raw_logits":[-1.8154296875,0.28125],"temperature":1.0,"type":"noul","source_id":"4585","source_row_hash":"29cb3192dcfaf8dd3c91f8560e6956e6d62314b02d2c3a19b339b44e7f94808e","input_tokens":56,"truncated":false},
- {"id":"massive-4593-choice","group_id":"massive-alarm-70c776b38b2c5f805dcd","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2b329bec09cbc6dbd489dba7bf5ba03c208c7c760c3b11dcae5beab03e9de1d9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.73828125,1.751953125,-0.728515625],"temperature":1.0,"type":"choice","source_id":"4593","source_row_hash":"573a25fd5fa0ccdcbdf4ebee81ca2c4bbaab53ba3cb1563b483549e39c6e6a7a","input_tokens":53,"truncated":false},
- {"id":"massive-4593-noul","group_id":"massive-alarm-70c776b38b2c5f805dcd","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c9d1252c93abe919faf36506ceee2a4d572863bd74e43eb30c6e82ad87f3f29e","option_order":["false","true"],"gold":"false","raw_logits":[-1.8310546875,0.95751953125],"temperature":1.0,"type":"noul","source_id":"4593","source_row_hash":"573a25fd5fa0ccdcbdf4ebee81ca2c4bbaab53ba3cb1563b483549e39c6e6a7a","input_tokens":48,"truncated":false},
- {"id":"massive-4744-choice","group_id":"massive-alarm-4328c171395268c31624","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d0d46e21b89af922fa963c14d2cecf2e2b20b70099999c4ad40900017de92075","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.828125,3.25,0.44384765625],"temperature":1.0,"type":"choice","source_id":"4744","source_row_hash":"a4566885577d6aab551b8f32aa9432224700f661cf64bebd0176e38f03274c83","input_tokens":54,"truncated":false},
- {"id":"massive-4744-noul","group_id":"massive-alarm-4328c171395268c31624","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"afae574d52cbd08a229a7819423ec0cbd438a7d7268bf06023f2f5e18648c7b9","option_order":["false","true"],"gold":"false","raw_logits":[-1.40234375,0.57763671875],"temperature":1.0,"type":"noul","source_id":"4744","source_row_hash":"a4566885577d6aab551b8f32aa9432224700f661cf64bebd0176e38f03274c83","input_tokens":49,"truncated":false},
- {"id":"massive-4929-choice","group_id":"massive-alarm-74a1feb1e119db1fb400","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"bc3b693c29ce78f19435be8dd99e9451443befc07587185de8be06a0574902de","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.08660888671875,-2.994140625,1.0556640625],"temperature":1.0,"type":"choice","source_id":"4929","source_row_hash":"53f4b87a6a89da6de1f9302d72950cd98c6c9ba1bc9d35a3b33b0a2e2c4769a1","input_tokens":53,"truncated":false},
- {"id":"massive-4929-noul","group_id":"massive-alarm-74a1feb1e119db1fb400","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"59f48bdeda5d59c536e5c4364cba1a673b4afed9d9b79f6d8eab2d3e95c2e25f","option_order":["false","true"],"gold":"true","raw_logits":[-2.052734375,1.2275390625],"temperature":1.0,"type":"noul","source_id":"4929","source_row_hash":"53f4b87a6a89da6de1f9302d72950cd98c6c9ba1bc9d35a3b33b0a2e2c4769a1","input_tokens":48,"truncated":false},
- {"id":"massive-5114-choice","group_id":"massive-alarm-7fdbaa1c48c1e3afefe1","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"43567ef7d5ff87f4c3710c3424ab5f2cb19756e69c4d0e44ed470313ce41347b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.53076171875,-0.1943359375,-0.9033203125],"temperature":1.0,"type":"choice","source_id":"5114","source_row_hash":"0429eec68e26dfecd519ba81f0c92d85793aad1c8d202737bb439666a411b485","input_tokens":54,"truncated":false},
- {"id":"massive-5114-noul","group_id":"massive-alarm-7fdbaa1c48c1e3afefe1","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"7e3f35cef593675bcafc126f93ea622d79fe105f82c4e8c9ab80d884ac2201e5","option_order":["false","true"],"gold":"false","raw_logits":[-1.6279296875,1.0166015625],"temperature":1.0,"type":"noul","source_id":"5114","source_row_hash":"0429eec68e26dfecd519ba81f0c92d85793aad1c8d202737bb439666a411b485","input_tokens":49,"truncated":false},
- {"id":"massive-5172-choice","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"4f6d1a6703c39c88cebaff032394052408cba7c60cc58c80bd70f6bfd4112ba9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.008392333984375,-2.896484375,-0.76806640625],"temperature":1.0,"type":"choice","source_id":"5172","source_row_hash":"25e5a31c20bb4859195c396f45f721767e4d6c8bb6eceaaf2c63dcfc20adeb64","input_tokens":52,"truncated":false},
- {"id":"massive-5172-noul","group_id":"massive-alarm-54272adaa97f5498627d","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"69e8607aaeebf1a1350d73f7ea71224817d665a684ac4f3cdfe8ec464ace63b8","option_order":["false","true"],"gold":"true","raw_logits":[-2.07421875,1.3017578125],"temperature":1.0,"type":"noul","source_id":"5172","source_row_hash":"25e5a31c20bb4859195c396f45f721767e4d6c8bb6eceaaf2c63dcfc20adeb64","input_tokens":47,"truncated":false},
- {"id":"massive-5210-choice","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"c668b88dd8e85f1b4e78bb29f8f80248aa67aab044cfc54d576c102766632bfa","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.8671875,-3.4296875,-3.0703125],"temperature":1.0,"type":"choice","source_id":"5210","source_row_hash":"fa1ecb16b6a4385a45785dd19132a817f21fff52934ee5a626a97c780f4ab078","input_tokens":53,"truncated":false},
- {"id":"massive-5210-noul","group_id":"massive-alarm-0e2bebdbe61eb354625b","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"067fdd02a518f2dd87f537b24dabb02f1d8133c6bab8803ba201115d10f6d82d","option_order":["false","true"],"gold":"false","raw_logits":[-2.16015625,0.9296875],"temperature":1.0,"type":"noul","source_id":"5210","source_row_hash":"fa1ecb16b6a4385a45785dd19132a817f21fff52934ee5a626a97c780f4ab078","input_tokens":48,"truncated":false},
- {"id":"massive-5319-choice","group_id":"massive-alarm-322a8df31b5ea91267e0","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"720d5cc7c24bb6954e0c41cc16a40fb2b5124e4bb102f2aed64b7acc5df81817","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.56640625,-1.7998046875,-1.109375],"temperature":1.0,"type":"choice","source_id":"5319","source_row_hash":"2d07404bc4a0e9de04cd7b8305b4d7fb4db95c545a145cc3f8fb6ab7b5934529","input_tokens":51,"truncated":false},
- {"id":"massive-5319-noul","group_id":"massive-alarm-322a8df31b5ea91267e0","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"334afe5034e053d889e7e832d8fe0b2fd80448a8f579445964b27a15650a3933","option_order":["false","true"],"gold":"true","raw_logits":[-2.2421875,1.4140625],"temperature":1.0,"type":"noul","source_id":"5319","source_row_hash":"2d07404bc4a0e9de04cd7b8305b4d7fb4db95c545a145cc3f8fb6ab7b5934529","input_tokens":46,"truncated":false},
- {"id":"massive-535-choice","group_id":"massive-alarm-669fb7803ab072ceb0b5","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"10b4de6d460909b2303c9dab7a9e033a29d50076d3e4bb27e4a936a36786ce62","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.300048828125,-1.794921875,-0.435546875],"temperature":1.0,"type":"choice","source_id":"535","source_row_hash":"0e3fcc2ba0c2c5c5f577632298fa42814922bfd1141206f7474fd190dc965c7e","input_tokens":54,"truncated":false},
- {"id":"massive-535-noul","group_id":"massive-alarm-669fb7803ab072ceb0b5","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"99646e241c5c848f95da454dcbfd7698231492b08a38270a0c1651139e49f9d3","option_order":["false","true"],"gold":"true","raw_logits":[-0.666015625,0.058319091796875],"temperature":1.0,"type":"noul","source_id":"535","source_row_hash":"0e3fcc2ba0c2c5c5f577632298fa42814922bfd1141206f7474fd190dc965c7e","input_tokens":49,"truncated":false},
- {"id":"massive-5492-choice","group_id":"massive-alarm-5791810e78edbebec8b4","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"55954676941878d2afda77659529a13bd38cd2f009e9eee4eb759261ff47ac73","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.234375,-3.1796875,1.62109375],"temperature":1.0,"type":"choice","source_id":"5492","source_row_hash":"6b09ef0b32c674bcb40f7ad0ca704a196efe0ea8ce77472e2e3088c05bd29ed3","input_tokens":53,"truncated":false},
- {"id":"massive-5492-noul","group_id":"massive-alarm-5791810e78edbebec8b4","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"c1cbc2786d808f88ca9a1d631a134993d9a3357acbad234c9de4ba4cff33ae20","option_order":["false","true"],"gold":"false","raw_logits":[-1.857421875,-0.35498046875],"temperature":1.0,"type":"noul","source_id":"5492","source_row_hash":"6b09ef0b32c674bcb40f7ad0ca704a196efe0ea8ce77472e2e3088c05bd29ed3","input_tokens":48,"truncated":false},
- {"id":"massive-5673-choice","group_id":"massive-alarm-fa15e151bbd8376bf727","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"4ca5a1fef0f2d153d7bd6110a0fc74ceca4b41ad79287ee6961d656f74828ddc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.291015625,-3.931640625,2.373046875],"temperature":1.0,"type":"choice","source_id":"5673","source_row_hash":"42f49991de76cd94ab5ee5743f3e57ae2a14f2fb657869b455a94bfbbd05648f","input_tokens":52,"truncated":false},
- {"id":"massive-5673-noul","group_id":"massive-alarm-fa15e151bbd8376bf727","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"d4cb011692189f194a6894f23e0488fa6be688c3166137780caf9244766206dc","option_order":["false","true"],"gold":"false","raw_logits":[-2.53125,0.496826171875],"temperature":1.0,"type":"noul","source_id":"5673","source_row_hash":"42f49991de76cd94ab5ee5743f3e57ae2a14f2fb657869b455a94bfbbd05648f","input_tokens":47,"truncated":false},
- {"id":"massive-5675-choice","group_id":"massive-alarm-3568a7b610cf5fedeafe","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"c22de0ed7a39f40c2c13850693773a456275d5726a53d6e53dcd7355931f7c7f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.67578125,-4.43359375,1.3515625],"temperature":1.0,"type":"choice","source_id":"5675","source_row_hash":"d4455f4931eebb0ae5c429523260bb0da106c1fb1f59166ff83aa27192fd603d","input_tokens":53,"truncated":false},
- {"id":"massive-5675-noul","group_id":"massive-alarm-3568a7b610cf5fedeafe","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"2e82b6e8cdeb672a76ee85b20814106355c89d5c9c981051cff180c0105ff9d3","option_order":["false","true"],"gold":"false","raw_logits":[-2.6015625,-0.00815582275390625],"temperature":1.0,"type":"noul","source_id":"5675","source_row_hash":"d4455f4931eebb0ae5c429523260bb0da106c1fb1f59166ff83aa27192fd603d","input_tokens":48,"truncated":false},
- {"id":"massive-5736-choice","group_id":"massive-alarm-cad8645cf0d1993997ce","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d4a111329224fc89c0ce98f064feba839677a403249e06d1d2a8f49cb285191b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.5673828125,1.474609375,-0.52197265625],"temperature":1.0,"type":"choice","source_id":"5736","source_row_hash":"70f64d8c12527bf3143b24720e5e124b9c7c43de9bf0b78aca2fe33336e7f60d","input_tokens":54,"truncated":false},
- {"id":"massive-5736-noul","group_id":"massive-alarm-cad8645cf0d1993997ce","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"cd7f7f33d54024ae047cc45e45135de22fddd92c2eff1bdc2cf57668f3ef0b9d","option_order":["false","true"],"gold":"false","raw_logits":[-0.8037109375,0.301025390625],"temperature":1.0,"type":"noul","source_id":"5736","source_row_hash":"70f64d8c12527bf3143b24720e5e124b9c7c43de9bf0b78aca2fe33336e7f60d","input_tokens":49,"truncated":false},
- {"id":"massive-5891-choice","group_id":"massive-alarm-58fbb3df6930e3f46619","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"10f3ea8840e48ecbe378b6635f27574b818a01d405ff5f5435cda368088af7a3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.654296875,-1.7041015625,-3.41796875],"temperature":1.0,"type":"choice","source_id":"5891","source_row_hash":"13a21672cc8d63f41b4c4f3be3e9ee23d26cf4d483b68904d54849e76a9572ed","input_tokens":55,"truncated":false},
- {"id":"massive-5891-noul","group_id":"massive-alarm-58fbb3df6930e3f46619","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"aebc66b1557724f2c8050b8b96f8bd79808521ed13b64bc0d421e5ddee9e2ed7","option_order":["false","true"],"gold":"false","raw_logits":[-2.353515625,0.875],"temperature":1.0,"type":"noul","source_id":"5891","source_row_hash":"13a21672cc8d63f41b4c4f3be3e9ee23d26cf4d483b68904d54849e76a9572ed","input_tokens":50,"truncated":false},
- {"id":"massive-5906-choice","group_id":"massive-alarm-70d28c8cc29115db7bf6","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d429a04be92b35783c5220b12fb0d69dc0cc209556a6e30d4ccab1ab1b6957e6","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.2734375,-3.880859375,-4.16015625],"temperature":1.0,"type":"choice","source_id":"5906","source_row_hash":"c3492f95295cde551179df79a12fe2b92977fb44481de9b512aa59ed0900a40f","input_tokens":52,"truncated":false},
- {"id":"massive-5906-noul","group_id":"massive-alarm-70d28c8cc29115db7bf6","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"73d6faa4a8c642c84be73a9055997aee319398417b64d904ff2cc240cdb18649","option_order":["false","true"],"gold":"true","raw_logits":[-2.685546875,1.5634765625],"temperature":1.0,"type":"noul","source_id":"5906","source_row_hash":"c3492f95295cde551179df79a12fe2b92977fb44481de9b512aa59ed0900a40f","input_tokens":47,"truncated":false},
- {"id":"massive-5981-choice","group_id":"massive-alarm-3c4995900541ca97b2c9","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2226fcac61f5f77ed8bcf1443cada6e714f0219501e5b999600332ddff8f436b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.36328125,-1.1357421875,-2.537109375],"temperature":1.0,"type":"choice","source_id":"5981","source_row_hash":"601253fff66fad5594ffa03232ae14805096eeb3edddfabc89f23fdf86061586","input_tokens":53,"truncated":false},
- {"id":"massive-5981-noul","group_id":"massive-alarm-3c4995900541ca97b2c9","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"757dbc799dab322bc0fdd47f72321799a7049226e9dd8e045289b95fd6aa6c52","option_order":["false","true"],"gold":"false","raw_logits":[-2.056640625,1.291015625],"temperature":1.0,"type":"noul","source_id":"5981","source_row_hash":"601253fff66fad5594ffa03232ae14805096eeb3edddfabc89f23fdf86061586","input_tokens":48,"truncated":false},
- {"id":"massive-60-choice","group_id":"massive-alarm-94ab087f60890852e463","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"dfc4bfc4a569b92c621afb5cc8c39261cd7db70bd6d3ee9f1fa4f709bc840b7f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.384765625,-2.48046875,1.759765625],"temperature":1.0,"type":"choice","source_id":"60","source_row_hash":"0a0aea9a63bd5ab159c323cdd5274587f022744194c0d7b63de7f75fc36db068","input_tokens":51,"truncated":false},
- {"id":"massive-60-noul","group_id":"massive-alarm-94ab087f60890852e463","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"8d2c25f4e5a2219cdb422d36d08c2f2c51bde324bd447a0785ba5716405cc86a","option_order":["false","true"],"gold":"false","raw_logits":[-3.03515625,1.7685546875],"temperature":1.0,"type":"noul","source_id":"60","source_row_hash":"0a0aea9a63bd5ab159c323cdd5274587f022744194c0d7b63de7f75fc36db068","input_tokens":46,"truncated":false},
- {"id":"massive-603-choice","group_id":"massive-alarm-2fe60ba4951e76d11029","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a2c3805f89c00f884c27440a1dc6ca929b7eeb892ce4ceada9d89cae658a68e2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.365234375,-4.1015625,0.50048828125],"temperature":1.0,"type":"choice","source_id":"603","source_row_hash":"c421b5916d3212e4b01829e3c79d953d94f710d02093a9cae17e52e58d84f7e8","input_tokens":53,"truncated":false},
- {"id":"massive-603-noul","group_id":"massive-alarm-2fe60ba4951e76d11029","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"16ed49a37ca746de796382b34b307c96bd23e13298e2da96f4ba2c4d43c40a1e","option_order":["false","true"],"gold":"true","raw_logits":[-0.6171875,0.98193359375],"temperature":1.0,"type":"noul","source_id":"603","source_row_hash":"c421b5916d3212e4b01829e3c79d953d94f710d02093a9cae17e52e58d84f7e8","input_tokens":48,"truncated":false},
- {"id":"massive-611-choice","group_id":"massive-alarm-8563c786dc56bc354169","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"7d47fe013fa4abd1bef6768aaa6e056adc5c74153c2b553727f66e3455fc540d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[7.2578125,-4.58203125,-4.5859375],"temperature":1.0,"type":"choice","source_id":"611","source_row_hash":"bcebdb7a1783441cd8df39b63bf7fcb15b183ef32d3ed586988011b72526dd5e","input_tokens":56,"truncated":false},
- {"id":"massive-611-noul","group_id":"massive-alarm-8563c786dc56bc354169","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"2c9aee36cc50bdb10c9f6773fa1550f9c16e930556d25a6985c09c8505a49de6","option_order":["false","true"],"gold":"true","raw_logits":[-2.419921875,0.92041015625],"temperature":1.0,"type":"noul","source_id":"611","source_row_hash":"bcebdb7a1783441cd8df39b63bf7fcb15b183ef32d3ed586988011b72526dd5e","input_tokens":51,"truncated":false},
- {"id":"massive-6161-choice","group_id":"massive-alarm-ce6e77c02022e073ef83","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a8c2d9d0d4c08ca16e412474a4c2e9403ffab98cbe9aa1d0e3d144dbc1b52a1a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.0234375,-3.31640625,1.15234375],"temperature":1.0,"type":"choice","source_id":"6161","source_row_hash":"3b9476d18f0671741201b748e7aae8b4bef4da841422e3890047663b8cfc6b58","input_tokens":49,"truncated":false},
- {"id":"massive-6161-noul","group_id":"massive-alarm-ce6e77c02022e073ef83","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"de9c4de3d736330a69d1500ab613cb567c5d8a860bdd585ae9492f5e000f903b","option_order":["false","true"],"gold":"false","raw_logits":[-1.6689453125,0.68359375],"temperature":1.0,"type":"noul","source_id":"6161","source_row_hash":"3b9476d18f0671741201b748e7aae8b4bef4da841422e3890047663b8cfc6b58","input_tokens":44,"truncated":false},
- {"id":"massive-7042-choice","group_id":"massive-alarm-d393bf328d9e34b61362","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"fbeee415bfc936dd946bd415f67e0202af1611d6e2e36b7a7c5975d736165f02","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.8984375,-4.5703125,-3.0546875],"temperature":1.0,"type":"choice","source_id":"7042","source_row_hash":"c88b745c72a0ae2239775dec1abf7adfa60007cd0d466769d82d72f70bfbfc16","input_tokens":55,"truncated":false},
- {"id":"massive-7042-noul","group_id":"massive-alarm-d393bf328d9e34b61362","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"df2947e1b837c93ad24d1ba998064dae52f9b9deb1ef572a8b7b2b5f12ea8595","option_order":["false","true"],"gold":"true","raw_logits":[-1.6767578125,0.1278076171875],"temperature":1.0,"type":"noul","source_id":"7042","source_row_hash":"c88b745c72a0ae2239775dec1abf7adfa60007cd0d466769d82d72f70bfbfc16","input_tokens":50,"truncated":false},
- {"id":"massive-7277-choice","group_id":"massive-alarm-debb470de19756d91704","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"24fbe87da2055f9d886b5876c64a68454a41f99da5507c2686870465a513f63e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.0390625,-3.416015625,-1.0498046875],"temperature":1.0,"type":"choice","source_id":"7277","source_row_hash":"aeae185a00d916458ae3ab6aa4bfff1cd313a140872c45d27d15d576fe0ca9bc","input_tokens":53,"truncated":false},
- {"id":"massive-7277-noul","group_id":"massive-alarm-debb470de19756d91704","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"05b9feb244fca27da4be4e37c2cf62fdd13a90d5927b297ec2467bee1ebcaf42","option_order":["false","true"],"gold":"true","raw_logits":[-2.09375,1.1689453125],"temperature":1.0,"type":"noul","source_id":"7277","source_row_hash":"aeae185a00d916458ae3ab6aa4bfff1cd313a140872c45d27d15d576fe0ca9bc","input_tokens":48,"truncated":false},
- {"id":"massive-7865-choice","group_id":"massive-alarm-d5d345224d210f90f9f5","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"8b0593a1fdf0d6c7ced2ac1338010fe12e88ba4292f508dc4866529053d416c3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.99609375,-3.4453125,-2.880859375],"temperature":1.0,"type":"choice","source_id":"7865","source_row_hash":"1d2893dc1fc8769432ee4ae1da4f838791060f36eeb8a5465921654fea2966a2","input_tokens":53,"truncated":false},
- {"id":"massive-7865-noul","group_id":"massive-alarm-d5d345224d210f90f9f5","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"ade30874b49054351cc4d461647189dd5673e5a4a54e10ba7a5316569792c804","option_order":["false","true"],"gold":"true","raw_logits":[-2.541015625,1.515625],"temperature":1.0,"type":"noul","source_id":"7865","source_row_hash":"1d2893dc1fc8769432ee4ae1da4f838791060f36eeb8a5465921654fea2966a2","input_tokens":48,"truncated":false},
- {"id":"massive-8252-choice","group_id":"massive-alarm-b0a22b8198f4993e1291","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"ba3abaddc76eb45141e707cb4d2cae8e7bbd610553b4c9cbfcb65bc0ee921400","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.5078125,-3.552734375,-1.6689453125],"temperature":1.0,"type":"choice","source_id":"8252","source_row_hash":"47b417240a61f167aba697da9695e406562d95fbcfa8317d40ff52b052c7329b","input_tokens":50,"truncated":false},
- {"id":"massive-8252-noul","group_id":"massive-alarm-b0a22b8198f4993e1291","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"f2f944c5c1af54dd0f1ea98d42d7a2f19324eb3f9c9be5ca3bba7a6701a567b6","option_order":["false","true"],"gold":"true","raw_logits":[-2.232421875,1.2470703125],"temperature":1.0,"type":"noul","source_id":"8252","source_row_hash":"47b417240a61f167aba697da9695e406562d95fbcfa8317d40ff52b052c7329b","input_tokens":45,"truncated":false},
- {"id":"massive-8374-choice","group_id":"massive-alarm-b2c391a264a7fc0e8d05","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"a54daba394b794d67ecb3fb82069387e93349d371a89e7d827f80635326ef79b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.12408447265625,-2.046875,-0.1451416015625],"temperature":1.0,"type":"choice","source_id":"8374","source_row_hash":"f9b7880829291847e8117d7cc48838e1f2289986e30778ecb70f1426263e361f","input_tokens":53,"truncated":false},
- {"id":"massive-8374-noul","group_id":"massive-alarm-b2c391a264a7fc0e8d05","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"81070c78d0e7723baaaf3f84c1722d81c18fe2e65502cfa00afa997dffefb8b0","option_order":["false","true"],"gold":"true","raw_logits":[-0.9482421875,1.0],"temperature":1.0,"type":"noul","source_id":"8374","source_row_hash":"f9b7880829291847e8117d7cc48838e1f2289986e30778ecb70f1426263e361f","input_tokens":48,"truncated":false},
- {"id":"massive-838-choice","group_id":"massive-alarm-9c1aa24d8856b1a9b594","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"2e7152c29865640d059a093638cf15c424a87c28b0fbe0919bfc7f40c72e596b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.068359375,-4.296875,1.1064453125],"temperature":1.0,"type":"choice","source_id":"838","source_row_hash":"0374f1afd3ddf573375d73c97e68b801e7847739ab120bddcddb5f24ddf9301f","input_tokens":52,"truncated":false},
- {"id":"massive-838-noul","group_id":"massive-alarm-9c1aa24d8856b1a9b594","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"5553f4f5fc0c112d6e39cc75d646adbf627ea97450649ed1d5fc19aef431748b","option_order":["false","true"],"gold":"false","raw_logits":[-1.46875,-0.341796875],"temperature":1.0,"type":"noul","source_id":"838","source_row_hash":"0374f1afd3ddf573375d73c97e68b801e7847739ab120bddcddb5f24ddf9301f","input_tokens":47,"truncated":false},
- {"id":"massive-8881-choice","group_id":"massive-alarm-a3097c674dea080a2da5","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d54982cbf06da42f2e2f43ca280cb876161678636d5bb6e8ffee7a0d3bb93e7a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.8544921875,-2.931640625,-2.2109375],"temperature":1.0,"type":"choice","source_id":"8881","source_row_hash":"b4a67a7ddf8a2064162255cea468172e3b6117bc5912380b2561538d15fc942f","input_tokens":55,"truncated":false},
- {"id":"massive-8881-noul","group_id":"massive-alarm-a3097c674dea080a2da5","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"158e662345ac34c7894aaae13fa572924b0d5f2caacf6e3b84764fb118746db5","option_order":["false","true"],"gold":"true","raw_logits":[-2.541015625,1.11328125],"temperature":1.0,"type":"noul","source_id":"8881","source_row_hash":"b4a67a7ddf8a2064162255cea468172e3b6117bc5912380b2561538d15fc942f","input_tokens":50,"truncated":false},
- {"id":"massive-8931-choice","group_id":"massive-alarm-db8555fe3197239b3206","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"d980fa7dedacf5e72ae37665538e2793b4f022673a8d82eb2e41284118fa51d0","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.3046875,-2.083984375,-1.1513671875],"temperature":1.0,"type":"choice","source_id":"8931","source_row_hash":"8f70a2fc9af29a53f173113c232962f4df7836f331b2282b37caf1ccab49a781","input_tokens":52,"truncated":false},
- {"id":"massive-8931-noul","group_id":"massive-alarm-db8555fe3197239b3206","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"7ec3b962a5c5a78cb83727987c51167dc8c8d7c76e5cbc8931b315728f12ca47","option_order":["false","true"],"gold":"true","raw_logits":[-1.7548828125,1.1884765625],"temperature":1.0,"type":"noul","source_id":"8931","source_row_hash":"8f70a2fc9af29a53f173113c232962f4df7836f331b2282b37caf1ccab49a781","input_tokens":47,"truncated":false},
- {"id":"massive-9113-choice","group_id":"massive-alarm-397aba5493c3b2aa0106","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"cbf50a8bce382e1c8069c25f24d9187c0581bbf3a4557c96ba8e8c0bd4d094e4","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.126953125,-4.140625,0.7939453125],"temperature":1.0,"type":"choice","source_id":"9113","source_row_hash":"fb4976f4148eaf6a56d495a37fe34a0ea3cbc0cde2a2b924dfb2bb9d43e3fcc9","input_tokens":55,"truncated":false},
- {"id":"massive-9113-noul","group_id":"massive-alarm-397aba5493c3b2aa0106","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"1eb2651db8e5fb0ffe9c87a86a1eea9a93c803613b6109ed7b243775301390bc","option_order":["false","true"],"gold":"true","raw_logits":[-0.2366943359375,0.02410888671875],"temperature":1.0,"type":"noul","source_id":"9113","source_row_hash":"fb4976f4148eaf6a56d495a37fe34a0ea3cbc0cde2a2b924dfb2bb9d43e3fcc9","input_tokens":50,"truncated":false},
- {"id":"massive-919-choice","group_id":"massive-alarm-4c0a77d504ba07897c11","task_id":"massive_alarm_choice_v1","split":"calibration","sample_hash":"1df3beb000463e7bc08b13bb810b877d176680c61686b53599cffc5b0d579beb","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.984375,-3.20703125,4.3359375],"temperature":1.0,"type":"choice","source_id":"919","source_row_hash":"b899f7728a260ed0af75e47ce729fb61bac2aa810bbdcc2b39e068dd142e1140","input_tokens":52,"truncated":false},
- {"id":"massive-919-noul","group_id":"massive-alarm-4c0a77d504ba07897c11","task_id":"massive_alarm_set_noul_v1","split":"calibration","sample_hash":"65b4e822aafcca31aac32995a5110244feba92743a197f2bb18d469b77384537","option_order":["false","true"],"gold":"false","raw_logits":[-1.474609375,0.272705078125],"temperature":1.0,"type":"noul","source_id":"919","source_row_hash":"b899f7728a260ed0af75e47ce729fb61bac2aa810bbdcc2b39e068dd142e1140","input_tokens":47,"truncated":false},
- {"id":"massive-0-choice","group_id":"massive-alarm-a37bbbb0764cc5bf8d8c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"77b0b32f31535aa2bdd9f6f747335478e4a43d19f2018d740cec1c1fa264a584","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.77197265625,-3.3671875,-0.1776123046875],"temperature":1.0,"type":"choice","source_id":"0","source_row_hash":"c7526effce043a65aeb540d2fd4ed2c638ea92efb82184525bfa946c59419162","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.77197265625,-3.3671875,-0.1776123046875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-0-noul","group_id":"massive-alarm-a37bbbb0764cc5bf8d8c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"35ff02590dfdcdf3d2f174132d6178af71b02166065f1ef047c1e306f4cae5f9","option_order":["false","true"],"gold":"true","raw_logits":[-1.6298828125,0.81640625],"temperature":1.0,"type":"noul","source_id":"0","source_row_hash":"c7526effce043a65aeb540d2fd4ed2c638ea92efb82184525bfa946c59419162","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.6298828125,0.81640625],"calibrated_temperature":5.0},
- {"id":"massive-1205-choice","group_id":"massive-alarm-46ec519726ad99259312","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c5336a0dbeaab10f6d75e474cd8ca690a4a17b614be1c36aa9bc422f3420b0e3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.78125,-4.015625,2.580078125],"temperature":1.0,"type":"choice","source_id":"1205","source_row_hash":"c6f272625d9dddb2157b268ff169f20764ccb2c15d8194a208dd4572661cf287","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-1.78125,-4.015625,2.580078125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1205-noul","group_id":"massive-alarm-46ec519726ad99259312","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ad37d94fa29dae7c5a19037407667cf3ec16629415f0f66c6d35b1bf2d99bd10","option_order":["false","true"],"gold":"false","raw_logits":[-1.56640625,0.179443359375],"temperature":1.0,"type":"noul","source_id":"1205","source_row_hash":"c6f272625d9dddb2157b268ff169f20764ccb2c15d8194a208dd4572661cf287","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.56640625,0.179443359375],"calibrated_temperature":5.0},
- {"id":"massive-1264-choice","group_id":"massive-alarm-81721bc34387bf04c532","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"5d1b1f024710e0ccc55d6f8617daa19c339f01457cfb0e6ce8f41ad766f25b55","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[1.2314453125,-1.3837890625,-0.91796875],"temperature":1.0,"type":"choice","source_id":"1264","source_row_hash":"b0e2b6e0b5645a28c5ef36a7b84a2f2c1967d0af1db72136cb632466ac4a2bde","input_tokens":53,"truncated":false,"calibrated_raw_logits":[1.2314453125,-1.3837890625,-0.91796875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1264-noul","group_id":"massive-alarm-81721bc34387bf04c532","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"132b85b5df1b397bcfc73c5408e767840171c95421a29b96e6f3ead9162b793b","option_order":["false","true"],"gold":"false","raw_logits":[-2.1015625,0.94873046875],"temperature":1.0,"type":"noul","source_id":"1264","source_row_hash":"b0e2b6e0b5645a28c5ef36a7b84a2f2c1967d0af1db72136cb632466ac4a2bde","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.1015625,0.94873046875],"calibrated_temperature":5.0},
- {"id":"massive-1452-choice","group_id":"massive-alarm-426bdf23716beff075f4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ba17e7a041b87e59af379c45db0671a59b60e5fa051c7555195d8a9da3aa2337","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.24609375,-3.59375,1.53515625],"temperature":1.0,"type":"choice","source_id":"1452","source_row_hash":"1bfe246bb5276edcdcb98bb0446151ac77373bd9a92f38c219359eb38e6f7682","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-3.24609375,-3.59375,1.53515625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1452-noul","group_id":"massive-alarm-426bdf23716beff075f4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"12849a0b88a8ff2a993968e6b0182681c95a4694070c5ccacee0ed5f2b6524a0","option_order":["false","true"],"gold":"false","raw_logits":[-1.767578125,0.370849609375],"temperature":1.0,"type":"noul","source_id":"1452","source_row_hash":"1bfe246bb5276edcdcb98bb0446151ac77373bd9a92f38c219359eb38e6f7682","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.767578125,0.370849609375],"calibrated_temperature":5.0},
- {"id":"massive-1454-choice","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ce41769466fb7357d3ccff960d17d722b3d6847a9cb11833b48f3c85dca068d5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.6337890625,-3.45703125,1.048828125],"temperature":1.0,"type":"choice","source_id":"1454","source_row_hash":"145608279739b62c2458975fa2f71167df19ca5eba37351354515f842abe0aee","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-1.6337890625,-3.45703125,1.048828125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1454-noul","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"792f47e0d63956a5accbee0429b61de43a74bda6d3f490cf082803c5b8f38138","option_order":["false","true"],"gold":"false","raw_logits":[-2.0078125,0.259521484375],"temperature":1.0,"type":"noul","source_id":"1454","source_row_hash":"145608279739b62c2458975fa2f71167df19ca5eba37351354515f842abe0aee","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.0078125,0.259521484375],"calibrated_temperature":5.0},
- {"id":"massive-1483-choice","group_id":"massive-alarm-2c21587cc19c72b110b3","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"f30db95f41dc4f0696ec72519901b57e25e672544e091d5cb2dc86ea24665f7e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[6.015625,-2.25390625,-2.994140625],"temperature":1.0,"type":"choice","source_id":"1483","source_row_hash":"890f145f8f9f0b5fd1d7091f1f3c2e12575f505e096fb255d1ead352d3634b9a","input_tokens":51,"truncated":false,"calibrated_raw_logits":[6.015625,-2.25390625,-2.994140625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1483-noul","group_id":"massive-alarm-2c21587cc19c72b110b3","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7ddce951250db5a4434c1ab2a0c87281815f837602586c62fb1730e47234864b","option_order":["false","true"],"gold":"false","raw_logits":[-1.54296875,0.85107421875],"temperature":1.0,"type":"noul","source_id":"1483","source_row_hash":"890f145f8f9f0b5fd1d7091f1f3c2e12575f505e096fb255d1ead352d3634b9a","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.54296875,0.85107421875],"calibrated_temperature":5.0},
- {"id":"massive-1635-choice","group_id":"massive-alarm-f9e0722a15d811b9f803","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"bb88865e7bc83986e78d070eaa64317d656ec97f98105b2ad26d92a38d668e08","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.658203125,0.01424407958984375,-2.0234375],"temperature":1.0,"type":"choice","source_id":"1635","source_row_hash":"d1953237537debb810d6f0e9c57dfd7c06257fcb558e7b2a16b137f96c4304d0","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.658203125,0.01424407958984375,-2.0234375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1635-noul","group_id":"massive-alarm-f9e0722a15d811b9f803","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ea73bed5f72a1db4db41433a4c84c6d710acf14be98e446f68d74d9b1dcdfd05","option_order":["false","true"],"gold":"false","raw_logits":[-1.5322265625,0.79248046875],"temperature":1.0,"type":"noul","source_id":"1635","source_row_hash":"d1953237537debb810d6f0e9c57dfd7c06257fcb558e7b2a16b137f96c4304d0","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.5322265625,0.79248046875],"calibrated_temperature":5.0},
- {"id":"massive-1646-choice","group_id":"massive-alarm-62749a68e3e345ba0baf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"64d7678b68d45c87ca45fc5bf39f26b3d45490f848ea4a4b5a760af9d994e293","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.42578125,-0.8388671875,-0.951171875],"temperature":1.0,"type":"choice","source_id":"1646","source_row_hash":"45be821060a8270e88285729f71d4851684eccb3f8e08dc3af41d53e22bd4a93","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.42578125,-0.8388671875,-0.951171875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1646-noul","group_id":"massive-alarm-62749a68e3e345ba0baf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fea796b609111eb3c000991a208e87211ae8c359a7cdc07e754bc56caa22d373","option_order":["false","true"],"gold":"false","raw_logits":[-2.109375,1.1884765625],"temperature":1.0,"type":"noul","source_id":"1646","source_row_hash":"45be821060a8270e88285729f71d4851684eccb3f8e08dc3af41d53e22bd4a93","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.109375,1.1884765625],"calibrated_temperature":5.0},
- {"id":"massive-1658-choice","group_id":"massive-alarm-364266652280dd7c910e","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0230e908a3762610b091476be9f4a05d8480c6e8cbaa6f719a31d29d363d5a58","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.298828125,-0.3212890625,-0.67919921875],"temperature":1.0,"type":"choice","source_id":"1658","source_row_hash":"93fa8ba611a374efaa1f7aaead3325da80c2330ae0ae622640f247c61e01d54b","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.298828125,-0.3212890625,-0.67919921875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1658-noul","group_id":"massive-alarm-364266652280dd7c910e","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"84713593dc3c74a749551857971c03d1a91d64f10e9b64a4c0f9dcde5e4a3ea7","option_order":["false","true"],"gold":"false","raw_logits":[-1.306640625,0.36962890625],"temperature":1.0,"type":"noul","source_id":"1658","source_row_hash":"93fa8ba611a374efaa1f7aaead3325da80c2330ae0ae622640f247c61e01d54b","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.306640625,0.36962890625],"calibrated_temperature":5.0},
- {"id":"massive-1675-choice","group_id":"massive-alarm-bf4ac6ee8fec3f0b558c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1466e3c3f90ac2ac30f3f4f1b0d94ead57c08f3d7497b6ef1f9ea9c4eac8711f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.48046875,-4.32421875,2.890625],"temperature":1.0,"type":"choice","source_id":"1675","source_row_hash":"4d208d74d0efe7e2c8ca898d3b22951bcde4612a05aa052742a78e2af0f54efa","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-3.48046875,-4.32421875,2.890625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1675-noul","group_id":"massive-alarm-bf4ac6ee8fec3f0b558c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"275b4e3cc0377df561399ef1fad6be14b832e24bd56aae9ad0a547a7d358c00e","option_order":["false","true"],"gold":"false","raw_logits":[-2.11328125,0.400146484375],"temperature":1.0,"type":"noul","source_id":"1675","source_row_hash":"4d208d74d0efe7e2c8ca898d3b22951bcde4612a05aa052742a78e2af0f54efa","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.11328125,0.400146484375],"calibrated_temperature":5.0},
- {"id":"massive-1719-choice","group_id":"massive-alarm-3aeb032bc1ab13756f0a","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8c5933a9aeb20b97adbd1e5b98c5cbd42e62e879e8e780ffa59c46e4d607a736","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.390625,-3.01171875,-1.6962890625],"temperature":1.0,"type":"choice","source_id":"1719","source_row_hash":"c920466329cad750fe4b3605a5197b2eb9ea1b411cb18b89cd499d0a209dd0da","input_tokens":56,"truncated":false,"calibrated_raw_logits":[4.390625,-3.01171875,-1.6962890625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1719-noul","group_id":"massive-alarm-3aeb032bc1ab13756f0a","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fd8b7628ff216af2d1a0577794f912897a4b1852599ad9d32b74f55fe23bfdc1","option_order":["false","true"],"gold":"true","raw_logits":[-1.40625,0.93798828125],"temperature":1.0,"type":"noul","source_id":"1719","source_row_hash":"c920466329cad750fe4b3605a5197b2eb9ea1b411cb18b89cd499d0a209dd0da","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.40625,0.93798828125],"calibrated_temperature":5.0},
- {"id":"massive-174-choice","group_id":"massive-alarm-ce0229b7c6149d882cf3","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7839990611efc7091e313b70ab12446f38e919bf3775e40e3288cb228c8e2b77","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.22265625,-2.859375,-1.3046875],"temperature":1.0,"type":"choice","source_id":"174","source_row_hash":"e8f8762e0d5998c6602731393d2fab6fdfb7662662c9a73a02e95041799cc0a9","input_tokens":54,"truncated":false,"calibrated_raw_logits":[4.22265625,-2.859375,-1.3046875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-174-noul","group_id":"massive-alarm-ce0229b7c6149d882cf3","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"46d0f88a1bb77721bef837b24d61db6244bac9678ca6744ac32fa010615a69c6","option_order":["false","true"],"gold":"true","raw_logits":[-2.4375,1.5888671875],"temperature":1.0,"type":"noul","source_id":"174","source_row_hash":"e8f8762e0d5998c6602731393d2fab6fdfb7662662c9a73a02e95041799cc0a9","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.4375,1.5888671875],"calibrated_temperature":5.0},
- {"id":"massive-1774-choice","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6b00cec1b29c2fa5a974ad219e812cc757b1e41937665cfacf265bb6c5a0a327","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.6337890625,-3.45703125,1.048828125],"temperature":1.0,"type":"choice","source_id":"1774","source_row_hash":"e2d9115544309a1c262318bb19f1d82615d1bb4704ad53d26c3bb5265d1cd2f0","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-1.6337890625,-3.45703125,1.048828125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1774-noul","group_id":"massive-alarm-3546d1ed79103c96be61","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"10fce9ced8e6eec5b8b870dbf1b532e302afac2b8bc85d93119e264ea2ffebe3","option_order":["false","true"],"gold":"false","raw_logits":[-2.0078125,0.259521484375],"temperature":1.0,"type":"noul","source_id":"1774","source_row_hash":"e2d9115544309a1c262318bb19f1d82615d1bb4704ad53d26c3bb5265d1cd2f0","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.0078125,0.259521484375],"calibrated_temperature":5.0},
- {"id":"massive-1862-choice","group_id":"massive-alarm-869c1474b1c82e78b00c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"81e679b9cc2bda219fc616a52ccb9d7eecdad4f0c7350f44c21a5b12e38f9f44","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.253173828125,-0.1468505859375,-0.475341796875],"temperature":1.0,"type":"choice","source_id":"1862","source_row_hash":"d7dcb916b78b9417322277ed5947fe9386a9b6a258ec5d7237bf0a8db3eab495","input_tokens":51,"truncated":false,"calibrated_raw_logits":[0.253173828125,-0.1468505859375,-0.475341796875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1862-noul","group_id":"massive-alarm-869c1474b1c82e78b00c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"03ca4bdaf3b5b365cb529d8d6fcd255ea2c26a37f058442867ac3939e66adcf1","option_order":["false","true"],"gold":"false","raw_logits":[-1.8515625,1.146484375],"temperature":1.0,"type":"noul","source_id":"1862","source_row_hash":"d7dcb916b78b9417322277ed5947fe9386a9b6a258ec5d7237bf0a8db3eab495","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.8515625,1.146484375],"calibrated_temperature":5.0},
- {"id":"massive-1913-choice","group_id":"massive-alarm-04a340be86134fb825f2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0c21d545d8a862238e988f7cb58faa8ea1930a3d9adddb2d93adc69289d959ef","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.48974609375,-2.974609375,-0.37744140625],"temperature":1.0,"type":"choice","source_id":"1913","source_row_hash":"fd24179d6c9dc537d667cb6ff6fe25ab9e00652ce36f045a39da3a2eb3efeeae","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-0.48974609375,-2.974609375,-0.37744140625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1913-noul","group_id":"massive-alarm-04a340be86134fb825f2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f821536cce50ca455cf0e83aa0589f117ccdc48ac96ee7b0de3c0e3f00fe5885","option_order":["false","true"],"gold":"true","raw_logits":[-2.65234375,1.6552734375],"temperature":1.0,"type":"noul","source_id":"1913","source_row_hash":"fd24179d6c9dc537d667cb6ff6fe25ab9e00652ce36f045a39da3a2eb3efeeae","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.65234375,1.6552734375],"calibrated_temperature":5.0},
- {"id":"massive-1982-choice","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"046ef58708776eb61d36aba5282ff98022582353215e6eae4cb389226c6c9ff2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.10546875,-3.75,-2.833984375],"temperature":1.0,"type":"choice","source_id":"1982","source_row_hash":"68a92316a11c74af3d569a87cc55e8d8e9f116b771db9601d3c65b44d659be25","input_tokens":55,"truncated":false,"calibrated_raw_logits":[5.10546875,-3.75,-2.833984375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-1982-noul","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"6f13803fd13041a079ba5f448a57ab23e1d609a8fd17f4c9b68635b312890988","option_order":["false","true"],"gold":"true","raw_logits":[-2.30859375,2.40625],"temperature":1.0,"type":"noul","source_id":"1982","source_row_hash":"68a92316a11c74af3d569a87cc55e8d8e9f116b771db9601d3c65b44d659be25","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-2.30859375,2.40625],"calibrated_temperature":5.0},
- {"id":"massive-201-choice","group_id":"massive-alarm-34510ceb5116eee89abf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1919418590df3569bb082e98b9368556569025bf81fd8c5c80d0f62fb8012279","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.8603515625,-2.9609375,3.517578125],"temperature":1.0,"type":"choice","source_id":"201","source_row_hash":"fec125430204fc240f59fa0b8afc5ab33fb231589e377ca142563556d36313c1","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.8603515625,-2.9609375,3.517578125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-201-noul","group_id":"massive-alarm-34510ceb5116eee89abf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e682c7af0260299ab1b72d73ba69134a323fd6c0a4488f9e8ebb899d12e2f95c","option_order":["false","true"],"gold":"false","raw_logits":[-2.462890625,0.419921875],"temperature":1.0,"type":"noul","source_id":"201","source_row_hash":"fec125430204fc240f59fa0b8afc5ab33fb231589e377ca142563556d36313c1","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.462890625,0.419921875],"calibrated_temperature":5.0},
- {"id":"massive-2134-choice","group_id":"massive-alarm-f8e7f38a4927528ba9b2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8cb6e812da09a75b6e72019b09c11e0a85c207aa5c8dbfddc570fd92d30458de","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.0035247802734375,-3.1875,-0.9208984375],"temperature":1.0,"type":"choice","source_id":"2134","source_row_hash":"7d7e16a4e1ed61a8de9442d3e01f17fafdc055b6548e09778de80844d7b5df78","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-0.0035247802734375,-3.1875,-0.9208984375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2134-noul","group_id":"massive-alarm-f8e7f38a4927528ba9b2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"9689620330e6aaa79089562fe3f74c74152b155ba69f48bbb97341c234944c63","option_order":["false","true"],"gold":"true","raw_logits":[-2.033203125,1.36328125],"temperature":1.0,"type":"noul","source_id":"2134","source_row_hash":"7d7e16a4e1ed61a8de9442d3e01f17fafdc055b6548e09778de80844d7b5df78","input_tokens":43,"truncated":false,"calibrated_raw_logits":[-2.033203125,1.36328125],"calibrated_temperature":5.0},
- {"id":"massive-2166-choice","group_id":"massive-alarm-9519286825bfef55a4e0","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"69ab8ef1a844ffe7974da7b965063744250ea864ceb8d5a1efbdbdaa3022c36c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.90625,-4.01171875,-3.65234375],"temperature":1.0,"type":"choice","source_id":"2166","source_row_hash":"06505fac3fedd536d679acbcbc516ba66217322dc627d35a70fed7066fc76658","input_tokens":54,"truncated":false,"calibrated_raw_logits":[5.90625,-4.01171875,-3.65234375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2166-noul","group_id":"massive-alarm-9519286825bfef55a4e0","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"473a9f5612897736e32cafd3b6a9730e7740e1758fb072b71087bd827cdaf1b6","option_order":["false","true"],"gold":"true","raw_logits":[-2.603515625,1.41796875],"temperature":1.0,"type":"noul","source_id":"2166","source_row_hash":"06505fac3fedd536d679acbcbc516ba66217322dc627d35a70fed7066fc76658","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.603515625,1.41796875],"calibrated_temperature":5.0},
- {"id":"massive-2167-choice","group_id":"massive-alarm-d02ea237a24fb1becb66","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cde8b8d2e32f6683425e46e437ca44af31aa51bb38d8e675498c818dde0ec4d8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-1.3681640625,-3.48046875,0.047393798828125],"temperature":1.0,"type":"choice","source_id":"2167","source_row_hash":"bc3911b9a1c3926a1c45f4eb47de667f89e300ef4b21a13428a67c529a0efa24","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.3681640625,-3.48046875,0.047393798828125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2167-noul","group_id":"massive-alarm-d02ea237a24fb1becb66","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"83343991208595707f936ac6728585bd5637a32b678c39ae9802bf00e123d2f7","option_order":["false","true"],"gold":"true","raw_logits":[-1.54296875,1.0595703125],"temperature":1.0,"type":"noul","source_id":"2167","source_row_hash":"bc3911b9a1c3926a1c45f4eb47de667f89e300ef4b21a13428a67c529a0efa24","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.54296875,1.0595703125],"calibrated_temperature":5.0},
- {"id":"massive-2168-choice","group_id":"massive-alarm-cff75109d691bab986bf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7d2f0a6e8d4f34fa7bbf37a4a5aed96e8976d9c64cfe47bbc2c9660bdaaf199c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-0.76611328125,-3.515625,2.119140625],"temperature":1.0,"type":"choice","source_id":"2168","source_row_hash":"43fe92fb79734cdcfc39bb3d6f1df8e11c61fff1e834e468a137c272fb88ddb4","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-0.76611328125,-3.515625,2.119140625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2168-noul","group_id":"massive-alarm-cff75109d691bab986bf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"b6eb787be2412c61dd4e39b1920656d4a1bd9f081a93f2fea79db2f421bd3f37","option_order":["false","true"],"gold":"false","raw_logits":[-1.7783203125,0.79736328125],"temperature":1.0,"type":"noul","source_id":"2168","source_row_hash":"43fe92fb79734cdcfc39bb3d6f1df8e11c61fff1e834e468a137c272fb88ddb4","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.7783203125,0.79736328125],"calibrated_temperature":5.0},
- {"id":"massive-2334-choice","group_id":"massive-alarm-68e901c570a8bc247ea5","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"dad5cc765a8b621e7d161351f5c7d96e7990e74890f935c8da2bcc8f4465dcaf","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.072265625,-4.1484375,-0.0853271484375],"temperature":1.0,"type":"choice","source_id":"2334","source_row_hash":"37ed15aaa5179f417ed1db5f5e5fbf3fd407b5f570abad133bf0c99a1f824b8a","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.072265625,-4.1484375,-0.0853271484375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2334-noul","group_id":"massive-alarm-68e901c570a8bc247ea5","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"979a7e2f21c7c3cc727090ad2a84fe0ff99b663a18c7fa9599ef652f08bb46d2","option_order":["false","true"],"gold":"true","raw_logits":[-0.97021484375,1.2099609375],"temperature":1.0,"type":"noul","source_id":"2334","source_row_hash":"37ed15aaa5179f417ed1db5f5e5fbf3fd407b5f570abad133bf0c99a1f824b8a","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-0.97021484375,1.2099609375],"calibrated_temperature":5.0},
- {"id":"massive-2564-choice","group_id":"massive-alarm-606907415c29daf2edfb","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c44f45e943b2a653cda222a15ba4d17ed7ac9b9d5bae688e436eca315eaa7e16","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.66796875,-3.73046875,-2.88671875],"temperature":1.0,"type":"choice","source_id":"2564","source_row_hash":"95d1ba4090b52ba0ce5fb3444edef0250d596ad13f4f69a621c4a90930827e9f","input_tokens":56,"truncated":false,"calibrated_raw_logits":[4.66796875,-3.73046875,-2.88671875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2564-noul","group_id":"massive-alarm-606907415c29daf2edfb","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"87f3caebd55ff64483153128b3dc7a98c34a40ae17290470a5b5123949a92735","option_order":["false","true"],"gold":"true","raw_logits":[-2.638671875,1.4833984375],"temperature":1.0,"type":"noul","source_id":"2564","source_row_hash":"95d1ba4090b52ba0ce5fb3444edef0250d596ad13f4f69a621c4a90930827e9f","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.638671875,1.4833984375],"calibrated_temperature":5.0},
- {"id":"massive-2673-choice","group_id":"massive-alarm-8691ee9b0cf94dc2728b","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2e1c9599701f771117cd159e0ce73abb95b349419aacf2f200688397632c2c17","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.77294921875,-4.08203125,-2.396484375],"temperature":1.0,"type":"choice","source_id":"2673","source_row_hash":"1afaa7047e42a574d8e3230ec7c390cd32494a5058aadd8ceba92aa8d8381c72","input_tokens":58,"truncated":false,"calibrated_raw_logits":[-0.77294921875,-4.08203125,-2.396484375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2673-noul","group_id":"massive-alarm-8691ee9b0cf94dc2728b","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7a310326662b1ceb900e0ce2c672d76155807ff8771cd5e70b6705cdbf11793a","option_order":["false","true"],"gold":"false","raw_logits":[-1.7099609375,0.22412109375],"temperature":1.0,"type":"noul","source_id":"2673","source_row_hash":"1afaa7047e42a574d8e3230ec7c390cd32494a5058aadd8ceba92aa8d8381c72","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-1.7099609375,0.22412109375],"calibrated_temperature":5.0},
- {"id":"massive-2870-choice","group_id":"massive-alarm-8a430eb6516ce866e134","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ad6afdc347ef21ce011640d137edaa887c55f4963941c347b23f6b7b21a67dd2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.9970703125,4.4453125,-2.201171875],"temperature":1.0,"type":"choice","source_id":"2870","source_row_hash":"30d1945137ce312fac5d58fb2da24a2c2971e65f2403ad3ac1204ea1451af255","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-0.9970703125,4.4453125,-2.201171875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2870-noul","group_id":"massive-alarm-8a430eb6516ce866e134","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"9d2b0868a4c40cd0a685f0a6b8d495f5d7d98653eab390d7b4b1421dac4b84f1","option_order":["false","true"],"gold":"false","raw_logits":[-1.0615234375,0.373046875],"temperature":1.0,"type":"noul","source_id":"2870","source_row_hash":"30d1945137ce312fac5d58fb2da24a2c2971e65f2403ad3ac1204ea1451af255","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.0615234375,0.373046875],"calibrated_temperature":5.0},
- {"id":"massive-2957-choice","group_id":"massive-alarm-ee34ced4068dccdafbd4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2fc5caa4ae2985c5045322f402b6fbaec115c14788046392c7a1b7eb9e7278ee","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.16796875,-0.132080078125,-0.9990234375],"temperature":1.0,"type":"choice","source_id":"2957","source_row_hash":"cad2ff93096738a0cc95d453cd87af036fed8518b9c6804a76474027052392b2","input_tokens":52,"truncated":false,"calibrated_raw_logits":[4.16796875,-0.132080078125,-0.9990234375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2957-noul","group_id":"massive-alarm-ee34ced4068dccdafbd4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"c01ab9e555be8266d496a98398689b99aa6cb691f9d6a87897ebcd8dd5ee0a67","option_order":["false","true"],"gold":"false","raw_logits":[-1.783203125,1.5283203125],"temperature":1.0,"type":"noul","source_id":"2957","source_row_hash":"cad2ff93096738a0cc95d453cd87af036fed8518b9c6804a76474027052392b2","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.783203125,1.5283203125],"calibrated_temperature":5.0},
- {"id":"massive-2983-choice","group_id":"massive-alarm-80f7027d6c0b934a23f1","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"92c9f6b64792017733c2fa0d9e61f8d9579da249ea18cb26cc1de02a7df19f04","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.037109375,-3.71875,0.493408203125],"temperature":1.0,"type":"choice","source_id":"2983","source_row_hash":"51967e6f2526f9032885c83079a3b964f6d61de7941722049a83749bad8643b9","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-3.037109375,-3.71875,0.493408203125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2983-noul","group_id":"massive-alarm-80f7027d6c0b934a23f1","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a91925e0946afa49892ffde2b9cb42fe9240c5b085c14bff1f9663280a3b296f","option_order":["false","true"],"gold":"false","raw_logits":[-2.271484375,0.56787109375],"temperature":1.0,"type":"noul","source_id":"2983","source_row_hash":"51967e6f2526f9032885c83079a3b964f6d61de7941722049a83749bad8643b9","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.271484375,0.56787109375],"calibrated_temperature":5.0},
- {"id":"massive-2985-choice","group_id":"massive-alarm-cadeebcba526c5ce11e1","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c77bbecf7a4f64203fd3018a54e31f1f7e265dd20b4402e784ae564bf479b282","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.66796875,-4.046875,2.580078125],"temperature":1.0,"type":"choice","source_id":"2985","source_row_hash":"45de5ba6489c1472d66739caf602f19f3dddeb4654edb0633098362cb7245dc5","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-3.66796875,-4.046875,2.580078125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-2985-noul","group_id":"massive-alarm-cadeebcba526c5ce11e1","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ce7d0fa964fdf4b79e9e0090ed6dab1bf8d2a0ba07586c8de0979042fd97fabe","option_order":["false","true"],"gold":"false","raw_logits":[-2.65625,0.5087890625],"temperature":1.0,"type":"noul","source_id":"2985","source_row_hash":"45de5ba6489c1472d66739caf602f19f3dddeb4654edb0633098362cb7245dc5","input_tokens":43,"truncated":false,"calibrated_raw_logits":[-2.65625,0.5087890625],"calibrated_temperature":5.0},
- {"id":"massive-3161-choice","group_id":"massive-alarm-76d4b32f23cb9a972b27","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"70f06e64a39a6ef6fe0fc89ca848b8a2d66e40dbe52d56c3b6b62aa198f16381","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.1484375,2.0390625,-0.23193359375],"temperature":1.0,"type":"choice","source_id":"3161","source_row_hash":"21b811432a21e010c74460ef7e1b4573c17346e6fd3f6b869abdd9ef6ae0c1ef","input_tokens":56,"truncated":false,"calibrated_raw_logits":[2.1484375,2.0390625,-0.23193359375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3161-noul","group_id":"massive-alarm-76d4b32f23cb9a972b27","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"2bbbb3a8362746db393fa624e9043f8dad23e400182090154aade1921dc28ffb","option_order":["false","true"],"gold":"false","raw_logits":[-1.357421875,0.58740234375],"temperature":1.0,"type":"noul","source_id":"3161","source_row_hash":"21b811432a21e010c74460ef7e1b4573c17346e6fd3f6b869abdd9ef6ae0c1ef","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.357421875,0.58740234375],"calibrated_temperature":5.0},
- {"id":"massive-3162-choice","group_id":"massive-alarm-6e7b23cce190bc2d85c5","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ea7591a2f602f59dcb876b7ae44597587400890381314433fe0308a51e1f0d6d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.9912109375,-0.8330078125,-0.1859130859375],"temperature":1.0,"type":"choice","source_id":"3162","source_row_hash":"d15cb4a55b87aa0a3910749116e4161f2684c7546aa4c19dc3c4f99f0f375afe","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.9912109375,-0.8330078125,-0.1859130859375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3162-noul","group_id":"massive-alarm-6e7b23cce190bc2d85c5","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"de8da33771b147c3186d74aa65038831e1cb9ca49cd32cbe4c043568bcde65ae","option_order":["false","true"],"gold":"false","raw_logits":[-1.0283203125,-0.179443359375],"temperature":1.0,"type":"noul","source_id":"3162","source_row_hash":"d15cb4a55b87aa0a3910749116e4161f2684c7546aa4c19dc3c4f99f0f375afe","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.0283203125,-0.179443359375],"calibrated_temperature":5.0},
- {"id":"massive-3213-choice","group_id":"massive-alarm-130f5b17e0470747d338","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0799db4758f49463128dbabd40c943aed44fcc559c87572c05e7ab6d128b4789","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.97900390625,-3.73828125,-0.10302734375],"temperature":1.0,"type":"choice","source_id":"3213","source_row_hash":"9fd345a0d9dc2aed398ed21031246ff893fe651a70a0d745ba90398f720b9acb","input_tokens":53,"truncated":false,"calibrated_raw_logits":[0.97900390625,-3.73828125,-0.10302734375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3213-noul","group_id":"massive-alarm-130f5b17e0470747d338","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f43c35f39a7dc113b60b52e5f60073b04f43737051ee83161483c56e9fe2cc57","option_order":["false","true"],"gold":"true","raw_logits":[-2.564453125,1.2177734375],"temperature":1.0,"type":"noul","source_id":"3213","source_row_hash":"9fd345a0d9dc2aed398ed21031246ff893fe651a70a0d745ba90398f720b9acb","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.564453125,1.2177734375],"calibrated_temperature":5.0},
- {"id":"massive-3398-choice","group_id":"massive-alarm-da466cfc858133a8c89c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"dcb2df6d7f361c5165fff930443cdd8d616f8bfac79a1cfc8eef312ee5956872","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.0,-4.55859375,-3.73046875],"temperature":1.0,"type":"choice","source_id":"3398","source_row_hash":"bf8610ee8a05e9800dd90fb7461d52940fcfc1bf777b1aef8138784628240668","input_tokens":54,"truncated":false,"calibrated_raw_logits":[5.0,-4.55859375,-3.73046875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3398-noul","group_id":"massive-alarm-da466cfc858133a8c89c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f268081c1d5a7aa3525ee6e1106a49e8f95cde1b4b9f2cee63a2f1b497c0ca98","option_order":["false","true"],"gold":"true","raw_logits":[-1.8974609375,1.115234375],"temperature":1.0,"type":"noul","source_id":"3398","source_row_hash":"bf8610ee8a05e9800dd90fb7461d52940fcfc1bf777b1aef8138784628240668","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-1.8974609375,1.115234375],"calibrated_temperature":5.0},
- {"id":"massive-3435-choice","group_id":"massive-alarm-462210a662d596b5be80","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"fd24fd1ddefae756012ea9809480034da52e1590ecb9f855fea742309bb004ad","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.26171875,-3.630859375,1.640625],"temperature":1.0,"type":"choice","source_id":"3435","source_row_hash":"e0d15beb0c523aab28095c68f8a36e66e1f0ce73e302239dc482aad749d2fc21","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-3.26171875,-3.630859375,1.640625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3435-noul","group_id":"massive-alarm-462210a662d596b5be80","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"da919c2441ffa633bc40be48a80679231ee8f87a5a09323f8d1b0de2feb94ef9","option_order":["false","true"],"gold":"false","raw_logits":[-1.9365234375,0.5263671875],"temperature":1.0,"type":"noul","source_id":"3435","source_row_hash":"e0d15beb0c523aab28095c68f8a36e66e1f0ce73e302239dc482aad749d2fc21","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.9365234375,0.5263671875],"calibrated_temperature":5.0},
- {"id":"massive-3694-choice","group_id":"massive-alarm-ba350675a0496c49938e","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"f1f8330521612e78a2cde0c6656f2fba239789708588e8a80b2fc20b737e7845","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.71875,-4.5234375,-4.33984375],"temperature":1.0,"type":"choice","source_id":"3694","source_row_hash":"d1eb20fde014fb2b17be65874b712b3c63c6f59e4a2a59b4dcf2d38b52cce3b5","input_tokens":56,"truncated":false,"calibrated_raw_logits":[6.71875,-4.5234375,-4.33984375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3694-noul","group_id":"massive-alarm-ba350675a0496c49938e","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a62d58a5600dea097aa91ce2c7c14ed15568eee6c9bfd4c6389f8e279899de18","option_order":["false","true"],"gold":"true","raw_logits":[-2.154296875,1.7275390625],"temperature":1.0,"type":"noul","source_id":"3694","source_row_hash":"d1eb20fde014fb2b17be65874b712b3c63c6f59e4a2a59b4dcf2d38b52cce3b5","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.154296875,1.7275390625],"calibrated_temperature":5.0},
- {"id":"massive-3695-choice","group_id":"massive-alarm-b52edfb5a71ea1dcee30","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"202cb568bede45bcb290436753573690741c6ce9aa2bd2488fea63edefc3c144","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.623046875,1.001953125,1.2080078125],"temperature":1.0,"type":"choice","source_id":"3695","source_row_hash":"5f84198d661ad46d3f9fd8c6e1da9ee2f4afe6fa1c77ec4b8ecbb1a18e582de6","input_tokens":52,"truncated":false,"calibrated_raw_logits":[1.623046875,1.001953125,1.2080078125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3695-noul","group_id":"massive-alarm-b52edfb5a71ea1dcee30","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"96938fc7026fb8a5848686387930d87db6f09856773963de5c13c5adc8ceaf17","option_order":["false","true"],"gold":"true","raw_logits":[-1.865234375,1.1982421875],"temperature":1.0,"type":"noul","source_id":"3695","source_row_hash":"5f84198d661ad46d3f9fd8c6e1da9ee2f4afe6fa1c77ec4b8ecbb1a18e582de6","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.865234375,1.1982421875],"calibrated_temperature":5.0},
- {"id":"massive-3702-choice","group_id":"massive-alarm-d974ab158bb24355505f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7e3fea92fc16c779c9bda789dafacaa6be5cbe622e4f5560bfa3fdc82a74801e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[6.21484375,-4.44140625,-4.55859375],"temperature":1.0,"type":"choice","source_id":"3702","source_row_hash":"e618c546f5bea2ac0fe003c4c1dc9c54f82f1a3ab0ae22346f48031406b951c4","input_tokens":55,"truncated":false,"calibrated_raw_logits":[6.21484375,-4.44140625,-4.55859375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3702-noul","group_id":"massive-alarm-d974ab158bb24355505f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"f74bf9955f1d89ea5586cb69e06a3ef816cc3cb2a5b7151d8273a1723839c8e2","option_order":["false","true"],"gold":"false","raw_logits":[-2.099609375,1.6357421875],"temperature":1.0,"type":"noul","source_id":"3702","source_row_hash":"e618c546f5bea2ac0fe003c4c1dc9c54f82f1a3ab0ae22346f48031406b951c4","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-2.099609375,1.6357421875],"calibrated_temperature":5.0},
- {"id":"massive-3761-choice","group_id":"massive-alarm-906532f048bd9c89f168","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"0599547b574a8679aa3e878e382da40df6e16c9d94f84a30b2d769fbe4b85de0","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-1.2392578125,-3.041015625,-0.41552734375],"temperature":1.0,"type":"choice","source_id":"3761","source_row_hash":"ca9d72e81a4b3f3d7e2235562d7d91f1f303f58748ed6e425cb54f84d73dd906","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.2392578125,-3.041015625,-0.41552734375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3761-noul","group_id":"massive-alarm-906532f048bd9c89f168","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4c43d963e2942b0e4b14bba53a12af4bfc6c6e9765a4a95b0b98a038cb326f38","option_order":["false","true"],"gold":"true","raw_logits":[-1.900390625,1.345703125],"temperature":1.0,"type":"noul","source_id":"3761","source_row_hash":"ca9d72e81a4b3f3d7e2235562d7d91f1f303f58748ed6e425cb54f84d73dd906","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.900390625,1.345703125],"calibrated_temperature":5.0},
- {"id":"massive-3771-choice","group_id":"massive-alarm-c45435c75f999b77b736","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"25a21359b2b1ed7367720091eb1992eb1dad9c87afdf98e01b88ad12e7c8b6bd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.8818359375,3.08203125,-1.3564453125],"temperature":1.0,"type":"choice","source_id":"3771","source_row_hash":"2e4a7daa393af56cd536025155d462dbecd3c006ec8952136555253adea64973","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-0.8818359375,3.08203125,-1.3564453125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3771-noul","group_id":"massive-alarm-c45435c75f999b77b736","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"65677218b234245fd090491dad4d645564fa868f49e1c5ff60d2157d2d64e3d0","option_order":["false","true"],"gold":"false","raw_logits":[-1.66796875,0.7587890625],"temperature":1.0,"type":"noul","source_id":"3771","source_row_hash":"2e4a7daa393af56cd536025155d462dbecd3c006ec8952136555253adea64973","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.66796875,0.7587890625],"calibrated_temperature":5.0},
- {"id":"massive-3829-choice","group_id":"massive-alarm-f60b61ceed3880a795df","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7b86cc5b86e8345e60c6261aabec21980fd9dfa33baff9d168197fd184a36dec","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-0.89892578125,-4.5,3.939453125],"temperature":1.0,"type":"choice","source_id":"3829","source_row_hash":"cbdd61b9e23a630e30d582423ee8353b28ecafc5c9b93cb0b7c7d327d1e9ac5c","input_tokens":54,"truncated":false,"calibrated_raw_logits":[-0.89892578125,-4.5,3.939453125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3829-noul","group_id":"massive-alarm-f60b61ceed3880a795df","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"591cecb6a8d39597824aa757e6a24856a9a87ab456991e3c95b4abf9246b2b5c","option_order":["false","true"],"gold":"false","raw_logits":[-2.30078125,0.60546875],"temperature":1.0,"type":"noul","source_id":"3829","source_row_hash":"cbdd61b9e23a630e30d582423ee8353b28ecafc5c9b93cb0b7c7d327d1e9ac5c","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.30078125,0.60546875],"calibrated_temperature":5.0},
- {"id":"massive-3870-choice","group_id":"massive-alarm-cc3a484250d2efcc2043","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1fb7c02d1b63c62c515f630c6f6b1a190f806d8933fa807c965f98c542ea3734","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[2.9140625,0.95361328125,-0.422119140625],"temperature":1.0,"type":"choice","source_id":"3870","source_row_hash":"1eb812098d9c31e9b8c54e259a28eb60b0a677a306bbd79412ca17e0fd267621","input_tokens":56,"truncated":false,"calibrated_raw_logits":[2.9140625,0.95361328125,-0.422119140625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3870-noul","group_id":"massive-alarm-cc3a484250d2efcc2043","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4b85b5fb27bfdd2ab902c1aeebbb8681e8bba5cd4ab7ecb99502da2692776530","option_order":["false","true"],"gold":"false","raw_logits":[-2.1953125,1.1767578125],"temperature":1.0,"type":"noul","source_id":"3870","source_row_hash":"1eb812098d9c31e9b8c54e259a28eb60b0a677a306bbd79412ca17e0fd267621","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.1953125,1.1767578125],"calibrated_temperature":5.0},
- {"id":"massive-3884-choice","group_id":"massive-alarm-021f9f46cfb6e0326d91","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"607244bfee195714e4c96cab20b3b0e4c30e225e8835abe4fc93cbf89fd30998","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.5439453125,-0.671875,-2.466796875],"temperature":1.0,"type":"choice","source_id":"3884","source_row_hash":"51b89a83ff9ab846f795d6b36a0f992d19c4db4c0a9f3ed29cdd233b99bfa59f","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.5439453125,-0.671875,-2.466796875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3884-noul","group_id":"massive-alarm-021f9f46cfb6e0326d91","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"0481094ed9ba36cd8746ca74f5c7144fd855684f2e353ce3d0763a903f4d024d","option_order":["false","true"],"gold":"false","raw_logits":[-1.958984375,0.455322265625],"temperature":1.0,"type":"noul","source_id":"3884","source_row_hash":"51b89a83ff9ab846f795d6b36a0f992d19c4db4c0a9f3ed29cdd233b99bfa59f","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.958984375,0.455322265625],"calibrated_temperature":5.0},
- {"id":"massive-3886-choice","group_id":"massive-alarm-661b37ed9fdd982c94bf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"27c0ff2ef671d83abaebeff897b97cd739dc1770c284ea8bd367da824127f203","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.759765625,-1.4873046875,0.60546875],"temperature":1.0,"type":"choice","source_id":"3886","source_row_hash":"d6664d601ddccd6310963193b24469d0e9d111c0a7f3f23bf551b85d96113ccf","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-1.759765625,-1.4873046875,0.60546875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3886-noul","group_id":"massive-alarm-661b37ed9fdd982c94bf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"1fc4b67408e23e34752d030fee7275c22033746a1eaecef7e6277bffd34047f8","option_order":["false","true"],"gold":"false","raw_logits":[-1.8095703125,1.1162109375],"temperature":1.0,"type":"noul","source_id":"3886","source_row_hash":"d6664d601ddccd6310963193b24469d0e9d111c0a7f3f23bf551b85d96113ccf","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.8095703125,1.1162109375],"calibrated_temperature":5.0},
- {"id":"massive-3986-choice","group_id":"massive-alarm-f2693fd957a1a13798ca","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"d542a5fa84c401daab24a69a30c193f6a2a09bf9263e9bc7a51bf617211fd54c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.13671875,-3.943359375,-2.3984375],"temperature":1.0,"type":"choice","source_id":"3986","source_row_hash":"a52237c94225b1553a4b9b64b99a6de6c5747841054a315e869690c5c40c0042","input_tokens":51,"truncated":false,"calibrated_raw_logits":[6.13671875,-3.943359375,-2.3984375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-3986-noul","group_id":"massive-alarm-f2693fd957a1a13798ca","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"b8b2c739fdb3846f88484b30634ef774a8bf95921c0db9777d23d18c672698ad","option_order":["false","true"],"gold":"true","raw_logits":[-1.990234375,1.25390625],"temperature":1.0,"type":"noul","source_id":"3986","source_row_hash":"a52237c94225b1553a4b9b64b99a6de6c5747841054a315e869690c5c40c0042","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.990234375,1.25390625],"calibrated_temperature":5.0},
- {"id":"massive-399-choice","group_id":"massive-alarm-5064cd0ab19f1ced3b63","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7f7f324de0a5abcdfb147204c078ff9f5e6627709efd66a92c90f2f11eda84c8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.845703125,-2.98046875,2.666015625],"temperature":1.0,"type":"choice","source_id":"399","source_row_hash":"1c0573b1dc35d2d56f63d762ce3123e67d36e4fe7a90b405693688e9f3c0fb74","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-2.845703125,-2.98046875,2.666015625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-399-noul","group_id":"massive-alarm-5064cd0ab19f1ced3b63","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"3f557ad186f1a17d7172adf20667e35f775f5335dbae9db5de749c0a85692384","option_order":["false","true"],"gold":"false","raw_logits":[-1.7451171875,0.732421875],"temperature":1.0,"type":"noul","source_id":"399","source_row_hash":"1c0573b1dc35d2d56f63d762ce3123e67d36e4fe7a90b405693688e9f3c0fb74","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.7451171875,0.732421875],"calibrated_temperature":5.0},
- {"id":"massive-4017-choice","group_id":"massive-alarm-7bab44deabd44eeba1d4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c6aa025ee3f6243b63d4112e57528c1a0d09804b794a47a52f827cd845c22365","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-2.89453125,-3.54296875,1.318359375],"temperature":1.0,"type":"choice","source_id":"4017","source_row_hash":"b950ef4e3cfec6148b5f70b8cc2699e018e3443f68871089bb3b3bac77a915b3","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-2.89453125,-3.54296875,1.318359375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4017-noul","group_id":"massive-alarm-7bab44deabd44eeba1d4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"91e79a9efc33926738ab1aca86bf6652646cab200330672ca9fdd14ff4a13f95","option_order":["false","true"],"gold":"false","raw_logits":[-2.76953125,0.5654296875],"temperature":1.0,"type":"noul","source_id":"4017","source_row_hash":"b950ef4e3cfec6148b5f70b8cc2699e018e3443f68871089bb3b3bac77a915b3","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.76953125,0.5654296875],"calibrated_temperature":5.0},
- {"id":"massive-4081-choice","group_id":"massive-alarm-0fbfcc011e34434db270","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"96a5e44b26bf23a409304bc33c251790f9f8bebf6fc652e9b1b60e8941003ce4","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-1.23828125,-2.173828125,-0.37451171875],"temperature":1.0,"type":"choice","source_id":"4081","source_row_hash":"ed08495bac8110f45e50260d015d1d52615f67a00d5c47ffcd960745bd32f98f","input_tokens":60,"truncated":false,"calibrated_raw_logits":[-1.23828125,-2.173828125,-0.37451171875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4081-noul","group_id":"massive-alarm-0fbfcc011e34434db270","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"db72d9a98ad646aab2a2727af0b22a3dd95d9afb6b7053284680c34d1e22848b","option_order":["false","true"],"gold":"true","raw_logits":[-1.77734375,0.5439453125],"temperature":1.0,"type":"noul","source_id":"4081","source_row_hash":"ed08495bac8110f45e50260d015d1d52615f67a00d5c47ffcd960745bd32f98f","input_tokens":55,"truncated":false,"calibrated_raw_logits":[-1.77734375,0.5439453125],"calibrated_temperature":5.0},
- {"id":"massive-4121-choice","group_id":"massive-alarm-06d7dae0b6a5251a04b7","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"856f98d8094e873994e21182bb63f23d0513e3176c63e27cd351ac91cb60e697","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.19140625,-1.458984375,-1.919921875],"temperature":1.0,"type":"choice","source_id":"4121","source_row_hash":"905a75e6d0482805183ad0e81410418cc470ec3cb4cfabe1d76f9f30a337c809","input_tokens":52,"truncated":false,"calibrated_raw_logits":[4.19140625,-1.458984375,-1.919921875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4121-noul","group_id":"massive-alarm-06d7dae0b6a5251a04b7","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a09915d1ead2a4e4ef77bcfb9cfdc28ff092aab9317064470bf2d91fa130fcf9","option_order":["false","true"],"gold":"false","raw_logits":[-1.306640625,1.3212890625],"temperature":1.0,"type":"noul","source_id":"4121","source_row_hash":"905a75e6d0482805183ad0e81410418cc470ec3cb4cfabe1d76f9f30a337c809","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.306640625,1.3212890625],"calibrated_temperature":5.0},
- {"id":"massive-4130-choice","group_id":"massive-alarm-e36c4391c2490fb6ea69","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"9e20290617b2c369552819c32b78e1c50dadbb853aaa45e3fbaa353b5ac9a1e1","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.81005859375,-0.61181640625,-0.282958984375],"temperature":1.0,"type":"choice","source_id":"4130","source_row_hash":"45b0bb6e019be688e56cc7d5540bc815c6cc6845e6f17adfa152bee81fedec9a","input_tokens":55,"truncated":false,"calibrated_raw_logits":[0.81005859375,-0.61181640625,-0.282958984375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4130-noul","group_id":"massive-alarm-e36c4391c2490fb6ea69","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ca8193f9310a7214cfdea09f2f789252a72d64e3aa770830d4d71eb920257fc9","option_order":["false","true"],"gold":"false","raw_logits":[-0.470458984375,0.58203125],"temperature":1.0,"type":"noul","source_id":"4130","source_row_hash":"45b0bb6e019be688e56cc7d5540bc815c6cc6845e6f17adfa152bee81fedec9a","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-0.470458984375,0.58203125],"calibrated_temperature":5.0},
- {"id":"massive-4145-choice","group_id":"massive-alarm-193674ca7cd9463d8724","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"d3989f582d7f1cb6a535ec95163ad30df3e578d1b8610a327b515b423f69c862","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.52734375,-3.193359375,-2.767578125],"temperature":1.0,"type":"choice","source_id":"4145","source_row_hash":"4e06467ae4327b6843c2fd71dea717e128c9c5e776c173ba67f85c47f7caadb4","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.52734375,-3.193359375,-2.767578125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4145-noul","group_id":"massive-alarm-193674ca7cd9463d8724","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"eeda34b2c3c8364306cca7d520527d07f47fb1da66cbbb5627cff7d9a0127a78","option_order":["false","true"],"gold":"false","raw_logits":[-2.39453125,1.0224609375],"temperature":1.0,"type":"noul","source_id":"4145","source_row_hash":"4e06467ae4327b6843c2fd71dea717e128c9c5e776c173ba67f85c47f7caadb4","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.39453125,1.0224609375],"calibrated_temperature":5.0},
- {"id":"massive-4319-choice","group_id":"massive-alarm-3a6e1e04dfdb2826f3e6","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"59054d0457ed9b0e150ca8e4946d327901f4ca6957ecfd3ece87322ba3eac9a5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.406494140625,-2.6796875,-0.0236053466796875],"temperature":1.0,"type":"choice","source_id":"4319","source_row_hash":"f81d298a66e00e967f0e503432b25d08e7c32fae30fc39b9d68651d1c7e22915","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.406494140625,-2.6796875,-0.0236053466796875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4319-noul","group_id":"massive-alarm-3a6e1e04dfdb2826f3e6","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"aeeb6f93631dbebaa9ddf60f34c6b43d9dc7a176e9667f7b3efb94eb3d775fa2","option_order":["false","true"],"gold":"true","raw_logits":[-1.3876953125,0.8994140625],"temperature":1.0,"type":"noul","source_id":"4319","source_row_hash":"f81d298a66e00e967f0e503432b25d08e7c32fae30fc39b9d68651d1c7e22915","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.3876953125,0.8994140625],"calibrated_temperature":5.0},
- {"id":"massive-4375-choice","group_id":"massive-alarm-d0710f72b80d46f21bd2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"91da7137e905fecc5e2cdbd252323de33015b18aefbddd160effc10409a0a0ec","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.3505859375,0.5517578125,1.2265625],"temperature":1.0,"type":"choice","source_id":"4375","source_row_hash":"2d5824d9d29c077e250c5812fe3e84908d5bf837ce7bc903743164f8d11624dd","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.3505859375,0.5517578125,1.2265625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4375-noul","group_id":"massive-alarm-d0710f72b80d46f21bd2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fbbeae282b0e1b151a5b50a77f85d81ef6c08f60ec2811a7a2a773c2e789e581","option_order":["false","true"],"gold":"false","raw_logits":[-1.2119140625,0.95458984375],"temperature":1.0,"type":"noul","source_id":"4375","source_row_hash":"2d5824d9d29c077e250c5812fe3e84908d5bf837ce7bc903743164f8d11624dd","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.2119140625,0.95458984375],"calibrated_temperature":5.0},
- {"id":"massive-4654-choice","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"77cebc8da0a8189f2262bdd3a39f31d61fc360a7956bcd0f65463b08dc21bf4f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.2578125,-2.5625,-2.025390625],"temperature":1.0,"type":"choice","source_id":"4654","source_row_hash":"b7e67a9331d7f20bc014914c88b8be8ba51ce9bf981915ffe1d2138b91b6a01e","input_tokens":53,"truncated":false,"calibrated_raw_logits":[4.2578125,-2.5625,-2.025390625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4654-noul","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"456e9b4d2821b1b3662f84db099f4a9923f03141ad9b289aa4cb01e7b434fac1","option_order":["false","true"],"gold":"true","raw_logits":[-1.9423828125,1.6611328125],"temperature":1.0,"type":"noul","source_id":"4654","source_row_hash":"b7e67a9331d7f20bc014914c88b8be8ba51ce9bf981915ffe1d2138b91b6a01e","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.9423828125,1.6611328125],"calibrated_temperature":5.0},
- {"id":"massive-4673-choice","group_id":"massive-alarm-84fb06bfa21c395f5aaf","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"bbc99315926deca3de03ece3f5e4184df6aa7262d5442d04979bf9dbf2a8cbd9","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-2.296875,3.982421875,-0.8330078125],"temperature":1.0,"type":"choice","source_id":"4673","source_row_hash":"b9cefc1b3d178f9664ac742c0d5fd6c3541b41453ce33986bd45ded2e5baa606","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-2.296875,3.982421875,-0.8330078125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4673-noul","group_id":"massive-alarm-84fb06bfa21c395f5aaf","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"29dce85d99aa74e32a858b60eb0de1ff518b8819f65f192a048517fb9a9d798b","option_order":["false","true"],"gold":"false","raw_logits":[-1.3115234375,0.343994140625],"temperature":1.0,"type":"noul","source_id":"4673","source_row_hash":"b9cefc1b3d178f9664ac742c0d5fd6c3541b41453ce33986bd45ded2e5baa606","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.3115234375,0.343994140625],"calibrated_temperature":5.0},
- {"id":"massive-4712-choice","group_id":"massive-alarm-2c6ac9f498036d649d66","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"934f5e684a6cbf814f1536b5b6b017245af8638cf21ae0290e64f1a32bd58c4c","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.005859375,-3.384765625,-0.11474609375],"temperature":1.0,"type":"choice","source_id":"4712","source_row_hash":"4386c04043d8c28ace7ee145b5b749efc7ff6ad5efb0664e6ffb4a359864a2ba","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.005859375,-3.384765625,-0.11474609375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4712-noul","group_id":"massive-alarm-2c6ac9f498036d649d66","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"17794909495f150f505eb1cd6a46d9bcaa7531479f2f9d0165ee8b8193811d67","option_order":["false","true"],"gold":"false","raw_logits":[-1.8779296875,0.9365234375],"temperature":1.0,"type":"noul","source_id":"4712","source_row_hash":"4386c04043d8c28ace7ee145b5b749efc7ff6ad5efb0664e6ffb4a359864a2ba","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.8779296875,0.9365234375],"calibrated_temperature":5.0},
- {"id":"massive-4776-choice","group_id":"massive-alarm-44615da0c0274196323c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"5fec435b1670ec5cf5fb855e00e31fce4474fe0e0fc3105fcb24aeceaeaeb6b5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.238037109375,-1.462890625,-1.5185546875],"temperature":1.0,"type":"choice","source_id":"4776","source_row_hash":"81079ab4bf8ea6953a66c7c91e9a60e6fe22b3e698c003ffa5c411c0da2afe20","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.238037109375,-1.462890625,-1.5185546875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4776-noul","group_id":"massive-alarm-44615da0c0274196323c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"0fc3aba2af6c95f1c6f27d8149e96b826c80dd710f2427776cd09b5660c919cf","option_order":["false","true"],"gold":"false","raw_logits":[-1.685546875,1.1787109375],"temperature":1.0,"type":"noul","source_id":"4776","source_row_hash":"81079ab4bf8ea6953a66c7c91e9a60e6fe22b3e698c003ffa5c411c0da2afe20","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.685546875,1.1787109375],"calibrated_temperature":5.0},
- {"id":"massive-4848-choice","group_id":"massive-alarm-a708c3be70feb0cd7499","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7522ec4a0488ba7e56fa08b528c3df20f40223c6894ac0dc46d8bd184912bcb3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.416015625,-2.8828125,-0.599609375],"temperature":1.0,"type":"choice","source_id":"4848","source_row_hash":"b0b8d755300f645d4383b66847a7614fea92b600c73c2cf5542dff3eca2e27bd","input_tokens":50,"truncated":false,"calibrated_raw_logits":[0.416015625,-2.8828125,-0.599609375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4848-noul","group_id":"massive-alarm-a708c3be70feb0cd7499","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"956ba45262eca715bf5c90a06427ede7e3364a644ee12016e6da052866a0845b","option_order":["false","true"],"gold":"true","raw_logits":[-1.8486328125,0.95556640625],"temperature":1.0,"type":"noul","source_id":"4848","source_row_hash":"b0b8d755300f645d4383b66847a7614fea92b600c73c2cf5542dff3eca2e27bd","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.8486328125,0.95556640625],"calibrated_temperature":5.0},
- {"id":"massive-4851-choice","group_id":"massive-alarm-0fe1a1db9c374422445f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"fdeb0308d07e167e345d0809a6395c4969a801c33d5f80ca07ffed816020bbcb","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[4.86328125,-3.0859375,-2.537109375],"temperature":1.0,"type":"choice","source_id":"4851","source_row_hash":"9fe5af17055439dfa1b1610ca5fb2973b66459a1e0614af0e2452d2ce7a1e77c","input_tokens":50,"truncated":false,"calibrated_raw_logits":[4.86328125,-3.0859375,-2.537109375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4851-noul","group_id":"massive-alarm-0fe1a1db9c374422445f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"133b6950426a63d754fc346a4d151cbafe939b8e37e37fc290c101f6dcc77a0f","option_order":["false","true"],"gold":"false","raw_logits":[-2.02734375,1.529296875],"temperature":1.0,"type":"noul","source_id":"4851","source_row_hash":"9fe5af17055439dfa1b1610ca5fb2973b66459a1e0614af0e2452d2ce7a1e77c","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.02734375,1.529296875],"calibrated_temperature":5.0},
- {"id":"massive-4921-choice","group_id":"massive-alarm-5c308695c6a8b9f486cd","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"72d43a129cdaffcad2fbed561838f632af3ac96bca5743580ba0401a3665095e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.8505859375,-3.353515625,1.2998046875],"temperature":1.0,"type":"choice","source_id":"4921","source_row_hash":"2c2b240390fb93b6986489979cea8c3377a2c30ff81aab5abde369bc1a921d66","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.8505859375,-3.353515625,1.2998046875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4921-noul","group_id":"massive-alarm-5c308695c6a8b9f486cd","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4437a78e7f0d9e73db14372aa4f2b1cb1ee2f75571651b05dcb50bef8dac6af6","option_order":["false","true"],"gold":"false","raw_logits":[-2.416015625,0.92822265625],"temperature":1.0,"type":"noul","source_id":"4921","source_row_hash":"2c2b240390fb93b6986489979cea8c3377a2c30ff81aab5abde369bc1a921d66","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.416015625,0.92822265625],"calibrated_temperature":5.0},
- {"id":"massive-4930-choice","group_id":"massive-alarm-54d0fa25a96d8072f5b5","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cb8fffe4d7f15dd6507c6a0d7fa59fd180ba95cd1fb191f63aa8c1fe6aaf1b3b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.9716796875,-3.0546875,-0.66943359375],"temperature":1.0,"type":"choice","source_id":"4930","source_row_hash":"fd5035b32bf0e5c2a53512a0663af50295468117efba589c8277e039fbaace04","input_tokens":54,"truncated":false,"calibrated_raw_logits":[1.9716796875,-3.0546875,-0.66943359375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-4930-noul","group_id":"massive-alarm-54d0fa25a96d8072f5b5","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"5f559d4cf48d8ed51abf809f544bc6ed69e826262d2c3aa6a1980713386add37","option_order":["false","true"],"gold":"true","raw_logits":[-2.16796875,1.2373046875],"temperature":1.0,"type":"noul","source_id":"4930","source_row_hash":"fd5035b32bf0e5c2a53512a0663af50295468117efba589c8277e039fbaace04","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.16796875,1.2373046875],"calibrated_temperature":5.0},
- {"id":"massive-5006-choice","group_id":"massive-alarm-36c1ee0067291e81dfe0","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"c0cf2686b16917d7ab80250b7f585134b8169f9fdc84e27e9c480875e60f94a2","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.9462890625,1.2626953125,-2.009765625],"temperature":1.0,"type":"choice","source_id":"5006","source_row_hash":"c23192b2816c3f5f07c27870d69fdf383d4bc79de8202365cce80d52d65ef321","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.9462890625,1.2626953125,-2.009765625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5006-noul","group_id":"massive-alarm-36c1ee0067291e81dfe0","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4a7e732be2948711919ca7d7eafa8af5c50381df15405ffb995261ae054c1fd5","option_order":["false","true"],"gold":"false","raw_logits":[-1.046875,0.1993408203125],"temperature":1.0,"type":"noul","source_id":"5006","source_row_hash":"c23192b2816c3f5f07c27870d69fdf383d4bc79de8202365cce80d52d65ef321","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.046875,0.1993408203125],"calibrated_temperature":5.0},
- {"id":"massive-5043-choice","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"9209f20ab1cba0e22314b3adfb3a5459bc888eb018e2160a81d37579238baa7a","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.076171875,-1.1318359375,-0.78173828125],"temperature":1.0,"type":"choice","source_id":"5043","source_row_hash":"837ce3dfa54a2caaa3feddb1add4699979a13624fa240aedd9d48eb841d7c3e3","input_tokens":53,"truncated":false,"calibrated_raw_logits":[3.076171875,-1.1318359375,-0.78173828125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5043-noul","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"24c1573dfcf63931fc5ed541e2ad4eb9e2eac99472bee9d3f85e0b3c80b7ee59","option_order":["false","true"],"gold":"true","raw_logits":[-2.505859375,1.9638671875],"temperature":1.0,"type":"noul","source_id":"5043","source_row_hash":"837ce3dfa54a2caaa3feddb1add4699979a13624fa240aedd9d48eb841d7c3e3","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-2.505859375,1.9638671875],"calibrated_temperature":5.0},
- {"id":"massive-5051-choice","group_id":"massive-alarm-16e090610da060f7511f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"e5fbc20b0a5aad3f0006a418d26781efe4e3d935dd3ae44b4fcc7219d7760b64","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.53125,-3.443359375,-0.4462890625],"temperature":1.0,"type":"choice","source_id":"5051","source_row_hash":"f3bbc1f439396e98e81c6c680dcba8b9fbe57a12e24a67d449622e4042bb9671","input_tokens":50,"truncated":false,"calibrated_raw_logits":[0.53125,-3.443359375,-0.4462890625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5051-noul","group_id":"massive-alarm-16e090610da060f7511f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"426104e055a8c3026abeb0f10d97c3466a4cfacf4037b56e3596670dc21c7368","option_order":["false","true"],"gold":"true","raw_logits":[-2.826171875,1.2958984375],"temperature":1.0,"type":"noul","source_id":"5051","source_row_hash":"f3bbc1f439396e98e81c6c680dcba8b9fbe57a12e24a67d449622e4042bb9671","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.826171875,1.2958984375],"calibrated_temperature":5.0},
- {"id":"massive-508-choice","group_id":"massive-alarm-41a421223f2457d4ae35","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2fc92d0e251aa21634ffc2b072bab5a2ca2442251f5e82ccf62197d9b4d9194d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[3.478515625,-3.337890625,-3.78515625],"temperature":1.0,"type":"choice","source_id":"508","source_row_hash":"71ecc0e574fd987cffcdeee6026d2de621df0225d0e6be68cf09db826a9bcd9b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[3.478515625,-3.337890625,-3.78515625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-508-noul","group_id":"massive-alarm-41a421223f2457d4ae35","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"61946bd8d19af979160d933e5b004a05b212d1f720b0866c522637f99f21ea7b","option_order":["false","true"],"gold":"false","raw_logits":[-2.60546875,0.544921875],"temperature":1.0,"type":"noul","source_id":"508","source_row_hash":"71ecc0e574fd987cffcdeee6026d2de621df0225d0e6be68cf09db826a9bcd9b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.60546875,0.544921875],"calibrated_temperature":5.0},
- {"id":"massive-5082-choice","group_id":"massive-alarm-dff7933fbae6d32c0fc9","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"65e59f1014104082d5ba399967ee15984ae0fa87f363cd154ec0f46b5075265f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.66650390625,-3.208984375,0.7880859375],"temperature":1.0,"type":"choice","source_id":"5082","source_row_hash":"eb3b740e165f04e6c1419581c82a67b0e075496a1bb875d944161266c0f98fd6","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.66650390625,-3.208984375,0.7880859375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5082-noul","group_id":"massive-alarm-dff7933fbae6d32c0fc9","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4d65d5fa341600065d5f523f2936bd98d886c1b3ddd05ef2f97bca7116d2413b","option_order":["false","true"],"gold":"true","raw_logits":[-2.341796875,1.470703125],"temperature":1.0,"type":"noul","source_id":"5082","source_row_hash":"eb3b740e165f04e6c1419581c82a67b0e075496a1bb875d944161266c0f98fd6","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.341796875,1.470703125],"calibrated_temperature":5.0},
- {"id":"massive-5103-choice","group_id":"massive-alarm-a0f523ed66d86dd66928","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"1474862757e271b75a651b0a5c023ae431e3471fbd2b83e930cc010904b7f605","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.208984375,-4.21484375,2.35546875],"temperature":1.0,"type":"choice","source_id":"5103","source_row_hash":"5e4324ac53ee00213541d5386e0f8f520d4b7930fe26bd416545ed9b3681e81c","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-3.208984375,-4.21484375,2.35546875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5103-noul","group_id":"massive-alarm-a0f523ed66d86dd66928","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7cc1683f408cddf9993f59351a691ac692dcd2f2be18820e5d088f5d607d95e5","option_order":["false","true"],"gold":"false","raw_logits":[-1.99609375,-0.01267242431640625],"temperature":1.0,"type":"noul","source_id":"5103","source_row_hash":"5e4324ac53ee00213541d5386e0f8f520d4b7930fe26bd416545ed9b3681e81c","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.99609375,-0.01267242431640625],"calibrated_temperature":5.0},
- {"id":"massive-5116-choice","group_id":"massive-alarm-eef8eb3baf2d9be9f032","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ff5afaf8010c0ec76afa02463532a835bf618957bb435b423b572a34d1f72b9b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.78125,-1.4833984375,2.59765625],"temperature":1.0,"type":"choice","source_id":"5116","source_row_hash":"81e95101c290ce21984ebbc4b02d5abbd6cc8ca4e784b97182671444871de55b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.78125,-1.4833984375,2.59765625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5116-noul","group_id":"massive-alarm-eef8eb3baf2d9be9f032","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e55857f2a894f6be93794ed4131d1b24b3d1e25bd0fb009c6bed1068297afa36","option_order":["false","true"],"gold":"false","raw_logits":[-1.5234375,-0.00075531005859375],"temperature":1.0,"type":"noul","source_id":"5116","source_row_hash":"81e95101c290ce21984ebbc4b02d5abbd6cc8ca4e784b97182671444871de55b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.5234375,-0.00075531005859375],"calibrated_temperature":5.0},
- {"id":"massive-5122-choice","group_id":"massive-alarm-a7c3622f745708add50b","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"b8cecf6e8d365c8cca3910552fdf85dce9653479d4d50a00d9437b329657fca3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.7109375,-2.01953125,-0.6689453125],"temperature":1.0,"type":"choice","source_id":"5122","source_row_hash":"60429c8fba1d01a9b236a868780f5aab91fdfdfd7d7193d219e29dda4ff40f7b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.7109375,-2.01953125,-0.6689453125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5122-noul","group_id":"massive-alarm-a7c3622f745708add50b","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"3c3cccaa225b3031122a45bab64f407013dfce7331773fbb68584397b853f187","option_order":["false","true"],"gold":"false","raw_logits":[-1.8408203125,0.66845703125],"temperature":1.0,"type":"noul","source_id":"5122","source_row_hash":"60429c8fba1d01a9b236a868780f5aab91fdfdfd7d7193d219e29dda4ff40f7b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.8408203125,0.66845703125],"calibrated_temperature":5.0},
- {"id":"massive-5292-choice","group_id":"massive-alarm-7479d40399b64ecb695e","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6b263f0198363465a79d92000b9f4549cac9e8fdec4c19381966d4af03853c2e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.376953125,-4.47265625,-4.5078125],"temperature":1.0,"type":"choice","source_id":"5292","source_row_hash":"813c05e407bea752a81f88e33c115e1e14729209d2ef4bb63bb371d3d394049c","input_tokens":50,"truncated":false,"calibrated_raw_logits":[3.376953125,-4.47265625,-4.5078125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5292-noul","group_id":"massive-alarm-7479d40399b64ecb695e","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"2bae5052bc8cf5c358997f4b0612f1f6d00852a3884983f2e3b12c9cacfa0e9d","option_order":["false","true"],"gold":"true","raw_logits":[-3.390625,1.625],"temperature":1.0,"type":"noul","source_id":"5292","source_row_hash":"813c05e407bea752a81f88e33c115e1e14729209d2ef4bb63bb371d3d394049c","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-3.390625,1.625],"calibrated_temperature":5.0},
- {"id":"massive-533-choice","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"99a85134b83e2e949a3e0c54f4e95c4e40f8bf0a6b5f395df71cd8a3e029bdec","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.0703125,-4.25,-3.58203125],"temperature":1.0,"type":"choice","source_id":"533","source_row_hash":"eca3b5ac906fb8a35343d18eb18cad818f1db7770f69dcac2648292b16b0ffbc","input_tokens":52,"truncated":false,"calibrated_raw_logits":[6.0703125,-4.25,-3.58203125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-533-noul","group_id":"massive-alarm-12048055efd70a8d768c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e0c12ffe2cd33b8d0a94ddaa9dab71028665b02ed35d0295366709686a915d67","option_order":["false","true"],"gold":"true","raw_logits":[-2.1328125,1.4267578125],"temperature":1.0,"type":"noul","source_id":"533","source_row_hash":"eca3b5ac906fb8a35343d18eb18cad818f1db7770f69dcac2648292b16b0ffbc","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.1328125,1.4267578125],"calibrated_temperature":5.0},
- {"id":"massive-5365-choice","group_id":"massive-alarm-d0dbaab04348ee02b2e4","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7dae85cf680af0945a163fff2386a84f8d4b0fe67d9c437a5527258473b91288","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.703125,-3.419921875,-1.2119140625],"temperature":1.0,"type":"choice","source_id":"5365","source_row_hash":"3e646d241a20c6d18260b38f64e3d5b4045894baca11ab213b09d65fc04fd4d5","input_tokens":53,"truncated":false,"calibrated_raw_logits":[4.703125,-3.419921875,-1.2119140625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5365-noul","group_id":"massive-alarm-d0dbaab04348ee02b2e4","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a3d43d9d9cfd2a79d27de7f606676fe8fd10e9ede4c75b655407327e83fe27ab","option_order":["false","true"],"gold":"true","raw_logits":[-1.537109375,1.2392578125],"temperature":1.0,"type":"noul","source_id":"5365","source_row_hash":"3e646d241a20c6d18260b38f64e3d5b4045894baca11ab213b09d65fc04fd4d5","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.537109375,1.2392578125],"calibrated_temperature":5.0},
- {"id":"massive-5475-choice","group_id":"massive-alarm-e7a25c599f07a27b07a8","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6ce27768ae10f0206a17038e7f7e3bd69bc0584f9ed44947f46347b4423df338","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.0107421875,-2.423828125,0.060760498046875],"temperature":1.0,"type":"choice","source_id":"5475","source_row_hash":"dc6c0165e59d0b35c1731ad73656915fbd6291deba80299bc358fc944a72978c","input_tokens":52,"truncated":false,"calibrated_raw_logits":[1.0107421875,-2.423828125,0.060760498046875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5475-noul","group_id":"massive-alarm-e7a25c599f07a27b07a8","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"264847e9ef6d6c6f317ea9dea8f6f8a17935af1b5fc265f2802d522dbf5af17d","option_order":["false","true"],"gold":"true","raw_logits":[-2.3203125,1.548828125],"temperature":1.0,"type":"noul","source_id":"5475","source_row_hash":"dc6c0165e59d0b35c1731ad73656915fbd6291deba80299bc358fc944a72978c","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-2.3203125,1.548828125],"calibrated_temperature":5.0},
- {"id":"massive-5568-choice","group_id":"massive-alarm-9d8d8e3f6fb99e20fed8","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8af7c9ac22d839179599a532150c0575129bb3db346ed4fcaf23996f2727c981","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[3.759765625,-1.7978515625,-1.2451171875],"temperature":1.0,"type":"choice","source_id":"5568","source_row_hash":"e2678a9b8f76e5325e37853048a1af7ef56128900590959c918238a2a0a87f67","input_tokens":54,"truncated":false,"calibrated_raw_logits":[3.759765625,-1.7978515625,-1.2451171875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5568-noul","group_id":"massive-alarm-9d8d8e3f6fb99e20fed8","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"6bb9d7bc2c8f0080617c2a6a730f73b45ea25f3c7a0e4fafdef8dbbfc6067d91","option_order":["false","true"],"gold":"true","raw_logits":[-2.484375,1.4775390625],"temperature":1.0,"type":"noul","source_id":"5568","source_row_hash":"e2678a9b8f76e5325e37853048a1af7ef56128900590959c918238a2a0a87f67","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.484375,1.4775390625],"calibrated_temperature":5.0},
- {"id":"massive-560-choice","group_id":"massive-alarm-3914f2c167cd3bc916a9","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"8b4bacc83095f32adfbe08922536f8e8f085e1c778c9ed57ed2a65ce83ee6ff5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-1.1005859375,-3.044921875,1.4658203125],"temperature":1.0,"type":"choice","source_id":"560","source_row_hash":"198c7a67f853f0a20207e974e8b9f5c52eb85e496988929af5c4cd1b2ba14b7a","input_tokens":50,"truncated":false,"calibrated_raw_logits":[-1.1005859375,-3.044921875,1.4658203125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-560-noul","group_id":"massive-alarm-3914f2c167cd3bc916a9","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"de516021eb9a4d76b704b912aab4923d0355631dc535f055c7f97319615c832e","option_order":["false","true"],"gold":"false","raw_logits":[-2.03515625,-0.5439453125],"temperature":1.0,"type":"noul","source_id":"560","source_row_hash":"198c7a67f853f0a20207e974e8b9f5c52eb85e496988929af5c4cd1b2ba14b7a","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.03515625,-0.5439453125],"calibrated_temperature":5.0},
- {"id":"massive-5623-choice","group_id":"massive-alarm-ee24ef5af4d9c0920de7","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"2a8568251fdcde56fbf31c4ee89e239a704fee954543e6ef90286e089bda6c6f","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.417236328125,-1.1318359375,0.07928466796875],"temperature":1.0,"type":"choice","source_id":"5623","source_row_hash":"d74f814f601c02c5df4a0b5f23516c6bf156ad8051a476422eb6fdea1660ae5e","input_tokens":50,"truncated":false,"calibrated_raw_logits":[0.417236328125,-1.1318359375,0.07928466796875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5623-noul","group_id":"massive-alarm-ee24ef5af4d9c0920de7","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"6af842101731694a1cd8e31608ddc4a5a0e95ab625ae4ebe22e246073a8f6700","option_order":["false","true"],"gold":"true","raw_logits":[-1.201171875,1.654296875],"temperature":1.0,"type":"noul","source_id":"5623","source_row_hash":"d74f814f601c02c5df4a0b5f23516c6bf156ad8051a476422eb6fdea1660ae5e","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-1.201171875,1.654296875],"calibrated_temperature":5.0},
- {"id":"massive-5635-choice","group_id":"massive-alarm-4beeb5931613d60f1341","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cc5c973ab31e3d3df68e2c8c5178bc62cccae85d6a38bef00637b677f9b3e2e0","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.031829833984375,-2.861328125,-0.1290283203125],"temperature":1.0,"type":"choice","source_id":"5635","source_row_hash":"40eb075b4d858f1411258909d4d77625f091cbc21c25f9c1484232129528cb67","input_tokens":52,"truncated":false,"calibrated_raw_logits":[0.031829833984375,-2.861328125,-0.1290283203125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5635-noul","group_id":"massive-alarm-4beeb5931613d60f1341","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"91e98a9d1309dfdefc5644e876e4141869104d2d08ee388a23a75d4dd1689a6a","option_order":["false","true"],"gold":"true","raw_logits":[-1.052734375,1.1396484375],"temperature":1.0,"type":"noul","source_id":"5635","source_row_hash":"40eb075b4d858f1411258909d4d77625f091cbc21c25f9c1484232129528cb67","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.052734375,1.1396484375],"calibrated_temperature":5.0},
- {"id":"massive-59-choice","group_id":"massive-alarm-2276a9a439ac95d3573f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"89b470c0d2ae0a5f39585af5b6ff60255952f2dd6dc6ec3e4a84b0ae006b2ecc","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-0.04351806640625,-2.509765625,1.7255859375],"temperature":1.0,"type":"choice","source_id":"59","source_row_hash":"cb6899117f503f233e7b525dd5cd06511f88ace9a9caaee665d48c413113a6d9","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.04351806640625,-2.509765625,1.7255859375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-59-noul","group_id":"massive-alarm-2276a9a439ac95d3573f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"872c7a88b6ca28b3a9194a3e4678c850b38ef642762b3fd454545679f04001f5","option_order":["false","true"],"gold":"false","raw_logits":[-1.94140625,0.13916015625],"temperature":1.0,"type":"noul","source_id":"59","source_row_hash":"cb6899117f503f233e7b525dd5cd06511f88ace9a9caaee665d48c413113a6d9","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.94140625,0.13916015625],"calibrated_temperature":5.0},
- {"id":"massive-5902-choice","group_id":"massive-alarm-fb6df7f77542683a3ff2","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"7fbcd8af2c716bdd66961270eacaa6a96622bbcf1372261fd4a96f855553ec8d","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.564453125,-0.93017578125,0.293212890625],"temperature":1.0,"type":"choice","source_id":"5902","source_row_hash":"d1fd61eb591646e537e465ded07ee0a48eeb5db31db81e0c4407cac197d93495","input_tokens":53,"truncated":false,"calibrated_raw_logits":[-0.564453125,-0.93017578125,0.293212890625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5902-noul","group_id":"massive-alarm-fb6df7f77542683a3ff2","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"c4be7c11de56ddc45a0601df7c80095583e9c5a7b51a97188443ff20f73e590e","option_order":["false","true"],"gold":"false","raw_logits":[-1.3876953125,1.02734375],"temperature":1.0,"type":"noul","source_id":"5902","source_row_hash":"d1fd61eb591646e537e465ded07ee0a48eeb5db31db81e0c4407cac197d93495","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.3876953125,1.02734375],"calibrated_temperature":5.0},
- {"id":"massive-5974-choice","group_id":"massive-alarm-f7cd5de4d0683af203ce","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"e7e220e3f9639398f9c6e56b8f38093b2018ad88a562b3f898609eacfb615f75","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[6.62890625,-4.55078125,-3.935546875],"temperature":1.0,"type":"choice","source_id":"5974","source_row_hash":"b4c1084215b35f9d593d4300ec53920326c4f994cb5bbe678cbc274e8068d703","input_tokens":54,"truncated":false,"calibrated_raw_logits":[6.62890625,-4.55078125,-3.935546875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-5974-noul","group_id":"massive-alarm-f7cd5de4d0683af203ce","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"964d28a1e6cbe408f1d74d44572fe81dc51b8fac4e3784c40464b96c34439c4d","option_order":["false","true"],"gold":"false","raw_logits":[-2.146484375,1.2470703125],"temperature":1.0,"type":"noul","source_id":"5974","source_row_hash":"b4c1084215b35f9d593d4300ec53920326c4f994cb5bbe678cbc274e8068d703","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.146484375,1.2470703125],"calibrated_temperature":5.0},
- {"id":"massive-6083-choice","group_id":"massive-alarm-950ec71310e9ece1aa7a","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"a11eee2509927e63186f13c256d44d20ed64932038a23b4a0806cab08a89405b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.064453125,-1.720703125,-0.33642578125],"temperature":1.0,"type":"choice","source_id":"6083","source_row_hash":"c98e1b18be265f4f3518a3d4f075393c40074e8f1aae799dbcefe012b867cd89","input_tokens":53,"truncated":false,"calibrated_raw_logits":[1.064453125,-1.720703125,-0.33642578125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-6083-noul","group_id":"massive-alarm-950ec71310e9ece1aa7a","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"5a396304607490821ff2e74144e5202da5c16f6aa3b19624edec62d3cbc0496a","option_order":["false","true"],"gold":"true","raw_logits":[-1.982421875,1.3046875],"temperature":1.0,"type":"noul","source_id":"6083","source_row_hash":"c98e1b18be265f4f3518a3d4f075393c40074e8f1aae799dbcefe012b867cd89","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.982421875,1.3046875],"calibrated_temperature":5.0},
- {"id":"massive-6101-choice","group_id":"massive-alarm-a63b13c9add3acdf4c9f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"06d76dcf51d497e0590fe5b28fa322168bfb9dce9eb88440c989d8a502850fc5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-0.08184814453125,-1.6630859375,-0.9814453125],"temperature":1.0,"type":"choice","source_id":"6101","source_row_hash":"3dce7763719f2037e2c03a6492336b70e796b37c6eec38a0243a9e146c04f22e","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-0.08184814453125,-1.6630859375,-0.9814453125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-6101-noul","group_id":"massive-alarm-a63b13c9add3acdf4c9f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"2eccf49f3d9075b4515ed86d93c2ae48a73534099b204323ff87e723347cbfb7","option_order":["false","true"],"gold":"false","raw_logits":[-0.765625,1.0244140625],"temperature":1.0,"type":"noul","source_id":"6101","source_row_hash":"3dce7763719f2037e2c03a6492336b70e796b37c6eec38a0243a9e146c04f22e","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-0.765625,1.0244140625],"calibrated_temperature":5.0},
- {"id":"massive-612-choice","group_id":"massive-alarm-6d2b063ac90aec6042cb","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"df2193b6cb5257904a68c032afc1bbe241c266f2fd966aff1d5f2543ef419e58","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[0.40576171875,-2.451171875,-0.64404296875],"temperature":1.0,"type":"choice","source_id":"612","source_row_hash":"437786bbb8397c686ef9c55a8e67c91d16d8087753ecc83dccbe5e64726224a6","input_tokens":57,"truncated":false,"calibrated_raw_logits":[0.40576171875,-2.451171875,-0.64404296875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-612-noul","group_id":"massive-alarm-6d2b063ac90aec6042cb","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"aa455667dd535b15199206b1efc70f5485bcda3c2b32ff04ff3fb4a6bcb5437c","option_order":["false","true"],"gold":"true","raw_logits":[-0.074951171875,1.0908203125],"temperature":1.0,"type":"noul","source_id":"612","source_row_hash":"437786bbb8397c686ef9c55a8e67c91d16d8087753ecc83dccbe5e64726224a6","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-0.074951171875,1.0908203125],"calibrated_temperature":5.0},
- {"id":"massive-6162-choice","group_id":"massive-alarm-7c67db34bf4904167369","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"6c494191c98e1d84f5a75d1fe41fc46f153edf386c185a7e596bf70f25d2bf15","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.43359375,-3.333984375,2.044921875],"temperature":1.0,"type":"choice","source_id":"6162","source_row_hash":"9476a44d1260488e881b3dc20d371d653c5e2f622f312a564be0ff9faebdc44a","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-3.43359375,-3.333984375,2.044921875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-6162-noul","group_id":"massive-alarm-7c67db34bf4904167369","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"7078e785d1141162ba7cc4503f84962ca97b9c6fdb845b23e0fe12d393519eeb","option_order":["false","true"],"gold":"false","raw_logits":[-1.7138671875,0.31005859375],"temperature":1.0,"type":"noul","source_id":"6162","source_row_hash":"9476a44d1260488e881b3dc20d371d653c5e2f622f312a564be0ff9faebdc44a","input_tokens":43,"truncated":false,"calibrated_raw_logits":[-1.7138671875,0.31005859375],"calibrated_temperature":5.0},
- {"id":"massive-63-choice","group_id":"massive-alarm-3f054a8e6d320a808954","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"b27424721301b616f34476c934b889aeef8fe2d25823afb618470a06db45d9cd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[-1.4794921875,3.935546875,-1.7333984375],"temperature":1.0,"type":"choice","source_id":"63","source_row_hash":"f9f143eecf685ee8dcf485e26ca404fb60e1e6244ad6133efb0ff64c6889fc7b","input_tokens":51,"truncated":false,"calibrated_raw_logits":[-1.4794921875,3.935546875,-1.7333984375],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-63-noul","group_id":"massive-alarm-3f054a8e6d320a808954","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"13c4f2f436a4a1cad7db316725fa8870b5829f21dd47c292beb942e925bd11d2","option_order":["false","true"],"gold":"false","raw_logits":[-0.765625,0.442626953125],"temperature":1.0,"type":"noul","source_id":"63","source_row_hash":"f9f143eecf685ee8dcf485e26ca404fb60e1e6244ad6133efb0ff64c6889fc7b","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-0.765625,0.442626953125],"calibrated_temperature":5.0},
- {"id":"massive-796-choice","group_id":"massive-alarm-1b94ce3d96f2d223de77","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"01608bf1f2ba0a3fe29f6ae16e61c8ee20458e453613bf297000ffae65be5d0b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[0.71142578125,-2.318359375,-2.154296875],"temperature":1.0,"type":"choice","source_id":"796","source_row_hash":"c0c137c04060af51a4172b5b368190deb3009c12c64fea91b2be12f7b694f28f","input_tokens":51,"truncated":false,"calibrated_raw_logits":[0.71142578125,-2.318359375,-2.154296875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-796-noul","group_id":"massive-alarm-1b94ce3d96f2d223de77","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e658950e96a415d7a90f36a0e8dbb02edf10a90e0ec6b9bf9b85672831d343d4","option_order":["false","true"],"gold":"false","raw_logits":[-1.1796875,0.55810546875],"temperature":1.0,"type":"noul","source_id":"796","source_row_hash":"c0c137c04060af51a4172b5b368190deb3009c12c64fea91b2be12f7b694f28f","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-1.1796875,0.55810546875],"calibrated_temperature":5.0},
- {"id":"massive-861-choice","group_id":"massive-alarm-105aeeed3519053d3f55","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"adb310a263a979b824132b06ef4ca2fd6f40f379cafa5c0583d2893422e48ffe","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_query","raw_logits":[5.2890625,-3.55078125,-2.4453125],"temperature":1.0,"type":"choice","source_id":"861","source_row_hash":"7c204b7b739698b3a6f70ee107ef52e531a47e8d7b51ed6b839e3583bc83351a","input_tokens":50,"truncated":false,"calibrated_raw_logits":[5.2890625,-3.55078125,-2.4453125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-861-noul","group_id":"massive-alarm-105aeeed3519053d3f55","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"a55bf3c906f2bba70c4aa1a2f07dc2ea87f35341ef6fe8be3d694dbabe98589e","option_order":["false","true"],"gold":"false","raw_logits":[-2.2734375,1.072265625],"temperature":1.0,"type":"noul","source_id":"861","source_row_hash":"7c204b7b739698b3a6f70ee107ef52e531a47e8d7b51ed6b839e3583bc83351a","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.2734375,1.072265625],"calibrated_temperature":5.0},
- {"id":"massive-8731-choice","group_id":"massive-alarm-c773d4135b5f98edcb5c","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"e992f3c9dd54dfa9e1f81ed8dea16513e434dff0114a353de45a25822f20fc4b","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[-0.36376953125,-4.3046875,-1.541015625],"temperature":1.0,"type":"choice","source_id":"8731","source_row_hash":"37b09493cdc06a1bc88cc3643150ae9782b3ed10eea61dba2cb9c6519f20d4a1","input_tokens":59,"truncated":false,"calibrated_raw_logits":[-0.36376953125,-4.3046875,-1.541015625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-8731-noul","group_id":"massive-alarm-c773d4135b5f98edcb5c","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"ad5e11d3a7f8e16dc300f0f163db1e89309331376cd7149fc62d155a16de6d35","option_order":["false","true"],"gold":"true","raw_logits":[-2.794921875,0.283203125],"temperature":1.0,"type":"noul","source_id":"8731","source_row_hash":"37b09493cdc06a1bc88cc3643150ae9782b3ed10eea61dba2cb9c6519f20d4a1","input_tokens":54,"truncated":false,"calibrated_raw_logits":[-2.794921875,0.283203125],"calibrated_temperature":5.0},
- {"id":"massive-8741-choice","group_id":"massive-alarm-79a03eb5dc7bab70336f","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"12f704d636ca16e99b76ca5cea7307740f6125027d09a9c89c544fe86c2342f3","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[6.1640625,-4.1875,-3.794921875],"temperature":1.0,"type":"choice","source_id":"8741","source_row_hash":"062a50b981a7bf96310a69f314e1b5e9e03ca5f3c1ff474bf49fbf780ad21844","input_tokens":52,"truncated":false,"calibrated_raw_logits":[6.1640625,-4.1875,-3.794921875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-8741-noul","group_id":"massive-alarm-79a03eb5dc7bab70336f","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"fb565c561562c1581b7292d775cfc16f8ebab4e949c1ac31e7d00f377e4d9834","option_order":["false","true"],"gold":"true","raw_logits":[-1.9677734375,1.234375],"temperature":1.0,"type":"noul","source_id":"8741","source_row_hash":"062a50b981a7bf96310a69f314e1b5e9e03ca5f3c1ff474bf49fbf780ad21844","input_tokens":47,"truncated":false,"calibrated_raw_logits":[-1.9677734375,1.234375],"calibrated_temperature":5.0},
- {"id":"massive-884-choice","group_id":"massive-alarm-42f057d6c35ee3122781","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"92579ff62a0442cf6f564f0ed96f983bec83f3f69dfe99d3ce8bb1259c1ac9d5","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[4.1171875,-1.9990234375,-1.5869140625],"temperature":1.0,"type":"choice","source_id":"884","source_row_hash":"55b0f00bd02e71e600e190c2bc9e46f2a1d354fe59c818a9cb43fd1332486182","input_tokens":51,"truncated":false,"calibrated_raw_logits":[4.1171875,-1.9990234375,-1.5869140625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-884-noul","group_id":"massive-alarm-42f057d6c35ee3122781","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4ab83fec7f87887afa7cfc2310e2c6a492f625fe0400fac7bc6ed4813067f280","option_order":["false","true"],"gold":"true","raw_logits":[-2.265625,1.5087890625],"temperature":1.0,"type":"noul","source_id":"884","source_row_hash":"55b0f00bd02e71e600e190c2bc9e46f2a1d354fe59c818a9cb43fd1332486182","input_tokens":46,"truncated":false,"calibrated_raw_logits":[-2.265625,1.5087890625],"calibrated_temperature":5.0},
- {"id":"massive-8867-choice","group_id":"massive-alarm-bfe246fb1d4b99a3f48a","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"cbde715e981d6e934a2aac954045f8eae203ebf02c68432c7343a6bf8c3cc940","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.603515625,-3.880859375,-2.998046875],"temperature":1.0,"type":"choice","source_id":"8867","source_row_hash":"a513b328db26ac85863558c72614b0034a2f8177453c5d71df24760e4e766cd7","input_tokens":53,"truncated":false,"calibrated_raw_logits":[2.603515625,-3.880859375,-2.998046875],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-8867-noul","group_id":"massive-alarm-bfe246fb1d4b99a3f48a","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"e2b236cc79afa43b3fc316499b938cc5aadf2150ff7a433da58df64c902ca760","option_order":["false","true"],"gold":"true","raw_logits":[-1.4541015625,0.333984375],"temperature":1.0,"type":"noul","source_id":"8867","source_row_hash":"a513b328db26ac85863558c72614b0034a2f8177453c5d71df24760e4e766cd7","input_tokens":48,"truncated":false,"calibrated_raw_logits":[-1.4541015625,0.333984375],"calibrated_temperature":5.0},
- {"id":"massive-8882-choice","group_id":"massive-alarm-84c38cd9104789d07c40","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"ff0063357703eca9276281b428c4dda0916cc9e39a13740943d5e2776399f15e","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[2.3515625,-2.701171875,-1.8369140625],"temperature":1.0,"type":"choice","source_id":"8882","source_row_hash":"e1c5f172f9f02cade5685772af79ebded1b74b376026d93341832931a32676f9","input_tokens":57,"truncated":false,"calibrated_raw_logits":[2.3515625,-2.701171875,-1.8369140625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-8882-noul","group_id":"massive-alarm-84c38cd9104789d07c40","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"94a0ac44c2af3923c53e676b3d98f5ee321b0cc3a3a31c051d034ec2c89445ce","option_order":["false","true"],"gold":"true","raw_logits":[-2.046875,1.22265625],"temperature":1.0,"type":"noul","source_id":"8882","source_row_hash":"e1c5f172f9f02cade5685772af79ebded1b74b376026d93341832931a32676f9","input_tokens":52,"truncated":false,"calibrated_raw_logits":[-2.046875,1.22265625],"calibrated_temperature":5.0},
- {"id":"massive-920-choice","group_id":"massive-alarm-84e033aebf70598798ff","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"d2f20e3d91af6e06b4f1acf721dcd73de93d27d83c024195da8ad52b1bceecdd","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_remove","raw_logits":[-3.517578125,-4.140625,4.17578125],"temperature":1.0,"type":"choice","source_id":"920","source_row_hash":"ed0a9ccdf3b6e428b4770ab41b3c2b59fa4327b9580291f077f18f49a54b44b2","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-3.517578125,-4.140625,4.17578125],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-920-noul","group_id":"massive-alarm-84e033aebf70598798ff","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"4cc035ea2dff281db57f6de7c194ec38bfe20021c39aadf5fc817997e722b652","option_order":["false","true"],"gold":"false","raw_logits":[-2.470703125,-0.1820068359375],"temperature":1.0,"type":"noul","source_id":"920","source_row_hash":"ed0a9ccdf3b6e428b4770ab41b3c2b59fa4327b9580291f077f18f49a54b44b2","input_tokens":44,"truncated":false,"calibrated_raw_logits":[-2.470703125,-0.1820068359375],"calibrated_temperature":5.0},
- {"id":"massive-939-choice","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"b39b50798b10226d8212f2e2f9e3a8929c244b25102bcf6e856c5df4b13d4ee8","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[5.4375,-3.669921875,-0.99853515625],"temperature":1.0,"type":"choice","source_id":"939","source_row_hash":"936109e1a09f4aecb014f31c6bbb213ce6de4023624f13dfe11e9c083dd0320e","input_tokens":54,"truncated":false,"calibrated_raw_logits":[5.4375,-3.669921875,-0.99853515625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-939-noul","group_id":"massive-alarm-3ae746e1691596003503","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"dd6f955f414596fad6478d08e6f36bea3627253a74ec2d137249375a66c63afb","option_order":["false","true"],"gold":"true","raw_logits":[-2.650390625,0.978515625],"temperature":1.0,"type":"noul","source_id":"939","source_row_hash":"936109e1a09f4aecb014f31c6bbb213ce6de4023624f13dfe11e9c083dd0320e","input_tokens":49,"truncated":false,"calibrated_raw_logits":[-2.650390625,0.978515625],"calibrated_temperature":5.0},
- {"id":"massive-940-choice","group_id":"massive-alarm-5b03b657dea06a387027","task_id":"massive_alarm_choice_v1","split":"test","sample_hash":"58112a614bee3f995d34f5cd7698b4fc18f8696a65393c5d108082656961fb29","option_order":["alarm_set","alarm_query","alarm_remove"],"gold":"alarm_set","raw_logits":[1.388671875,-2.8046875,-0.0709228515625],"temperature":1.0,"type":"choice","source_id":"940","source_row_hash":"78635f925b3274baf503c5823a54dfb84e1a144e79ad8a8ef3ffbd50edfc6c4b","input_tokens":50,"truncated":false,"calibrated_raw_logits":[1.388671875,-2.8046875,-0.0709228515625],"calibrated_temperature":3.1355035588563744},
- {"id":"massive-940-noul","group_id":"massive-alarm-5b03b657dea06a387027","task_id":"massive_alarm_set_noul_v1","split":"test","sample_hash":"d534399a37dd93763633f7d9628ca5918f84cacf599473cd36b168ef8fcf2198","option_order":["false","true"],"gold":"true","raw_logits":[-2.1015625,1.1630859375],"temperature":1.0,"type":"noul","source_id":"940","source_row_hash":"78635f925b3274baf503c5823a54dfb84e1a144e79ad8a8ef3ffbd50edfc6c4b","input_tokens":45,"truncated":false,"calibrated_raw_logits":[-2.1015625,1.1630859375],"calibrated_temperature":5.0}
- ]
-}
diff --git a/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.md b/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.md
deleted file mode 100644
index 2360a9c0..00000000
--- a/research/benchmarks/zh_reliability/results/massive_alarm_2026-09-24.md
+++ /dev/null
@@ -1,43 +0,0 @@
-# MASSIVE Chinese alarm calibration evaluation
-
-Measurement date: **2026-09-24**. Execution base: `23a17522aa4942da6cce53a995a275760320b691`; contribution: `37cc58465da5cdabddf1ff543234397d666ad546`.
-
-Historical inference was executed on 2026-09-24 from base commit 23a17522aa4942da6cce53a995a275760320b691 with a dirty contribution worktree subsequently recorded in 37cc58465da5cdabddf1ff543234397d666ad546. The recorded execution source-file hashes are authoritative for the measured code. These measurements were not rerun on the later upstream rebase or evaluation-only refactor.
-
-E0 and E1 use the same multilingual checkpoint and raw decision logits. E1 fits one temperature per question type on calibration data. No model weights are updated.
-
-| Condition | Slice | n | Accuracy | Macro F1 | NLL | Brier | ECE |
-|---|---|---:|---:|---:|---:|---:|---:|
-| E0 | overall | 186 | 0.532258 | 0.440840 | 1.383747 | 0.730071 | 0.366488 |
-| E0 | choice | 93 | 0.634409 | 0.580928 | 1.296043 | 0.520825 | 0.253426 |
-| E0 | noul | 93 | 0.430108 | 0.300752 | 1.471450 | 0.939317 | 0.494611 |
-| E1 | overall | 186 | 0.532258 | 0.440840 | 0.785033 | 0.506898 | 0.197529 |
-| E1 | choice | 93 | 0.634409 | 0.580928 | 0.837271 | 0.474032 | 0.169905 |
-| E1 | noul | 93 | 0.430108 | 0.300752 | 0.732796 | 0.539765 | 0.289635 |
-
-Positive scalar temperature preserves argmax: accuracy and F1 are unchanged.
-
-| Type | Calibration n | Temperature | Bound reached |
-|---|---:|---:|---|
-| choice | 64 | 3.135504 | False |
-| noul | 64 | 5.000000 | True |
-
-Paired differences are E1 minus E0; negative NLL, Brier and ECE differences are better.
-
-| Metric | Difference | 95% group bootstrap interval |
-|---|---:|---|
-| accuracy | +0.000000 | [+0.000000, +0.000000] |
-| macro_f1 | +0.000000 | [+0.000000, +0.000000] |
-| nll | -0.598713 | [-0.865774, -0.369099] |
-| brier | -0.223173 | [-0.298826, -0.152890] |
-| ece | -0.168958 | [-0.224995, -0.098547] |
-
-## Scope and limits
-
-- Test: 93 source utterances, 89 heuristic groups, 186 correlated decisions. Calibration: 64 source utterances, 62 groups, 128 decisions. The paired question types are not independent samples.
-- This is a derived MASSIVE 1.1 alarm subset with upstream human intent-review evidence, not the full benchmark, naturally occurring customer-service logs, or independent local human review.
-- One fixed split/seed; 1,000 group bootstrap draws. Intervals condition on the fixed checkpoint and fitted temperatures, omit calibration uncertainty, and are descriptive without multiplicity correction.
-- Unknown checkpoint exposure to MASSIVE/SLURP; grouping may miss semantic dependencies. These results do not establish general Chinese improvement or unchanged behavior on other tasks.
-- The baseline noul fit reaches T=5, the permitted upper bound. Calibration settings were fixed without choosing on test. Risk/coverage does not improve at every coverage.
-- Configured max_len=512, head_max_len=192, FP16. Historical test inputs were at most 60 tokens; this is not a 512-token stress measurement.
-- The compact evidence omits source text and retains source IDs/hashes. It can recompute numerical results and check pairing; full source-schema/annotation checks require the prepared public dataset.
diff --git a/research/benchmarks/zh_reliability/runtime.py b/research/benchmarks/zh_reliability/runtime.py
deleted file mode 100644
index 0eb6b5ae..00000000
--- a/research/benchmarks/zh_reliability/runtime.py
+++ /dev/null
@@ -1,277 +0,0 @@
-"""Environment diagnostics and a thin adapter over the repository's actual Agent."""
-import importlib.metadata
-import json
-import os
-from pathlib import Path
-import platform
-import shutil
-import subprocess
-import sys
-import time
-
-from .data import atomic_json, digest, file_hash, options
-
-MODEL_ID = 'convaiinnovations/laya-multilingual'
-INPUT_KEYS = ('input_ids', 'attention_mask', 'marker_pos', 'marker_mask', 'qtype')
-
-
-def command(args):
- try:
- p = subprocess.run(args, capture_output=True, text=True, timeout=20)
- return p.stdout.strip() if p.returncode == 0 else p.stderr.strip()
- except (OSError, subprocess.TimeoutExpired) as e:
- return str(e)
-
-
-def environment():
- result = {'python': sys.version, 'executable': sys.executable, 'os': platform.platform(),
- 'commit': command(['git', 'rev-parse', 'HEAD']),
- 'dirty': bool(command(['git', 'status', '--porcelain'])),
- 'source_hashes': {str(p): file_hash(p) for p in sorted(Path(__file__).parent.glob('*.py'))},
- 'environment': {k: os.environ[k] for k in ('CUDA_VISIBLE_DEVICES', 'CUDA_DEVICE_ORDER',
- 'HF_HUB_OFFLINE', 'USE_TF', 'USE_TORCH', 'TOKENIZERS_PARALLELISM', 'OMP_NUM_THREADS')
- if k in os.environ}, 'dependencies': {},
- 'nvidia_smi': command(['nvidia-smi', '--query-gpu=index,name,uuid,memory.total,driver_version',
- '--format=csv,noheader']), 'gpus': []}
- for package in ('torch', 'transformers', 'numpy', 'safetensors', 'huggingface_hub', 'tokenizers'):
- try:
- result['dependencies'][package] = importlib.metadata.version(package)
- except importlib.metadata.PackageNotFoundError:
- result['dependencies'][package] = None
- try:
- import torch
- result.update(cuda=torch.version.cuda, cudnn=torch.backends.cudnn.version(),
- pytorch_arches=torch.cuda.get_arch_list())
- for i in range(torch.cuda.device_count()):
- p = torch.cuda.get_device_properties(i)
- gpu = {'process_index': i, 'name': p.name, 'uuid': str(getattr(p, 'uuid', 'unavailable')),
- 'memory_bytes': p.total_memory, 'capability': [p.major, p.minor]}
- if gpu['uuid'] != 'unavailable' and not gpu['uuid'].startswith('GPU-'):
- gpu['uuid'] = 'GPU-' + gpu['uuid']
- try:
- with torch.cuda.device(i):
- x = torch.ones((16, 16), device=f'cuda:{i}')
- value = (x @ x).sum().item()
- torch.cuda.synchronize(i)
- if value != 4096:
- raise RuntimeError('CUDA tensor result mismatch')
- del x
- gpu['status'] = 'SUCCESS'
- except Exception as e:
- gpu.update(status='FAILED', reason=str(e))
- # UUID matching avoids confusing process-local index with physical index.
- matches = [line.split(',')[0].strip() for line in result['nvidia_smi'].splitlines()
- if gpu['uuid'] != 'unavailable' and gpu['uuid'] in line]
- gpu['physical_index'] = matches[0] if len(matches) == 1 else None
- result['gpus'].append(gpu)
- result['gpu_status'] = ('SUCCESS' if result['gpus'] and all(g['status'] == 'SUCCESS' for g in result['gpus'])
- else 'FAILED' if result['gpus'] else 'NOT_RUN')
- except Exception as e:
- result.update(gpu_status='NOT_RUN', reason=str(e))
- return result
-
-
-def select_device(selector, env):
- matches = [g for g in env['gpus'] if selector.lower() in g['name'].lower() or selector == g['uuid']
- or selector == f"cuda:{g['process_index']}"]
- if len(matches) != 1 or matches[0]['status'] != 'SUCCESS':
- raise RuntimeError('device must uniquely match one visible GPU with a successful CUDA operation: ' + selector)
- return 'cuda:' + str(matches[0]['process_index'])
-
-
-def resolve_model(source, revision, offline, work):
- """Make a private snapshot so Agent's tokenizer compatibility patch cannot touch shared caches."""
- source_path = Path(source)
- sha = None
- if not source_path.is_dir():
- if source != MODEL_ID:
- raise ValueError('only the multilingual model is supported')
- from huggingface_hub import HfApi, snapshot_download
- if not offline:
- sha = HfApi().model_info(MODEL_ID, revision=revision).sha
- path = snapshot_download(MODEL_ID, revision=sha or revision, local_files_only=offline,
- allow_patterns=['rl_agent_config.json', 'model.safetensors',
- 'encoder/*.json', 'tokenizer/*'])
- source_path = Path(path)
- sha = source_path.name
- lineage_path = source_path / 'zh_lineage.json'
- lineage = json.loads(lineage_path.read_text()) if lineage_path.exists() else None
- if lineage and (lineage['model_id'] != MODEL_ID or lineage.get('trained')):
- raise ValueError('evaluation requires an original multilingual checkpoint')
- cfg = json.loads((source_path / 'rl_agent_config.json').read_text())
- # Local imports require explicit, inspectable multilingual identity, not just a folder name.
- if not lineage and not any(x in str(cfg.get('encoder', '')).lower() for x in ('mmbert', 'multilingual')):
- raise ValueError('local checkpoint config does not identify a multilingual encoder')
- files = [source_path / 'model.safetensors', source_path / 'rl_agent_config.json']
- files += sorted((source_path / 'encoder').glob('*.json')) + sorted((source_path / 'tokenizer').glob('*'))
- hashes = {str(p.relative_to(source_path)): file_hash(p) for p in files if p.is_file()}
- if lineage and lineage.get('weights_hash') and lineage['weights_hash'] != hashes['model.safetensors']:
- raise ValueError('checkpoint weight hash mismatch')
- # Preserve the base identity shape so existing calibration artifacts still match.
- identity = {'model_id': MODEL_ID, 'revision': sha or (lineage or {}).get('base_revision'),
- 'content_hash': digest(hashes), 'files': hashes,
- 'trained': False}
- target = Path(work) / 'model_input'
- target.mkdir()
- for name in hashes:
- dst = target / name
- dst.parent.mkdir(parents=True, exist_ok=True)
- shutil.copyfile(source_path / name, dst)
- atomic_json(Path(work) / 'model_identity.json', identity)
- return target, identity
-
-
-def ensure_device(agent, device):
- import torch
- expected = torch.device(device)
- if agent.device != expected or any(p.device != expected for p in agent.model.parameters()):
- raise RuntimeError('device mismatch / SDK CPU fallback: requested ' + device)
-
-
-class Adapter:
- def __init__(self, agent, identity, max_len=512, head_max_len=192, dtype='fp16'):
- import torch
- self.agent, self.identity = agent, identity
- self.max_len, self.head_max_len, self.dtype = max_len, head_max_len, dtype
- self.device = str(agent.device)
- if head_max_len < 16 or max_len <= head_max_len + 4:
- raise ValueError('max_len must leave state space beyond head_max_len + 4')
- agent.amp_enabled = dtype == 'fp16' and agent.device.type == 'cuda'
- agent.dtype = torch.float16 if agent.amp_enabled else torch.float32
- self.original = {'temperature_raw': agent.temperature_raw,
- 'temperature_by_options_raw': agent.temperature_by_options_raw,
- 'temperature': list(agent.temperature),
- 'temperature_by_options': dict(agent.temperature_by_options),
- 'lang_temperatures': dict(agent.lang_temperatures)}
-
- def temperature(self, row):
- from laya.common import QTYPES, temp_bucket
- qt = QTYPES[row['question']['type']]
- cfg = self.agent.lang_temperatures.get('zh', {'temperature': self.agent.temperature,
- 'temperature_by_options': self.agent.temperature_by_options})
- return cfg['temperature_by_options'].get(temp_bucket(qt, len(options(row))), cfg['temperature'][qt])
-
- def prepare(self, row):
- from laya.common import build_sequence, render_options
- a, q = self.agent, row['question']
- a._check_question(row['id'], q)
- internal = a._to_internal(q)
- item = a._encode_state(row['state'], [row['id']], {row['id']: internal},
- self.max_len, self.head_max_len)[0]
- state_ids = a.tok(row['state'].replace(a.tok.mask_token, ' '), add_special_tokens=False)['input_ids']
- empty, markers = build_sequence(a.tok, '', internal, self.max_len, self.head_max_len, state_ids=[])
- if len(markers) != len(options(row)) or any(item['ids'][m] != a.tok.mask_token_id for m in markers):
- raise ValueError('option marker truncated')
- retained = max(0, len(item['ids']) - len(empty))
- head = a.tok(f"{internal['t']} question: {internal['ins'].replace(a.tok.mask_token, ' ')}",
- add_special_tokens=False)['input_ids']
- opt_full = [a.tok(' ' + o.replace(a.tok.mask_token, ' '), add_special_tokens=False)['input_ids']
- for o in render_options(internal)]
- head_cut = len(head) > markers[0] - 2
- opt_cut = any(len(full) > (markers[i + 1] if i + 1 < len(markers) else len(empty) - 2) - m - 1
- for i, (m, full) in enumerate(zip(markers, opt_full)))
- item['label'] = options(row).index(row['label'])
- tokens = {'input_tokens': len(item['ids']), 'state_tokens': len(state_ids), 'state_retained': retained,
- 'option_count': len(markers), 'state_truncated': retained < len(state_ids),
- 'instructions_truncated': head_cut, 'options_truncated': opt_cut,
- 'truncated': retained < len(state_ids) or head_cut or opt_cut}
- return item, tokens
-
- def batch(self, items):
- from laya.common import collate_items
- return collate_items([items], self.agent.tok.pad_token_id)
-
- def forward(self, batch):
- import torch
- ensure_device(self.agent, self.device)
- with torch.autocast(device_type=self.agent.device.type, dtype=torch.float16,
- enabled=self.dtype == 'fp16' and self.agent.device.type == 'cuda'):
- logits, _action_logits = self.agent.model(
- **{k: batch[k].to(self.agent.device) for k in INPUT_KEYS})
- if not torch.isfinite(logits).all():
- raise RuntimeError('nonfinite decision logits')
- return logits
-
- def predict(self, rows, condition, split):
- import torch
- from .metrics import probability_record
- self.agent.model.eval()
- results = []
- # One state has exactly one decision sequence in this schema.
- with torch.no_grad():
- for r in rows:
- item, tokens = self.prepare(r)
- logits = self.forward(self.batch([item]))[0, :len(options(r))].float().cpu().tolist()
- rec = probability_record(r, logits, self.temperature(r))
- rec.update(tokens=tokens, condition=condition, split=split['assignments'][r['id']], model=self.identity)
- results.append(rec)
- return results
-
- def parity(self, rows):
- import numpy as np
- from .metrics import probability_record
- import torch
- errors = []
- self.agent.model.eval()
- for row in rows:
- item, _ = self.prepare(row)
- with torch.no_grad():
- z = self.forward(self.batch([item]))[0, :len(options(row))].cpu().tolist()
- p = probability_record(row, z, self.temperature(row))['probabilities']
- fallback_count = self.agent.cpu_fallback_count
- answer = self.agent.system_one(row['state'], {row['id']: row['question']}, lang='zh',
- max_len=self.max_len, head_max_len=self.head_max_len)['answers'][row['id']]
- ensure_device(self.agent, self.device)
- if self.agent.cpu_fallback_count != fallback_count:
- raise RuntimeError('Agent parity used CPU fallback')
- public = ([1 - answer['noul'], answer['noul']] if row['question']['type'] == 'noul'
- else [answer['probabilities'][k] for k in options(row)])
- error = float(np.max(np.abs(np.array(p) - public)))
- if error > .000051:
- raise AssertionError('Agent parity exceeds four-decimal rounding: ' + str(error))
- errors.append(error)
- return {'n': len(rows), 'max_absolute_error': max(errors), 'rounding_tolerance': .000051}
-
-
-def measure(adapter, rows, warmup=10, batches=30):
- import numpy as np
- import torch
- if adapter.agent.device.type != 'cuda':
- raise RuntimeError('GPU measurements prohibit CPU fallback')
- if warmup < 1 or batches < 1 or not rows:
- raise ValueError('measurement needs positive warmup/batches and nonempty samples')
- device = adapter.agent.device
- adapter.agent.model.eval()
- prepared = [adapter.batch([adapter.prepare(r)[0]]) for r in rows]
- prepared_gpu = [{k: b[k].to(device) for k in INPUT_KEYS} for b in prepared]
- result = {'warmup_batches': warmup, 'measured_batches': batches, 'batch_states': 1,
- 'batch_decisions': 1, 'independent_samples': len(rows),
- 'tokens': [adapter.prepare(r)[1] for r in rows], 'repetitions': batches / len(rows),
- 'model_stage_transfers': 'excludes H2D/D2H; inputs already on device'}
- from .metrics import probability_record
- with torch.no_grad():
- for stage in ('end_to_end', 'model_stage'):
- times = []
- torch.cuda.synchronize(device)
- torch.cuda.reset_peak_memory_stats(device)
- for i in range(warmup + batches):
- row = rows[i % len(rows)]
- torch.cuda.synchronize(device)
- start = time.perf_counter()
- if stage == 'end_to_end':
- item, _ = adapter.prepare(row)
- z = adapter.forward(adapter.batch([item]))[0, :len(options(row))].cpu().tolist()
- probability_record(row, z, adapter.temperature(row))
- else:
- adapter.forward(prepared_gpu[i % len(rows)])
- torch.cuda.synchronize(device)
- elapsed = time.perf_counter() - start
- if i >= warmup:
- times.append(elapsed)
- result[stage] = {'p50_batch_ms': float(np.quantile(times, .5) * 1000),
- 'p95_batch_ms': float(np.quantile(times, .95) * 1000),
- 'state_per_s': batches / sum(times), 'decision_per_s': batches / sum(times),
- 'max_memory_allocated': torch.cuda.max_memory_allocated(device),
- 'max_memory_reserved': torch.cuda.max_memory_reserved(device),
- 'memory_scope': 'inference including warmup, resident model and prepared inputs; excludes model loading'}
- return result
diff --git a/tests/test_zh_annotation_provenance.py b/tests/test_zh_annotation_provenance.py
deleted file mode 100644
index 04e92219..00000000
--- a/tests/test_zh_annotation_provenance.py
+++ /dev/null
@@ -1,150 +0,0 @@
-"""Offline regressions for upstream review evidence; all rows here are synthetic."""
-import copy
-import json
-from pathlib import Path
-import sys
-import tempfile
-import unittest
-
-sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
-from research.benchmarks.zh_reliability.data import partition, read_data
-
-
-class UpstreamAnnotationProvenanceTests(unittest.TestCase):
- def setUp(self):
- self.directory = tempfile.TemporaryDirectory()
- self.addCleanup(self.directory.cleanup)
- self.path = Path(self.directory.name) / 'data.jsonl'
- self.row = {
- 'id': 'synthetic_alarm_1', 'group_id': 'synthetic_source_1', 'task_id': 'alarm_operation',
- 'state': '设置明早的闹钟', 'split': 'train',
- 'question': {'type': 'choice', 'instructions': '选择闹钟操作',
- 'criteria': {'alarm_set': '设置闹钟', 'alarm_remove': '删除闹钟'}},
- 'label': 'alarm_set', 'tags': [],
- 'provenance': {
- 'kind': 'public_human_annotated', 'human_verified': False,
- 'source': 'synthetic source for offline validation only', 'license': 'CC-BY-4.0',
- 'annotation_status': 'upstream intent judgments; no local human review',
- 'source_intent': 'alarm_set',
- 'upstream_annotation': {
- 'dataset': 'MASSIVE', 'version': '1.1', 'source_id': '1', 'intent': 'alarm_set',
- 'source_row_hash': 'b' * 64,
- 'evidence_url': 'https://github.com/alexa/massive/blob/' + 'a' * 40 + '/README.md#data-format',
- 'judgments': [{'worker_id': str(i), 'intent_score': 1} for i in range(3)],
- },
- },
- }
-
- def read(self, rows, formal=True):
- self.path.write_text(''.join(json.dumps(r, ensure_ascii=False) + '\n' for r in rows), encoding='utf-8')
- return read_data(self.path, formal=formal)
-
- def test_source_review_keeps_local_review_false_and_preserves_group_splits(self):
- rows = []
- for i, split in enumerate(('train', 'dev', 'calibration', 'test')):
- source = copy.deepcopy(self.row)
- source.update(id=str(i), group_id='source_' + str(i), split=split)
- source['provenance']['upstream_annotation']['source_id'] = str(i)
- binary = copy.deepcopy(source)
- binary.update(id=str(i) + '_binary', task_id='alarm_set_binary', label='true')
- binary['question'] = {'type': 'noul', 'instructions': '是否设置闹钟?',
- 'criteria': {'false': '不是设置闹钟', 'true': '设置闹钟'}}
- rows.extend((source, binary))
- loaded = self.read(rows)
- self.assertTrue(all(r['provenance']['human_verified'] is False for r in loaded))
- manifest = partition(loaded, formal=True)
- self.assertEqual(manifest['group_counts'], {s: 1 for s in ('train', 'dev', 'calibration', 'test')})
- self.assertEqual(manifest['assignments'], {r['id']: r['split'] for r in rows})
- rows[-1]['split'] = 'train'
- with self.assertRaisesRegex(ValueError, 'split a group'):
- partition(self.read(rows), formal=True)
-
- def test_missing_or_invalid_identity_evidence_is_rejected(self):
- changes = {
- 'dataset': (None, 'Other'), 'version': (None, '1.0'),
- 'source_id': (None, 1, '', ' '), 'intent': (None, '', 'alarm_remove'),
- 'source_row_hash': (None, '', 'b' * 63, 'g' * 64),
- 'evidence_url': (None, 'https://github.com/alexa/massive/blob/main/README.md',
- 'https://example.com/' + 'a' * 40 + '/README.md',
- 'https://github.com/alexa/massive/blob/' + 'a' * 40 + '/README.md?branch=main'),
- }
- for field, values in changes.items():
- for value in values:
- with self.subTest(field=field, value=value):
- row = copy.deepcopy(self.row)
- row['provenance']['upstream_annotation'][field] = value
- with self.assertRaisesRegex(ValueError, 'source-review evidence'):
- self.read([row])
- for value in (None, [], 'claimed review'):
- row = copy.deepcopy(self.row)
- row['provenance']['upstream_annotation'] = value
- with self.assertRaisesRegex(ValueError, 'source-review evidence'):
- self.read([row])
-
- def test_contested_missing_or_duplicate_judgments_are_rejected(self):
- valid = self.row['provenance']['upstream_annotation']['judgments']
- invalid = [None, {}, [], valid[:2], valid[:2] + [valid[0]], valid + [None]]
- for score in (0, 2, True, 1.0, '1', None):
- judgments = copy.deepcopy(valid)
- judgments[2]['intent_score'] = score
- invalid.append(judgments)
- for worker in ('', ' ', None, 2):
- judgments = copy.deepcopy(valid)
- judgments[2]['worker_id'] = worker
- invalid.append(judgments)
- invalid.append(valid + [{'worker_id': 'fourth', 'intent_score': 0}])
- for judgments in invalid:
- with self.subTest(judgments=judgments):
- row = copy.deepcopy(self.row)
- row['provenance']['upstream_annotation']['judgments'] = judgments
- with self.assertRaisesRegex(ValueError, 'source-review evidence'):
- self.read([row])
-
- def test_public_kind_cannot_claim_local_human_review(self):
- row = copy.deepcopy(self.row)
- row['provenance']['human_verified'] = True
- for formal in (False, True):
- with self.subTest(formal=formal), self.assertRaisesRegex(ValueError, 'source-review evidence'):
- self.read([row], formal=formal)
-
- def test_formal_source_license_status_and_split_are_still_required(self):
- for field in ('source', 'license', 'annotation_status'):
- row = copy.deepcopy(self.row)
- del row['provenance'][field]
- with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'formal data requires'):
- self.read([row])
- row = copy.deepcopy(self.row)
- del row['split']
- with self.assertRaisesRegex(ValueError, 'formal data requires'):
- self.read([row])
- row = copy.deepcopy(self.row)
- del row['provenance']['source_intent']
- with self.assertRaisesRegex(ValueError, 'source-review evidence'):
- self.read([row])
-
- def test_pinned_raw_document_url_is_accepted(self):
- row = copy.deepcopy(self.row)
- row['provenance']['upstream_annotation']['evidence_url'] = (
- 'https://raw.githubusercontent.com/alexa/massive/' + 'a' * 40 + '/README.md')
- self.assertEqual(len(self.read([row])), 1)
-
- def test_existing_local_verification_and_fixture_rules_are_preserved(self):
- row = copy.deepcopy(self.row)
- row['provenance']['kind'] = 'user_annotated'
- row['provenance']['human_verified'] = True
- del row['provenance']['upstream_annotation']
- self.assertEqual(len(self.read([row])), 1)
- row['provenance']['kind'] = 'agent_authored_fixture'
- with self.assertRaisesRegex(ValueError, 'cannot claim human verification'):
- self.read([row])
- row['provenance']['human_verified'] = False
- self.assertEqual(len(self.read([row], formal=False)), 1)
- with self.assertRaisesRegex(ValueError, 'formal data requires'):
- self.read([row])
- row['provenance']['kind'] = 'unverified_external'
- with self.assertRaisesRegex(ValueError, 'formal data requires'):
- self.read([row])
-
-
-if __name__ == '__main__':
- unittest.main(verbosity=2)
diff --git a/tests/test_zh_massive.py b/tests/test_zh_massive.py
deleted file mode 100644
index 5b149544..00000000
--- a/tests/test_zh_massive.py
+++ /dev/null
@@ -1,440 +0,0 @@
-"""Offline MASSIVE converter checks using fabricated, test-only source records."""
-import copy
-import io
-from pathlib import Path
-import sys
-import tarfile
-import tempfile
-import unittest
-from unittest import mock
-
-sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
-from research.benchmarks.zh_reliability import prepare_massive as massive
-from research.benchmarks.zh_reliability.data import atomic_json, digest, file_hash, partition, read_data
-
-
-class MassiveConversionTests(unittest.TestCase):
- def setUp(self):
- self.directory = tempfile.TemporaryDirectory()
- self.addCleanup(self.directory.cleanup)
- self.path = Path(self.directory.name)
- self.source = {'archive_sha256': 'b' * 64}
- self.chinese, self.english = [], []
- for i, split in enumerate(('train', 'train', 'train', 'train', 'dev', 'test')):
- self.add(str(i), split, tuple(massive.INTENTS)[i % 3])
-
- def add(self, source_id, split, intent='alarm_set', utterance=None, template=None):
- chinese = {
- 'id': source_id, 'locale': 'zh-CN', 'partition': split, 'scenario': 'alarm', 'intent': intent,
- 'utt': utterance or '仅供测试的合成句子' + source_id,
- 'annot_utt': '仅供测试的合成句子' + source_id,
- 'judgments': [{'worker_id': str(i), 'intent_score': 1} for i in range(3)],
- }
- english = {**copy.deepcopy(chinese), 'locale': 'en-US',
- 'utt': 'Fabricated source for test only ' + source_id,
- 'annot_utt': template or 'Fabricated source for test only ' + source_id}
- self.chinese.append(chinese)
- self.english.append(english)
- return chinese, english
-
- def convert(self, **kwargs):
- return massive.convert(self.chinese, self.english, self.source, **kwargs)
-
- @staticmethod
- def by_source(rows):
- result = {}
- for row in rows:
- source_id = row['provenance']['upstream_annotation']['source_id']
- result.setdefault(source_id, {})[row['question']['type']] = row
- return result
-
- def test_exact_labels_shared_groups_and_upstream_provenance(self):
- rows, diagnostics, summary = self.convert()
- self.assertEqual(diagnostics, [])
- self.assertEqual(summary['derived_decisions'], 2 * len(self.chinese))
- for source in self.chinese:
- pair = self.by_source(rows)[source['id']]
- choice, noul = pair['choice'], pair['noul']
- self.assertEqual(choice['label'], source['intent'])
- self.assertEqual(choice['question']['criteria'], massive.INTENTS)
- self.assertEqual(noul['label'], 'true' if source['intent'] == 'alarm_set' else 'false')
- self.assertEqual(list(noul['question']['criteria']), ['false', 'true'])
- self.assertEqual(choice['group_id'], noul['group_id'])
- self.assertEqual(choice['split'], noul['split'])
- self.assertEqual(choice['state'], source['utt'])
- self.assertEqual(noul['state'], source['utt'])
- provenance = choice['provenance']
- self.assertIs(provenance['human_verified'], False)
- self.assertIs(provenance['local_human_review'], False)
- self.assertEqual(provenance['kind'], 'public_human_annotated')
- evidence = provenance['upstream_annotation']
- self.assertEqual(evidence['intent'], source['intent'])
- self.assertEqual(evidence['source_row_hash'], digest(source))
- self.assertEqual(evidence['judgments'], source['judgments'])
- self.assertEqual(evidence['evidence_url'], massive.EVIDENCE_URL)
- massive.write_jsonl(self.path / 'derived.jsonl', rows)
- validated = read_data(self.path / 'derived.jsonl', formal=True)
- self.assertEqual(partition(validated, formal=True)['counts'],
- {s: summary['by_split'][s]['decisions'] for s in summary['by_split']})
-
- def test_official_dev_test_are_preserved_and_calibration_is_only_from_train(self):
- rows, _, summary = self.convert(calibration_fraction=.5)
- pairs = self.by_source(rows)
- self.assertEqual(pairs['4']['choice']['split'], 'dev')
- self.assertEqual(pairs['5']['choice']['split'], 'test')
- source_splits = {r['id']: r['partition'] for r in self.chinese}
- for source_id, pair in pairs.items():
- if pair['choice']['split'] == 'calibration':
- self.assertEqual(source_splits[source_id], 'train')
- else:
- self.assertEqual(pair['choice']['split'], source_splits[source_id])
- self.assertEqual(summary['by_split']['train']['source_rows'], 2)
- self.assertEqual(summary['by_split']['calibration']['source_rows'], 2)
- self.assertEqual(sum(x['source_rows'] for x in summary['by_split'].values()), summary['retained_source_rows'])
- self.assertEqual(sum(x['decisions'] for x in summary['by_split'].values()), len(rows))
- self.assertEqual(sum(x['groups'] for x in summary['by_split'].values()), len({r['group_id'] for r in rows}))
-
- def test_english_delexicalized_template_overlap_quarantines_train(self):
- self.add('overlap_train', 'train', template='Test only: set an alarm at [time : seven]')
- self.add('overlap_test', 'test', template='Test only: set an alarm at [time : eleven]')
- rows, diagnostics, summary = self.convert()
- pairs = self.by_source(rows)
- self.assertNotIn('overlap_train', pairs)
- self.assertEqual(pairs['overlap_test']['choice']['split'], 'test')
- excluded = next(r for r in diagnostics if r['source_id'] == 'overlap_train')
- self.assertIn('shares_source_template_or_text_with_test', excluded['reasons'])
- self.assertEqual(excluded['group_id'], pairs['overlap_test']['choice']['group_id'])
- self.assertEqual(summary['retained_source_rows'] + summary['diagnostic_source_rows'], len(self.chinese))
-
- def test_chinese_normalization_and_transitive_template_grouping(self):
- self.add('chain_train', 'train', utterance='测试:ABC!', template='Synthetic bridge one [time : seven]')
- self.add('chain_dev', 'dev', utterance='测 试 abc', template='Synthetic bridge two [time : eight]')
- self.add('chain_test', 'test', utterance='仅供测试的第三句', template='Synthetic bridge two [time : ten]')
- rows, diagnostics, _ = self.convert()
- pairs = self.by_source(rows)
- self.assertNotIn('chain_train', pairs)
- self.assertNotIn('chain_dev', pairs)
- owner_group = pairs['chain_test']['choice']['group_id']
- excluded = [r for r in diagnostics if r['source_id'].startswith('chain_')]
- self.assertEqual({r['source_id'] for r in excluded}, {'chain_train', 'chain_dev'})
- self.assertTrue(all(r['group_id'] == owner_group for r in excluded))
- self.assertTrue(all('shares_source_template_or_text_with_test' in r['reasons'] for r in excluded))
-
- def test_duplicate_chinese_sources_in_one_partition_share_a_group(self):
- self.add('duplicate_a', 'test', utterance='测试:相同句子。')
- self.add('duplicate_b', 'test', utterance='测试 相同句子')
- rows, diagnostics, summary = self.convert()
- pairs = self.by_source(rows)
- self.assertEqual(pairs['duplicate_a']['choice']['group_id'], pairs['duplicate_b']['choice']['group_id'])
- self.assertEqual(summary['by_split']['test']['source_rows'], 3)
- self.assertEqual(summary['by_split']['test']['groups'], 2)
- self.assertEqual(diagnostics, [])
-
- def test_conflicting_chinese_labels_are_retained_only_in_diagnostics(self):
- self.add('conflict_a', 'test', intent='alarm_set', utterance='测试:相同但冲突的句子')
- self.add('conflict_b', 'test', intent='alarm_remove', utterance='测试 相同但冲突的句子。')
- rows, diagnostics, summary = self.convert()
- pairs = self.by_source(rows)
- self.assertNotIn('conflict_a', pairs)
- self.assertNotIn('conflict_b', pairs)
- excluded = [r for r in diagnostics if r['source_id'].startswith('conflict_')]
- self.assertEqual(len(excluded), 2)
- self.assertTrue(all('conflicting_intents_for_identical_normalized_chinese' in r['reasons'] for r in excluded))
- self.assertEqual(summary['excluded_reasons']['conflicting_intents_for_identical_normalized_chinese'], 2)
-
- def test_contested_or_missing_reviews_are_preserved_in_diagnostics(self):
- for source_id, score in (('contested', 0), ('reasonable_only', 2), ('boolean_not_score', True)):
- row, _ = self.add(source_id, 'test')
- row['judgments'][2]['intent_score'] = score
- missing, _ = self.add('missing_review', 'test')
- del missing['judgments']
- duplicate, _ = self.add('duplicate_review', 'test')
- duplicate['judgments'][2] = copy.deepcopy(duplicate['judgments'][0])
- original = copy.deepcopy(self.chinese)
- rows, diagnostics, summary = self.convert()
- expected = {'contested', 'reasonable_only', 'boolean_not_score', 'missing_review', 'duplicate_review'}
- self.assertEqual({r['source_id'] for r in diagnostics}, expected)
- self.assertTrue(expected.isdisjoint(self.by_source(rows)))
- self.assertTrue(all('upstream_intent_review_not_unanimous_or_missing' in r['reasons'] for r in diagnostics))
- source_lookup = {r['id']: r for r in original}
- self.assertTrue(all(d['source_row'] == source_lookup[d['source_id']] for d in diagnostics))
- self.assertEqual(self.chinese, original)
- self.assertEqual(summary['diagnostic_source_rows'], len(expected))
-
- def test_source_mismatch_is_rejected(self):
- variants = [('locale', 'zh-TW'), ('partition', 'test'), ('intent', 'alarm_remove')]
- for field, value in variants:
- english = copy.deepcopy(self.english)
- english[0][field] = value
- with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'source mismatch'):
- massive.convert(self.chinese, english, self.source)
- with self.assertRaisesRegex(ValueError, 'source mismatch'):
- massive.convert(self.chinese, self.english[1:], self.source)
- chinese = copy.deepcopy(self.chinese)
- chinese[0]['locale'] = 'zh-TW'
- with self.assertRaisesRegex(ValueError, 'source mismatch'):
- massive.convert(chinese, self.english, self.source)
-
- def test_deterministic_rerun_and_source_input_not_mutated(self):
- source_before = copy.deepcopy((self.chinese, self.english, self.source))
- first = self.convert(seed=7, calibration_fraction=.5)
- second = self.convert(seed=7, calibration_fraction=.5)
- self.assertEqual(first, second)
- self.assertEqual((self.chinese, self.english, self.source), source_before)
- reversed_result = massive.convert(list(reversed(self.chinese)), list(reversed(self.english)), self.source,
- seed=7, calibration_fraction=.5)
- self.assertEqual(first, reversed_result)
-
- def test_insufficient_groups_and_invalid_calibration_fraction_fail(self):
- for fraction in (0, 1, -.1, 1.1):
- with self.subTest(fraction=fraction), self.assertRaisesRegex(ValueError, 'calibration fraction'):
- self.convert(calibration_fraction=fraction)
- with self.assertRaisesRegex(ValueError, 'insufficient independent train groups'):
- massive.convert(self.chinese[3:], self.english[3:], self.source)
-
-
-class MassiveSourceChecksTests(unittest.TestCase):
- def setUp(self):
- self.directory = tempfile.TemporaryDirectory()
- self.addCleanup(self.directory.cleanup)
- self.path = Path(self.directory.name)
-
- def make_source(self):
- source = self.path / 'source'
- source.mkdir()
- for name in massive.SOURCE_FILES:
- text = 'FABRICATED TEST-ONLY DOCUMENT ' + name
- if name == 'NOTICE.md':
- text += '\nCC BY 4.0\n'
- if name == 'DATA_LICENSE':
- text += '\nAttribution 4.0 International\n'
- (source / name).write_text(text, encoding='utf-8')
- archive = source / 'massive-1.1.tar.gz'
- archive.write_bytes(b'FABRICATED TEST-ONLY ARCHIVE')
- metadata = {'archive_url': massive.ARCHIVE_URL, 'publisher_revision': massive.PUBLISHER_REVISION,
- 'archive_sha256': file_hash(archive),
- 'files': {name: file_hash(source / name) for name in massive.SOURCE_FILES}}
- atomic_json(source / 'source_manifest.json', metadata)
- return source, metadata
-
- def test_source_file_and_archive_checksum_mismatch_are_rejected(self):
- source, metadata = self.make_source()
- self.assertEqual(massive.check_source(source), metadata)
- text = (source / 'zh-CN.jsonl').read_bytes()
- (source / 'zh-CN.jsonl').write_bytes(text + b'changed')
- with self.assertRaisesRegex(ValueError, 'source file hash mismatch: zh-CN.jsonl'):
- massive.check_source(source)
- (source / 'zh-CN.jsonl').write_bytes(text)
- (source / 'massive-1.1.tar.gz').write_bytes(b'changed archive')
- with self.assertRaisesRegex(ValueError, 'source archive hash mismatch'):
- massive.check_source(source)
-
- def test_version_and_notice_mismatch_are_rejected(self):
- source, metadata = self.make_source()
- for field, value in (('archive_url', 'https://example.com/not-massive'), ('publisher_revision', 'main')):
- changed = {**metadata, field: value}
- atomic_json(source / 'source_manifest.json', changed)
- with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'unexpected dataset version'):
- massive.check_source(source)
- (source / 'NOTICE.md').write_text('test-only document without required notice')
- metadata['files']['NOTICE.md'] = file_hash(source / 'NOTICE.md')
- atomic_json(source / 'source_manifest.json', metadata)
- with self.assertRaisesRegex(ValueError, 'missing publisher dataset-license notice'):
- massive.check_source(source)
-
- @staticmethod
- def archive_bytes(extra=None):
- out = io.BytesIO()
- members = [('release/data/zh-CN.jsonl', b'{"test_only":true}\n'),
- ('release/data/en-US.jsonl', b'{"test_only":true}\n'),
- ('release/LICENSE', b'FABRICATED TEST ONLY Attribution 4.0 International')]
- if extra:
- members.append(extra)
- with tarfile.open(fileobj=out, mode='w:gz') as archive:
- for name, value in members:
- member = tarfile.TarInfo(name)
- if value is None:
- member.type = tarfile.SYMTYPE
- member.linkname = '../../outside'
- archive.addfile(member)
- else:
- member.size = len(value)
- archive.addfile(member, io.BytesIO(value))
- return out.getvalue()
-
- def responses(self, archive):
- def open_url(url, timeout):
- self.assertGreater(timeout, 0)
- if url == massive.ARCHIVE_URL:
- return io.BytesIO(archive)
- self.assertTrue(url.startswith(massive.RAW_DOCS))
- return io.BytesIO(b'FABRICATED TEST-ONLY DOCUMENT CC BY 4.0')
- return open_url
-
- def test_download_reads_selected_regular_members_without_extracting_paths(self):
- archive = self.archive_bytes(('../outside', b'not allowed outside source'))
- destination = self.path / 'download'
- with mock.patch.object(massive.urllib.request, 'urlopen', side_effect=self.responses(archive)):
- massive.fetch_source(destination)
- self.assertFalse((self.path / 'outside').exists())
- self.assertFalse((destination / 'release').exists())
- self.assertEqual((destination / 'zh-CN.jsonl').read_text(), '{"test_only":true}\n')
- self.assertEqual(massive.check_source(destination)['archive_sha256'], file_hash(destination / 'massive-1.1.tar.gz'))
-
- def test_download_rejects_selected_symlinks_and_duplicate_members(self):
- for i, extra in enumerate((('other/data/zh-CN.jsonl', None),
- ('other/data/en-US.jsonl', b'duplicate test-only member'))):
- archive = self.archive_bytes(extra)
- with self.subTest(extra=extra), mock.patch.object(massive.urllib.request, 'urlopen',
- side_effect=self.responses(archive)):
- with self.assertRaisesRegex(ValueError, 'unexpected/duplicate archive member'):
- massive.fetch_source(self.path / ('download_' + str(i)))
-
-
-class MassiveParquetTests(unittest.TestCase):
- """Fabricated HF-shaped records and transport mocks; no Parquet library or network."""
- def setUp(self):
- self.directory = tempfile.TemporaryDirectory()
- self.addCleanup(self.directory.cleanup)
- self.path = Path(self.directory.name)
- self.features = {
- 'scenario': {'_type': 'ClassLabel', 'names': ['music', 'alarm']},
- 'intent': {'_type': 'ClassLabel', 'names': ['alarm_query', 'alarm_remove', 'alarm_set']},
- }
- self.row = {
- 'id': 'test-only-1', 'locale': 'zh-CN', 'partition': 'dev', 'scenario': 1, 'intent': 2,
- 'utt': '仅供测试:请设置AM七点的闹钟!',
- 'annot_utt': '仅供测试:请设置[time : AM七点]的闹钟!',
- 'judgments': {'worker_id': ['0', '1', '2'], 'intent_score': [1, 1, 1],
- 'grammar_score': [3, 4, 3]},
- 'slot_method': {'slot': ['time'], 'method': ['localization']},
- }
-
- def decode(self, row=None, features=None):
- return massive.decode_parquet_row(self.row if row is None else row,
- self.features if features is None else features,
- 'zh-CN', 'validation')
-
- def test_decoder_uses_embedded_labels_and_transposes_sequences_without_text_changes(self):
- original = copy.deepcopy(self.row)
- result = self.decode()
- self.assertEqual(result['scenario'], 'alarm')
- self.assertEqual(result['intent'], 'alarm_set')
- self.assertEqual(result['utt'], original['utt'])
- self.assertEqual(result['annot_utt'], original['annot_utt'])
- self.assertEqual(result['partition'], 'dev')
- self.assertEqual(result['judgments'], [
- {'worker_id': '0', 'intent_score': 1, 'grammar_score': 3},
- {'worker_id': '1', 'intent_score': 1, 'grammar_score': 4},
- {'worker_id': '2', 'intent_score': 1, 'grammar_score': 3},
- ])
- self.assertEqual(result['slot_method'], [{'slot': 'time', 'method': 'localization'}])
- self.assertEqual(self.row, original)
- self.row['slot_method'] = {'slot': [], 'method': []}
- self.assertEqual(self.decode()['slot_method'], [])
-
- def test_decoder_rejects_invalid_label_indices_and_feature_type(self):
- for field in ('scenario', 'intent'):
- for value in (-1, len(self.features[field]['names']), True, 1.0, '1', None):
- row = copy.deepcopy(self.row)
- row[field] = value
- with self.subTest(field=field, value=value), self.assertRaisesRegex(ValueError, 'invalid parquet class label'):
- self.decode(row)
- features = copy.deepcopy(self.features)
- features[field]['_type'] = 'Value'
- with self.assertRaisesRegex(ValueError, 'invalid parquet class label'):
- self.decode(features=features)
-
- def test_decoder_rejects_wrong_locale_or_partition(self):
- for field, value in (('locale', 'en-US'), ('partition', 'validation'), ('partition', 'test')):
- row = copy.deepcopy(self.row)
- row[field] = value
- with self.subTest(field=field, value=value), self.assertRaisesRegex(ValueError, 'locale/partition mismatch'):
- self.decode(row)
- row = copy.deepcopy(self.row)
- row['partition'] = 'train'
- self.assertEqual(massive.decode_parquet_row(row, self.features, 'zh-CN', 'train')['partition'], 'train')
-
- def test_decoder_rejects_malformed_or_unequal_sequence_columns(self):
- for field in ('judgments', 'slot_method'):
- for value in (None, [], {}, {'one': 'not a list'}, {'one': [1], 'two': [1, 2]}):
- row = copy.deepcopy(self.row)
- row[field] = value
- with self.subTest(field=field, value=value), self.assertRaisesRegex(ValueError, 'parquet sequence'):
- self.decode(row)
-
- def make_source(self):
- directory = self.path / 'source'
- directory.mkdir()
- for name in massive.SOURCE_FILES + massive.PARQUET_FILES:
- text = 'FABRICATED TEST-ONLY SOURCE ' + name
- if name == 'NOTICE.md':
- text += ' CC BY 4.0'
- if name == 'DATA_LICENSE':
- text += ' Attribution 4.0 International'
- (directory / name).write_text(text)
- metadata = {
- 'archive_url': massive.ARCHIVE_URL, 'publisher_revision': massive.PUBLISHER_REVISION,
- 'transport': 'hf_parquet', 'mirror_revision': massive.PARQUET_REVISION,
- 'mirror_base_url': massive.PARQUET_BASE, 'license_text_url': massive.LICENSE_TEXT_URL,
- 'files': {name: file_hash(directory / name) for name in massive.SOURCE_FILES + massive.PARQUET_FILES},
- }
- atomic_json(directory / 'source_manifest.json', metadata)
- return directory, metadata
-
- def test_source_check_requires_fixed_mirror_metadata_and_parquet_hashes(self):
- directory, metadata = self.make_source()
- self.assertEqual(massive.check_source(directory), metadata)
- for field, value in (('mirror_revision', 'main'), ('mirror_base_url', 'https://example.com/'),
- ('license_text_url', 'https://example.com/LICENSE')):
- atomic_json(directory / 'source_manifest.json', {**metadata, field: value})
- with self.subTest(field=field), self.assertRaisesRegex(ValueError, 'unexpected parquet source revision'):
- massive.check_source(directory)
- atomic_json(directory / 'source_manifest.json', metadata)
- for name in massive.PARQUET_FILES:
- target = directory / name
- original = target.read_bytes()
- target.write_bytes(original + b'CHANGED')
- with self.subTest(name=name), self.assertRaisesRegex(ValueError, 'source file hash mismatch'):
- massive.check_source(directory)
- target.write_bytes(original)
-
- def test_mirror_fetch_uses_pinned_urls_and_publishes_completed_files(self):
- directory = self.path / 'download'
- expected = {f'{locale}-{split}.parquet': f'{massive.PARQUET_BASE}{locale}/{split}/0000.parquet'
- for locale in ('zh-CN', 'en-US') for split in ('train', 'validation', 'test')}
- expected.update({name: massive.RAW_DOCS + name for name in massive.SOURCE_FILES[3:]})
- expected['DATA_LICENSE'] = massive.LICENSE_TEXT_URL
- def open_url(url, timeout):
- self.assertIn(url, expected.values())
- self.assertGreater(timeout, 0)
- return io.BytesIO(('FABRICATED TEST-ONLY ' + url).encode())
- with mock.patch.object(massive.urllib.request, 'urlopen', side_effect=open_url) as download, \
- mock.patch.object(massive, 'prepare_parquet_source', return_value=directory) as prepare:
- self.assertEqual(massive.fetch_parquet_source(directory), directory)
- self.assertEqual(download.call_count, len(expected))
- prepare.assert_called_once_with(directory)
- self.assertEqual({p.name for p in directory.iterdir()}, set(expected))
- self.assertFalse(list(directory.glob('*.part')))
- for name, url in expected.items():
- self.assertEqual((directory / name).read_bytes(), ('FABRICATED TEST-ONLY ' + url).encode())
-
- def test_interrupted_mirror_download_is_not_published_or_prepared(self):
- class InterruptedStream(io.BytesIO):
- def read(self, size=-1):
- data = super().read(size)
- if not data:
- raise OSError('fabricated download interruption')
- return data
- directory = self.path / 'download'
- with mock.patch.object(massive.urllib.request, 'urlopen', return_value=InterruptedStream(b'test only')), \
- mock.patch.object(massive, 'prepare_parquet_source') as prepare:
- with self.assertRaisesRegex(OSError, 'fabricated download interruption'):
- massive.fetch_parquet_source(directory)
- prepare.assert_not_called()
- self.assertFalse((directory / massive.PARQUET_FILES[0]).exists())
- self.assertEqual((directory / (massive.PARQUET_FILES[0] + '.part')).read_bytes(), b'test only')
-
-
-if __name__ == '__main__':
- unittest.main(verbosity=2)
diff --git a/tests/test_zh_reliability.py b/tests/test_zh_reliability.py
deleted file mode 100644
index e58ee703..00000000
--- a/tests/test_zh_reliability.py
+++ /dev/null
@@ -1,383 +0,0 @@
-"""Offline logic tests; tiny random BERT is NOT a real multilingual Laya experiment."""
-import copy
-import contextlib
-import io
-import json
-import math
-import os
-from pathlib import Path
-import sys
-import tempfile
-import unittest
-from unittest.mock import patch
-
-os.environ.setdefault('USE_TF', '0')
-os.environ.setdefault('USE_TORCH', '1')
-sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
-
-import numpy as np
-import torch
-from safetensors.torch import save_file
-from tokenizers import Tokenizer
-from tokenizers.models import WordLevel
-from tokenizers.pre_tokenizers import Whitespace
-from transformers import BertConfig, BertModel, PreTrainedTokenizerFast
-
-from research.benchmarks.zh_reliability.calibration import calibrate, fit_temperature, validate_calibration
-from research.benchmarks.zh_reliability.data import FIXTURES, atomic_json, options, partition, read_data
-from research.benchmarks.zh_reliability.metrics import log_probs, paired_bootstrap, probability_record, quality, summarize
-from research.benchmarks.zh_reliability.runtime import Adapter, ensure_device, select_device
-from laya.agent import Agent
-from laya.common import DecisionModel, collate_items
-
-
-class DataMetricsTests(unittest.TestCase):
- def setUp(self):
- self.rows = read_data(FIXTURES)
- self.split = partition(self.rows)
- self.model = {'content_hash': 'mock-only', 'trained': False}
-
- def prediction(self, row=None, logits=(0., 0.), t=1.):
- row = row or self.rows[0]
- p = probability_record(row, logits, t)
- p.update(split=self.split['assignments'][row['id']], model=self.model)
- return p
-
- def test_schema_groups_manifest_and_relations(self):
- self.assertEqual(len(self.rows), 80)
- self.assertEqual(len(self.split['groups']), 40)
- self.assertEqual(self.split['counts'], {'test': 12, 'train': 48, 'dev': 8, 'calibration': 12})
- with tempfile.TemporaryDirectory() as tmp:
- path = Path(tmp) / 'split.json'
- saved = partition(self.rows, path)
- self.assertEqual(partition(self.rows, path, seed=1), saved)
- changed = copy.deepcopy(self.rows)
- changed[0]['state'] += '!'
- with self.assertRaisesRegex(ValueError, 'hash'):
- partition(changed, path)
- saved['assignments'][self.rows[0]['id']] = 'train'
- atomic_json(path, saved)
- with self.assertRaisesRegex(ValueError, 'leakage'):
- partition(self.rows, path)
-
- def test_schema_rejections(self):
- for change in ({'label': 'unknown'}, {'provenance': {}}, {'tags': 'negation'},
- {'question': {'type': 'score', 'instructions': 'bad'}},
- {'provenance': {'kind': 'agent_authored', 'human_verified': True}}):
- with self.subTest(change=change), tempfile.TemporaryDirectory() as tmp:
- rows = copy.deepcopy(self.rows)
- rows[0].update(change)
- path = Path(tmp) / 'data.jsonl'
- path.write_text('\n'.join(json.dumps(r) for r in rows))
- with self.assertRaises(ValueError):
- read_data(path)
- with self.assertRaises(ValueError):
- read_data(FIXTURES, formal=True)
-
- def test_template_family_may_not_cross_groups(self):
- rows = copy.deepcopy(self.rows)
- rows[2]['provenance']['template_family'] = rows[0]['provenance']['template_family']
- with tempfile.TemporaryDirectory() as tmp:
- p = Path(tmp) / 'data.jsonl'
- p.write_text('\n'.join(json.dumps(r) for r in rows))
- with self.assertRaisesRegex(ValueError, 'family crosses'):
- read_data(p)
-
- def test_formal_requires_fixed_licensed_verified_data(self):
- rows = copy.deepcopy(self.rows)
- for r in rows:
- r['provenance'].update(kind='user_supplied', human_verified=True, source='test source',
- license='CC0', annotation_status='double reviewed')
- r['split'] = self.split['assignments'][r['id']]
- with tempfile.TemporaryDirectory() as tmp:
- p = Path(tmp) / 'real.jsonl'
- p.write_text('\n'.join(json.dumps(r) for r in rows))
- accepted = read_data(p, formal=True)
- self.assertEqual(partition(accepted, formal=True)['assignments'], self.split['assignments'])
- accepted[0]['split'] = 'train'
- with self.assertRaisesRegex(ValueError, 'split a group'):
- partition(accepted, formal=True)
-
- def test_choice_permutation_preserves_semantic_mapping(self):
- row = self.rows[40]
- before = probability_record(row, [4., 3., 2., 1.], 1.)
- perm = copy.deepcopy(row)
- perm['question']['criteria'] = dict(reversed(list(perm['question']['criteria'].items())))
- after = probability_record(perm, [1., 2., 3., 4.], 1.)
- self.assertEqual(before['predicted'], after['predicted'])
- self.assertEqual(before['gold'], after['gold'])
- for key, value in zip(before['option_order'], before['probabilities']):
- self.assertAlmostEqual(value, dict(zip(after['option_order'], after['probabilities']))[key])
-
- def test_noul_probability_is_true_despite_display_labels(self):
- row = copy.deepcopy(self.rows[0])
- row['question']['labels'] = {'false': '拒绝', 'true': '接受'}
- row['question']['criteria'] = dict(reversed(list(row['question']['criteria'].items())))
- p = self.prediction(row, [0., math.log(3)])
- self.assertEqual(p['option_order'], ['false', 'true'])
- self.assertAlmostEqual(p['p_true'], .75)
- self.assertAlmostEqual(p['confidence'], .75)
-
- def test_valid_mask_and_temperature_invariance(self):
- expected = log_probs([1., 2.], 1.)
- np.testing.assert_allclose(expected, log_probs([1., 99., 2.], 1., [True, False, True]))
- np.testing.assert_allclose(np.exp(expected), torch.softmax(torch.tensor([1., 2.], dtype=torch.float64), 0))
- for t in (.5, 1., 2., 5.):
- self.assertEqual(log_probs([1., 4., -2.], t).argmax(), 1)
- for t in (0., -.1, .1, 6., float('nan'), float('inf')):
- with self.assertRaises(ValueError):
- log_probs([0., 1.], t)
-
- def test_hand_computable_nll_brier_ece_and_confidence(self):
- p = self.prediction(logits=(0., math.log(3))) # gold true, correct with p=.75
- m = quality([p])
- self.assertAlmostEqual(m['nll'], -math.log(.75))
- self.assertAlmostEqual(m['brier'], .125)
- self.assertAlmostEqual(m['ece'], .25)
- self.assertAlmostEqual(p['confidence'], .75) # not 1 - entropy/log(2)
- self.assertEqual(m['accuracy'], 1.)
- uniform = self.prediction()
- self.assertEqual(uniform['confidence'], .5)
- self.assertIsNone(quality([])['nll'])
- json.dumps(summarize([]), allow_nan=False)
-
- def test_nll_does_not_underflow_via_probabilities(self):
- p = self.prediction(logits=(10000., -10000.))
- self.assertEqual(quality([p])['nll'], 20000.)
-
- def test_risk_ties_do_not_depend_on_labels(self):
- a, b = self.prediction(), self.prediction()
- a.update(id='a', gold='false')
- b.update(id='b', gold='true')
- self.assertEqual(quality([b, a])['risk_coverage'][0]['risk'], 0)
- a['gold'], b['gold'] = 'true', 'false'
- self.assertEqual(quality([b, a])['risk_coverage'][0]['risk'], 1)
- self.assertEqual(quality([a, b])['risk_coverage'], quality([b, a])['risk_coverage'])
-
- def test_task_local_macro_f1(self):
- a, b = self.prediction(), self.prediction()
- a.update(task_id='a', gold='false', predicted='false')
- b.update(task_id='b', gold='true', predicted='true')
- # Each task has one perfectly predicted class and one absent class: F1=.5 each.
- self.assertEqual(quality([a, b])['macro_f1'], .5)
-
- def calibration_rows(self):
- return [self.prediction(r, [0.] * len(options(r))) for r in self.rows
- if self.split['assignments'][r['id']] == 'calibration']
-
- def test_fit_export_reload_no_double_scaling_or_bucket_shadowing(self):
- rows = self.calibration_rows()
- original = {'temperature': [2., 3., 4.], 'temperature_by_options': {'choice:3-5': 5., 'score:2': 2.},
- 'lang_temperatures': {}}
- a = calibrate(rows, self.model, self.split, original, minimum=1, smoke=True)
- with tempfile.TemporaryDirectory() as tmp:
- p = Path(tmp) / 'cal.json'
- atomic_json(p, a)
- a = json.loads(p.read_text())
- cfg = validate_calibration(a, self.model, self.split, True)
- self.assertEqual(cfg['zh']['temperature'][1], 3.)
- self.assertNotIn('choice:3-5', cfg['zh']['temperature_by_options'])
- self.assertEqual(cfg['zh']['temperature_by_options']['score:2'], 2.)
- altered = copy.deepcopy(rows)
- for r in altered:
- r['temperature'] = 5.
- r['probabilities'] = [0.] * len(r['raw_logits'])
- self.assertEqual(fit_temperature(rows, 1), fit_temperature(altered, 1))
- with self.assertRaisesRegex(ValueError, 'smoke'):
- validate_calibration(a, self.model, self.split)
- with self.assertRaisesRegex(ValueError, 'model or data'):
- validate_calibration(a, {'content_hash': 'other'}, self.split, True)
- bad = copy.deepcopy(self.split)
- bad['data_hash'] = 'wrong'
- with self.assertRaises(ValueError):
- validate_calibration(a, self.model, bad, True)
- bad = copy.deepcopy(a)
- bad['calibration_hash'] = 'wrong'
- with self.assertRaises(ValueError):
- validate_calibration(bad, self.model, self.split, True)
- bad = copy.deepcopy(a)
- bad['lang_temperatures']['zh']['temperature'][0] = .1
- with self.assertRaises(ValueError):
- validate_calibration(bad, self.model, self.split, True)
-
- def test_fit_bounds_skipped_and_partition_guard(self):
- rows = self.calibration_rows()
- self.assertEqual(fit_temperature(rows)['status'], 'SKIPPED')
- a = self.prediction(logits=(10., -10.))
- result = fit_temperature([a] * 30)
- self.assertEqual(result['temperature'], 5.)
- self.assertTrue(result['boundary_hit'])
- self.assertLessEqual(result['nll_fitted'], result['nll_t1'])
- rows[0]['split'] = 'test'
- with self.assertRaisesRegex(ValueError, 'calibration partition'):
- calibrate(rows, self.model, self.split, {})
-
- def test_bootstrap_pairs_and_insufficient_groups(self):
- rows = self.calibration_rows()
- self.assertEqual(paired_bootstrap(rows, rows)['status'], 'SKIPPED')
- result = paired_bootstrap(rows, rows, repeats=10, min_groups=2)
- self.assertEqual(result['after_minus_before']['nll']['ci95'], [0., 0.])
- with self.assertRaises(ValueError):
- paired_bootstrap(rows, rows[:-1])
-
- def test_visible_device_selector_and_cpu_fallback(self):
- env = {'gpus': [{'name': 'TITAN RTX', 'process_index': 0, 'uuid': 'GPU-b', 'status': 'SUCCESS'},
- {'name': 'RTX 2080 Ti', 'process_index': 1, 'uuid': 'GPU-a', 'status': 'FAILED'}]}
- self.assertEqual(select_device('GPU-b', env), 'cuda:0')
- with self.assertRaises(RuntimeError):
- select_device('2080', env)
- with self.assertRaises(RuntimeError):
- select_device('physical:0', env)
- fake = type('A', (), {'device': torch.device('cpu')})()
- with self.assertRaisesRegex(RuntimeError, 'fallback'):
- ensure_device(fake, 'cuda:0')
-
- def test_cli_failure_is_nonzero_and_cannot_reuse_success(self):
- from research.benchmarks.zh_reliability.__main__ import main
- with tempfile.TemporaryDirectory() as tmp:
- run = Path(tmp) / 'failed'
- with patch('research.benchmarks.zh_reliability.__main__.environment', return_value={'gpus': [], 'gpu_status': 'NOT_RUN'}):
- self.assertEqual(main(['import-data', '--data', str(run / 'missing'), '--run-dir', str(run)]), 1)
- m = json.loads((run / 'manifest.json').read_text())
- self.assertEqual(m['status'], 'FAILED')
- self.assertEqual(m['error']['stage'], 'data')
- self.assertEqual(main(['import-data', '--run-dir', str(run)]), 2)
-
- def test_no_cuda_smoke_and_report_saved_results(self):
- from research.benchmarks.zh_reliability.__main__ import main
- with tempfile.TemporaryDirectory() as tmp:
- run, out = Path(tmp) / 'no-gpu', Path(tmp) / 'report'
- with patch('research.benchmarks.zh_reliability.__main__.environment', return_value={'gpus': [], 'gpu_status': 'NOT_RUN'}):
- self.assertEqual(main(['smoke', '--gpu', '--run-dir', str(run)]), 0)
- self.assertEqual(json.loads((run / 'manifest.json').read_text())['status'], 'SKIPPED')
- self.assertEqual(main(['report', '--source', str(run), '--run-dir', str(out)]), 0)
- self.assertEqual((run / 'split_manifest.json').read_bytes(), (out / 'split_manifest.json').read_bytes())
- self.assertEqual((run / 'metrics.json').read_bytes(), (out / 'metrics.json').read_bytes())
- self.assertIn('SKIPPED', (out / 'report.md').read_text())
-
- def test_all_cli_help(self):
- from research.benchmarks.zh_reliability.__main__ import parser
- for name in ('check-env', 'import-data', 'smoke', 'eval', 'calibrate', 'report'):
- with self.subTest(name=name), contextlib.redirect_stdout(io.StringIO()), self.assertRaises(SystemExit) as e:
- parser().parse_args([name, '--help'])
- self.assertEqual(e.exception.code, 0)
-
-
-class TinyEvaluationTests(unittest.TestCase):
- @classmethod
- def setUpClass(cls):
- torch.set_num_threads(1)
- cls.tmp = tempfile.TemporaryDirectory()
- cls.addClassCleanup(cls.tmp.cleanup)
- cls.base = Path(cls.tmp.name) / 'tiny'
- cls.base.mkdir()
- torch.manual_seed(7)
- config = BertConfig(vocab_size=16, hidden_size=16, num_hidden_layers=1,
- num_attention_heads=1, intermediate_size=32)
- config.save_pretrained(cls.base / 'encoder')
- backend = Tokenizer(WordLevel({'[PAD]': 0, '[UNK]': 1, '[CLS]': 2, '[SEP]': 3, '[MASK]': 4,
- 'false': 5, 'true': 6, 'billing': 7, 'technical': 8,
- 'shipping': 9, 'other': 10}, unk_token='[UNK]'))
- backend.pre_tokenizer = Whitespace()
- tok = PreTrainedTokenizerFast(tokenizer_object=backend, pad_token='[PAD]', unk_token='[UNK]',
- cls_token='[CLS]', sep_token='[SEP]', mask_token='[MASK]')
- tok.save_pretrained(cls.base / 'tokenizer')
- model = DecisionModel(BertModel(config), head_layers=1)
- save_file(model.state_dict(), cls.base / 'model.safetensors')
- atomic_json(cls.base / 'rl_agent_config.json', {'encoder': 'tiny-test-only', 'head_layers': 1,
- 'act_costs': {'act': 0}, 'temperature': [2., 1., 3.],
- 'temperature_by_options': {'choice:3-5': 1.5}, 'max_len': 128, 'head_max_len': 64})
- cls.rows = read_data(FIXTURES)
- cls.split = partition(cls.rows)
-
- def adapter(self):
- return Adapter(Agent(str(self.base), device='cpu', fast=False, compile=False),
- {'model_id': 'tiny-mock-only', 'revision': None, 'content_hash': 'tiny', 'trained': False},
- max_len=128, head_max_len=64, dtype='fp32')
-
- def test_raw_forward_official_parity_and_mask(self):
- a = self.adapter()
- result = a.parity([self.rows[0], self.rows[40]])
- self.assertLessEqual(result['max_absolute_error'], .000051)
- items = [a.prepare(r)[0] for r in (self.rows[0], self.rows[40])]
- b = collate_items([items], a.agent.tok.pad_token_id)
- self.assertEqual(b['marker_mask'].tolist(), [[True, True, False, False], [True] * 4])
- a.agent.model.eval()
- with torch.no_grad():
- z = a.forward(b)
- self.assertTrue(torch.equal(z[0, 2:], torch.full((2,), -1e4)))
-
- def test_mock_decision_not_action_and_noul_display_labels(self):
- a = self.adapter()
- row = copy.deepcopy(self.rows[0])
- row['question']['labels'] = {'false': '不', 'true': '是'}
- with patch.object(a.agent.model, 'forward', return_value=(torch.tensor([[0., 3.]]), torch.tensor([[999., -999.]]))):
- p = a.predict([row], 'E0', self.split)[0]
- self.assertAlmostEqual(p['p_true'], 1 / (1 + math.exp(-1)))
- a.parity([row])
-
- def test_calibrated_evaluation_keeps_logits_and_restores_baseline(self):
- from research.benchmarks.zh_reliability.__main__ import evaluate
- adapter = self.adapter()
- weights = {name: value.clone() for name, value in adapter.agent.model.state_dict().items()}
- baseline, base_metrics = evaluate(adapter, self.rows, self.split)
- calibration_rows, _ = evaluate(adapter, self.rows, self.split, selection='calibration')
- artifact = calibrate(calibration_rows, adapter.identity, self.split, adapter.original,
- minimum=1, smoke=True)
- calibrated, calibrated_metrics = evaluate(adapter, self.rows, self.split, artifact, smoke=True)
- self.assertTrue(all(row['condition'] == 'E0' and row['split'] == 'test' for row in baseline))
- self.assertTrue(all(row['condition'] == 'E1' and row['split'] == 'test' for row in calibrated))
- self.assertEqual([row['raw_logits'] for row in baseline], [row['raw_logits'] for row in calibrated])
- self.assertEqual([row['predicted'] for row in baseline], [row['predicted'] for row in calibrated])
- self.assertEqual(base_metrics['overall']['accuracy'], calibrated_metrics['overall']['accuracy'])
- for before, after in zip(baseline, calibrated):
- fitted = artifact['fitted'][after['question']['type']]['temperature']
- np.testing.assert_allclose(after['log_probabilities'], log_probs(before['raw_logits'], fitted))
- restored, restored_metrics = evaluate(adapter, self.rows, self.split)
- self.assertEqual(restored, baseline)
- self.assertEqual(restored_metrics, base_metrics)
- self.assertTrue(all(torch.equal(value, weights[name])
- for name, value in adapter.agent.model.state_dict().items()))
-
- def test_official_parity_rejects_restored_device_cpu_fallback(self):
- adapter = self.adapter()
- original = adapter.agent.system_one
- def fallback(*args, **kwargs):
- result = original(*args, **kwargs)
- adapter.agent.cpu_fallback_count += 1
- return result
- with patch.object(adapter.agent, 'system_one', side_effect=fallback):
- with self.assertRaisesRegex(RuntimeError, 'fallback'):
- adapter.parity(self.rows[:1])
-
- def test_token_lengths_state_truncation_and_missing_marker(self):
- a = self.adapter()
- r = copy.deepcopy(self.rows[0])
- r['state'] = 'hello ' * 500
- item, tokens = a.prepare(r)
- self.assertTrue(tokens['state_truncated'])
- self.assertEqual(tokens['input_tokens'], 128)
- self.assertLess(tokens['state_retained'], tokens['state_tokens'])
- a.max_len = 4
- with self.assertRaises(ValueError):
- a.prepare(r)
-
-
[email protected](os.environ.get('LAYA_ZH_REAL_MODEL'), 'opt-in: set LAYA_ZH_REAL_MODEL to multilingual local weights')
-class RealCheckpointTests(unittest.TestCase):
- def test_official_parity(self):
- if not torch.cuda.is_available():
- self.skipTest('CUDA unavailable')
- # Keep opt-in load within a private copy; never let Agent patch a shared snapshot.
- from research.benchmarks.zh_reliability.runtime import resolve_model
- with tempfile.TemporaryDirectory() as tmp:
- source, identity = resolve_model(os.environ['LAYA_ZH_REAL_MODEL'], 'main', True, tmp)
- agent = Agent(str(source), device='cuda:0', fast=False, compile=False)
- ensure_device(agent, 'cuda:0')
- a = Adapter(agent, identity, 256, 192, 'fp16')
- rows = read_data(FIXTURES)
- a.parity(rows[:2] + rows[40:42])
-
-
-if __name__ == '__main__':
- unittest.main()
diff --git a/tests/test_zh_results.py b/tests/test_zh_results.py
deleted file mode 100644
index 35931d15..00000000
--- a/tests/test_zh_results.py
+++ /dev/null
@@ -1,95 +0,0 @@
-"""Recompute the published E0/E1 evidence and reject inconsistent archived results."""
-import copy
-import json
-from pathlib import Path
-import sys
-import unittest
-
-sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
-
-from research.benchmarks.zh_reliability.aggregate_massive import render_report, verify_artifact
-from research.benchmarks.zh_reliability.data import digest
-
-RESULTS = Path(__file__).resolve().parents[1] / 'research/benchmarks/zh_reliability/results'
-
-
-class PublishedResultsTests(unittest.TestCase):
- @classmethod
- def setUpClass(cls):
- cls.artifact = json.loads((RESULTS / 'massive_alarm_2026-09-24.json').read_text(encoding='utf-8'))
- cls.result = verify_artifact(cls.artifact)
-
- def altered(self):
- return copy.deepcopy(self.artifact)
-
- def test_published_table_recomputes_from_decisions(self):
- self.assertEqual(self.result['counts']['test_source_utterances'], 93)
- self.assertEqual(self.result['counts']['test_groups'], 89)
- self.assertEqual(self.result['counts']['test_decisions'], 186)
- before, after = (self.result[c]['overall'] for c in ('E0', 'E1'))
- self.assertAlmostEqual(before['accuracy'], 99 / 186)
- self.assertEqual(before['accuracy'], after['accuracy'])
- self.assertEqual(before['macro_f1'], after['macro_f1'])
- self.assertAlmostEqual(before['nll'], 1.383747, places=6)
- self.assertAlmostEqual(after['nll'], .785033, places=6)
- self.assertEqual(self.result['fitted']['noul']['temperature'], 5.)
- self.assertTrue(self.result['fitted']['noul']['boundary_hit'])
- paired = self.result['paired']['E1-E0']
- self.assertEqual(paired['repeats'], 1000)
- self.assertEqual(paired['after_minus_before']['accuracy']['ci95'], [0., 0.])
- self.assertLess(paired['after_minus_before']['nll']['ci95'][1], 0.)
- self.assertEqual(render_report(self.artifact, self.result),
- (RESULTS / 'massive_alarm_2026-09-24.md').read_text(encoding='utf-8'))
-
- def test_records_hash_rejects_changed_logits(self):
- artifact = self.altered()
- artifact['records'][0]['raw_logits'][0] += 1.
- with self.assertRaisesRegex(ValueError, 'record hash mismatch'):
- verify_artifact(artifact, bootstrap=False)
-
- def test_rehashed_logits_must_match_between_conditions(self):
- artifact = self.altered()
- record = next(r for r in artifact['records'] if r['split'] == 'test')
- record['calibrated_raw_logits'][0] += 1.
- artifact['records_sha256'] = digest(artifact['records'])
- with self.assertRaisesRegex(ValueError, 'changed raw decision logits'):
- verify_artifact(artifact, bootstrap=False)
-
- def test_rehashed_duplicate_ids_are_rejected(self):
- artifact = self.altered()
- artifact['records'][1]['id'] = artifact['records'][0]['id']
- artifact['records_sha256'] = digest(artifact['records'])
- with self.assertRaisesRegex(ValueError, 'duplicate decision IDs'):
- verify_artifact(artifact, bootstrap=False)
-
- def test_rehashed_cross_partition_group_is_rejected(self):
- artifact = self.altered()
- calibration = next(r for r in artifact['records'] if r['split'] == 'calibration')
- record = next(r for r in artifact['records'] if r['split'] == 'test')
- record['group_id'] = calibration['group_id']
- artifact['records_sha256'] = digest(artifact['records'])
- with self.assertRaisesRegex(ValueError, 'group leakage'):
- verify_artifact(artifact, bootstrap=False)
-
- def test_source_pair_hash_must_match(self):
- artifact = self.altered()
- artifact['records'][0]['source_row_hash'] = '0' * 64
- artifact['records_sha256'] = digest(artifact['records'])
- with self.assertRaisesRegex(ValueError, 'source utterance split/group/hash mismatch'):
- verify_artifact(artifact, bootstrap=False)
-
- def test_temperature_is_refitted_from_calibration_only(self):
- artifact = self.altered()
- artifact['calibration']['fitted']['choice']['temperature'] += .1
- with self.assertRaisesRegex(ValueError, 'fitted temperature/objective mismatch'):
- verify_artifact(artifact, bootstrap=False)
-
- def test_stored_metric_cannot_override_recomputed_value(self):
- artifact = self.altered()
- artifact['summary']['E1']['overall']['accuracy'] = 1.
- with self.assertRaisesRegex(ValueError, 'stored summary mismatch: E1'):
- verify_artifact(artifact, bootstrap=False)
-
-
-if __name__ == '__main__':
- unittest.main()