The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
Merge pull request #37 from JamesKrW/gateway/compact-raw (+1 more)
Kangrui Wang218aa2745eef
diff --git a/docs/RUNNING.md b/docs/RUNNING.mdindex 699bb43ff8..6ef9357bc9 100644--- a/docs/RUNNING.md+++ b/docs/RUNNING.md@@ -56,6 +56,14 @@ results site with `scripts/write_task_results.py --out` outside the repository (next to the checkout, gets `<batch>/<task>/<run id>/`; `RESULTS=` skips it), updates `$JOBS/<batch>.summary.md`, andstops its lane when a trial shows a model-account failure.+To watch a batch live: `scripts/live_board.py --jobs <JOBS>` (repeatable; stdlib only) serves a page on 127.0.0.1 that+refreshes every 5 s. It shows every trial as pending, setup, running (agent time against the task's limit), verifying,+success or failed, with live tokens when the usage gateway runs, the reward, the videos and the tail of the agent's+output. Pending trials come from `$JOBS/.plan/<batch>.json`, which the runner writes when it starts. Reach the page from+another machine with `ssh -N -L 8765:127.0.0.1:<port> <host>`; it shows the agent's output, so keep it off public+addresses. It has no login (every user of a shared host can open it) and serves a running trial's files before the+runner's secret scan has redacted them.+Runner-side tools (they run on the host, never in a task): `scripts/log_mirror.py` (in the `rcb-log-mirror` container: anappend-only copy of every trial's `agent/` record that neither the agent nor the host user can delete),`scripts/mem_guard.py` (keeps 2 GiB of host memory available by killing the largest benchmark container first),@@ -90,7 +98,7 @@ Per trial, next to Harbor's `agent/` and `verifier/` (outside the agent's mounts```<trial>/gateway/usage.jsonl one line per request: time, duration, host, path, status, model, input / cached / output tokens<trial>/gateway/totals.json running totals, rewritten after every request-<trial>/gateway/raw/ every request and response body (GATEWAY_RAW=1, the default with the gateway on), gzipped JSON (headers without credentials)+<trial>/gateway/raw.jsonl.xz every request and response body (GATEWAY_RAW=1, the default with the gateway on; headers without credentials)```Input counts what the model read (Anthropic: input + cache read + cache creation; OpenAI: `input_tokens`, cachedincluded); output includes reasoning: the same convention as Harbor's `n_input_tokens` / `n_output_tokens`, which the batch@@ -100,7 +108,12 @@ before logging (measured 2026-10-01 on 9 benchmarks: the difference is exactly t(default 100M / 10M; 0 = no limit) gets every further model request refused (HTTP 400, "token budget exceeded"), sothe agent stops; `totals.json` records `over_token_budget = 1`, the batch summary marks it, and `write_task_results.py`counts it as failed (PROTOCOL §5). Tested 2026-10-02: Codex 0.159.2 ends its turn on the first refusal, without retrying.+The raw log stores each conversation item once and a request as references to the items it resends, like Codex's and+Claude Code's own session files, and is lossless: `scripts/usage_gateway/rawlog.py cat <trial>/gateway` prints every+request and response in full. The gateway appends to `raw.jsonl.gz` while the trial runs; after each job the runner+compacts it to `raw.jsonl.xz` (one 1,994-request trial: 5.9 GB as one gzip file per request, the format before+2026-10-02, 32 MB now; `rawlog.py pack --delete <dir>` converts such `raw/` directories).Switches: `GATEWAY=1` (record usage and, by default, raw bodies), `GATEWAY_RAW=0` (usage only; raw bodies are private: they hold thewhole conversation). Known limits: no WebSocket (Codex's built-in provider falls back to HTTPS after a few refused-attempts); Codex request bodies are zstd-compressed and are stored as base64 unless `zstandard` is installed; OpenAI-reasoning arrives encrypted. Requests no trial can be matched to go to `<jobs>/.gateway/unattributed.jsonl`.+attempts); Codex request bodies are zstd-compressed and are stored as base64, and so not deduplicated, unless+`zstandard` is installed on the runner host; OpenAI reasoning arrives encrypted. Requests no trial can be matched to go to `<jobs>/.gateway/unattributed.jsonl`.diff --git a/scripts/live_board.py b/scripts/live_board.pynew file mode 100644index 0000000000..c403cb5742--- /dev/null+++ b/scripts/live_board.py@@ -0,0 +1,430 @@+#!/usr/bin/env python3+"""Live board of batch evaluations: one web page with every trial under Harbor jobs directories and its state, refreshed+every few seconds while the batch runs. A runner-side tool (stdlib only); it only reads.++ scripts/live_board.py [--jobs jobs/agents] [--jobs jobs] [--port N] [--bind 127.0.0.1]+ # prints its address (default: the first free port from 8765); from a laptop: ssh -N -L 8765:127.0.0.1:<port> <host>,+ # then open http://localhost:8765++States: pending (in a batch's plan, not started; scripts/run_agent_batch.sh writes <jobs>/.plan/<batch>.json), setup+(environment and agent install), running (with the agent's time limit), verifying, then success / failed, or aborted+(the job ended without finishing the trial). Per trial: tokens (live from the usage gateway's totals.json when it runs,+else Harbor's count at the end), reward, exception, the replay / live videos, and the tail of the agent's output.++It serves the agent's output and the videos, which can hold the whole conversation: keep it on 127.0.0.1 (the default)+and reach it through an SSH tunnel; bind elsewhere only on a trusted network. There is no login: on a shared host every+local user can open it, and files of a running trial are served before the runner's secret scan has redacted them. On+127.0.0.1 it answers only requests addressed to localhost / 127.0.0.1 / [::1] (a web page cannot reach it through DNS+rebinding).+"""+from __future__ import annotations++import argparse+import datetime as dt+import json+import os+import pathlib+import re+import sys+import threading+import time+import tomllib+import urllib.parse+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++REPO = pathlib.Path(__file__).resolve().parents[1]+SKIP = {"agent", "artifacts", "verifier", "gateway", "sessions", "_logs", "node_modules"}+MEDIA = {".mp4": "video/mp4", ".png": "image/png", ".jpg": "image/jpeg", ".md": "text/plain; charset=utf-8",+ ".json": "application/json", ".txt": "text/plain; charset=utf-8", ".log": "text/plain; charset=utf-8"}+ARGS: argparse.Namespace+LOOPBACK = {"localhost", "127.0.0.1", "::1"}+LOOPBACK_BIND = True # bound to a loopback address: answer only requests addressed to one (main sets it)+CACHE: dict = {"t": 0.0, "state": None}+LOCK = threading.Lock()+++def jload(p: pathlib.Path) -> dict:+ try:+ return json.loads(p.read_text())+ except (OSError, ValueError):+ return {}+++def ts(s: str | None) -> float | None:+ if not s:+ return None+ try:+ t = dt.datetime.fromisoformat(s.replace("Z", "+00:00"))+ except ValueError:+ return None+ return (t if t.tzinfo else t.astimezone()).timestamp() # job-level times are local, trial-level UTC+++_TASKS: dict[str, dict] = {}+++def task_info(task: str) -> dict:+ """Benchmark (the task name's prefix: robocasa-gr1-..., robotwin_2_0_..., kinder-...) and agent time limit."""+ if task not in _TASKS:+ try:+ t = tomllib.loads((REPO / "tasks" / task / "task.toml").read_text())+ except (OSError, ValueError):+ t = {}+ m = re.match(r"(robocasa-gr1|robocasa365|mujoco_playground|robotwin_2_0|[a-z0-9]+)", task)+ bench = m.group(1) if m else task+ _TASKS[task] = {"bench": bench, "timeout": (t.get("agent") or {}).get("timeout_sec")}+ return _TASKS[task]+++def run_id(agent: dict) -> str:+ """<harness>-<model>-<effort> as scripts/run_agent_batch.sh derives it, except that an Azure deployment replica's+ suffix is dropped (gpt-6-luna-3 -> gpt6_luna; claude-fable-5-1 stays fable_5_1)."""+ f = lambda s: re.sub(r"^_+|_+$", "", re.sub(r"[^a-z0-9]+", "_", s.lower()))+ model = (agent.get("model_name") or "").split("/")[-1].lower()+ model = re.sub(r"^(gpt-\d+(?:\.\d+)?-[a-z]+)-\d+$", r"\1", model)+ model = re.sub(r"^gpt-(\d+)", r"gpt\1", re.sub(r"^claude-", "", model))+ effort = (agent.get("kwargs") or {}).get("reasoning_effort") or "default"+ return f"{f(agent.get('name') or '?')}-{f(model)}-{f(effort)}"+++def walk(root: pathlib.Path, depth: int = 0):+ """Job directories (a config.json, no __ in the name) below root."""+ try:+ entries = [e for e in os.scandir(root) if e.is_dir(follow_symlinks=False)]+ except OSError:+ return+ for e in entries:+ if e.name.startswith((".", "_")) or e.name in SKIP: # _superseded/, _logs/: set aside, not shown+ continue+ p = pathlib.Path(e.path)+ if (p / "config.json").exists() and "__" not in e.name:+ yield p+ elif depth < 5:+ yield from walk(p, depth + 1)+++def newest(paths: list[pathlib.Path]) -> float:+ m = 0.0+ for p in paths:+ try:+ m = max(m, p.stat().st_mtime)+ except OSError:+ pass+ return m+++def trial_state(t: pathlib.Path, job: dict, ri: int, root: pathlib.Path, plan_run: str | None) -> dict:+ cfg = jload(t / "config.json")+ r = jload(t / "result.json")+ task = pathlib.Path((cfg.get("task") or {}).get("path") or t.name.split("__")[0]).name+ info = task_info(task)+ gw = jload(t / "gateway" / "totals.json")+ ae = r.get("agent_execution") or {}+ ver = r.get("verifier") or {}+ rewards = (r.get("verifier_result") or {}).get("rewards") or {}+ reward = rewards.get("reward")+ if reward is None and (t / "verifier" / "reward.txt").exists():+ try:+ reward = float((t / "verifier" / "reward.txt").read_text().strip())+ except ValueError:+ pass+ agent_dir = t / "agent"+ agent_files = list(agent_dir.glob("*.txt")) if agent_dir.is_dir() else []+ now = time.time()+ a0, a1 = ts(ae.get("started_at")), ts(ae.get("finished_at"))+ if r.get("finished_at"):+ state = "success" if reward is not None and reward >= 1 and not gw.get("over_token_budget") else "failed"+ elif job.get("finished_at"):+ state = "aborted"+ elif ver.get("started_at") or (a1 and not ver.get("finished_at")):+ state = "verifying"+ elif a0 or agent_files:+ state = "running"+ else:+ state = "setup"+ if a0 is None and agent_files and state == "running":+ a0 = min(p.stat().st_ctime for p in agent_files)+ ar = r.get("agent_result") or {}+ tok_in = gw.get("input") if gw else ar.get("n_input_tokens")+ tok_out = gw.get("output") if gw else ar.get("n_output_tokens")+ media = []+ for d in ("verifier", "artifacts/data"):+ for p in sorted((t / d).glob("*")) if (t / d).is_dir() else []:+ if p.suffix in (".mp4", ".png") and p.name not in {m.split("/")[-1] for m in media}:+ media.append(str(p.relative_to(root)))+ exc = (r.get("exception_info") or {}).get("exception_type") or ""+ return {+ "id": f"{ri}:{t.relative_to(root)}", "task": task, "bench": info["bench"],+ "mode": "standard" if task.endswith("-standard") else "privileged",+ "run": plan_run or run_id((cfg.get("agent") or {})), "job": job.get("job_name") or t.parent.name,+ "trial": t.name, "state": state, "reward": reward, "exception": exc,+ "agent_start": a0, "agent_end": a1 or (now if state == "running" else None), "timeout": info["timeout"],+ "tok_in": tok_in, "tok_out": tok_out, "requests": gw.get("requests"), "gateway": bool(gw),+ "over_budget": bool(gw.get("over_token_budget")),+ "updated": newest([t / "result.json", t / "trial.log", t / "gateway" / "totals.json", *agent_files]),+ "media": media, "summary": str((t.parent / "summary.md").relative_to(root)) if (t.parent / "summary.md").exists() else None,+ }+++def collect() -> dict:+ trials, plans = [], []+ for ri, root in enumerate(ARGS.jobs):+ plan_of: dict[str, str] = {}+ for pf in sorted((root / ".plan").glob("*.json")) if (root / ".plan").is_dir() else []:+ p = jload(pf)+ for j in p.get("jobs") or []:+ plan_of[j["job"]] = p.get("run_id")+ plans.append((ri, p, j))+ for jd in walk(root):+ jc = jload(jd / "config.json")+ jr = jload(jd / "result.json")+ name = jc.get("job_name") or jd.name+ try:+ tds = sorted(jd.iterdir())+ except OSError: # the job directory moved away (archived) during the scan+ continue+ for t in tds:+ try:+ if t.is_dir() and "__" in t.name and (t / "config.json").exists():+ trials.append(trial_state(t, {"job_name": name, **jr}, ri, root,+ plan_of.get(name) or plan_of.get(jd.name)))+ except OSError: # a file vanished mid-scan: the next refresh picks the trial up+ continue+ started = {(t["job"], t["task"]) for t in trials}+ for ri, p, j in plans:+ if (j["job"], j["task"]) in started:+ continue+ info = task_info(j["task"])+ trials.append({"id": None, "task": j["task"], "bench": info["bench"], "mode": j.get("mode"),+ "run": p.get("run_id"), "job": j["job"], "trial": None, "state": "pending",+ "timeout": info["timeout"], "batch": p.get("batch")})+ return {"now": time.time(), "roots": [str(r) for r in ARGS.jobs], "trials": trials}+++def state() -> dict:+ with LOCK:+ if CACHE["state"] is None or time.time() - CACHE["t"] > 3:+ CACHE["state"], CACHE["t"] = collect(), time.time()+ return CACHE["state"]+++def resolve(tid: str) -> tuple[pathlib.Path, pathlib.Path] | None:+ ri, _, rel = tid.partition(":")+ try:+ root = ARGS.jobs[int(ri)]+ except (ValueError, IndexError):+ return None+ p = (root / rel).resolve()+ return (root, p) if p.is_relative_to(root.resolve()) and p.exists() else None+++def tail(p: pathlib.Path, n: int = 80, maxb: int = 64_000) -> str:+ try:+ with open(p, "rb") as f:+ f.seek(0, 2)+ f.seek(max(0, f.tell() - maxb))+ return "\n".join(f.read().decode("utf-8", "replace").splitlines()[-n:])+ except OSError:+ return ""+++def detail(tid: str) -> dict:+ rp = resolve(tid)+ if rp is None:+ return {"error": "no such trial"}+ _, t = rp+ r = jload(t / "result.json")+ out = sorted((t / "agent").glob("*.txt"), key=lambda p: p.stat().st_size, reverse=True) if (t / "agent").is_dir() else []+ exc = r.get("exception_info") or {}+ return {"reward": jload(t / "verifier" / "reward.json") or (r.get("verifier_result") or {}).get("rewards"),+ "gateway": jload(t / "gateway" / "totals.json"),+ "exception": ((exc.get("exception_type") or "") + ": " + (exc.get("exception_message") or "")[:2000]) if exc else "",+ "agent_log": out[0].name if out else None, "agent_tail": tail(out[0]) if out else "",+ "trial_log_tail": tail(t / "trial.log", 25)}+++PAGE = r"""<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">+<title>Live Board</title><style>+:root{--bg:#f7f7f5;--fg:#1d1d1b;--mut:#6b6b66;--card:#fff;--line:#e3e2dd;--ok:#1f8a4c;--bad:#c2410c;--run:#2563eb;--ver:#7c3aed;--pend:#9ca3af;--warn:#b45309;--on:#fff}+@media (prefers-color-scheme:dark){:root:not([data-theme=light]){--bg:#141413;--fg:#ecebe6;--mut:#9b9a93;--card:#1e1e1c;--line:#33332f;--ok:#4ade80;--bad:#fb923c;--run:#60a5fa;--ver:#a78bfa;--pend:#6b7280;--warn:#fbbf24;--on:#111}}+:root[data-theme=dark]{--bg:#141413;--fg:#ecebe6;--mut:#9b9a93;--card:#1e1e1c;--line:#33332f;--ok:#4ade80;--bad:#fb923c;--run:#60a5fa;--ver:#a78bfa;--pend:#6b7280;--warn:#fbbf24;--on:#111}+*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--fg);font:14px/1.45 system-ui,-apple-system,Segoe UI,sans-serif}+header{padding:16px;display:flex;flex-wrap:wrap;gap:12px;align-items:baseline}h1{font-size:18px;margin:0}.mut{color:var(--mut)}+main{padding:0 16px 32px}.cards{display:grid;grid-template-columns:repeat(auto-fill,minmax(260px,1fr));gap:12px;margin-bottom:16px}+.card{background:var(--card);border:1px solid var(--line);border-radius:10px;padding:12px}.card h2{font-size:14px;margin:0 0 6px;font-family:ui-monospace,monospace}+.bar{display:flex;height:8px;border-radius:4px;overflow:hidden;background:var(--line);margin:8px 0}.bar span{display:block}+.legend{display:flex;flex-wrap:wrap;gap:10px;font-size:12px}.dot{display:inline-block;width:8px;height:8px;border-radius:50%;margin-right:4px}+.tools{display:flex;flex-wrap:wrap;gap:8px;margin-bottom:10px}select,input,button{font:inherit;color:var(--fg);background:var(--card);border:1px solid var(--line);border-radius:6px;padding:4px 8px}+.wrap{overflow-x:auto;background:var(--card);border:1px solid var(--line);border-radius:10px}table{border-collapse:collapse;width:100%}+th,td{padding:6px 10px;border-bottom:1px solid var(--line);text-align:left;white-space:nowrap}th{font-size:12px;color:var(--mut);font-weight:600;position:sticky;top:0;background:var(--card)}+tr.row{cursor:pointer}tr.row:hover{background:color-mix(in srgb,var(--run) 6%,transparent)}td.num{text-align:right;font-variant-numeric:tabular-nums}+.st{display:inline-block;padding:1px 8px;border-radius:999px;font-size:12px;font-weight:600;color:var(--on)}+.st.success{background:var(--ok)}.st.failed{background:var(--bad)}.st.running{background:var(--run)}.st.verifying{background:var(--ver)}+.st.setup{background:var(--warn)}.st.pending{background:var(--pend)}.st.aborted{background:#6b7280}+.prog{width:90px;height:6px;background:var(--line);border-radius:3px;display:inline-block;vertical-align:middle;margin-right:6px}.prog i{display:block;height:100%;background:var(--run);border-radius:3px}+.mx td,.mx th{text-align:center}.mx td:first-child,.mx th:first-child{text-align:left}+#drawer{position:fixed;top:0;right:0;height:100%;width:min(720px,100%);background:var(--card);border-left:1px solid var(--line);transform:translateX(100%);visibility:hidden;transition:transform .2s,visibility .2s;overflow-y:auto;padding:16px;z-index:5}+#drawer.open{transform:none;visibility:visible;box-shadow:-8px 0 24px #0003}#drawer pre{background:var(--bg);border:1px solid var(--line);border-radius:6px;padding:8px;overflow-x:auto;font-size:12px;white-space:pre-wrap;word-break:break-word;max-height:420px}+video,img{max-width:100%;border-radius:6px;border:1px solid var(--line)}.close{float:right}+</style></head><body>+<header><h1>Live Board</h1><span class="mut" id="meta"></span></header>+<main><div class="cards" id="cards"></div>+<div class="tools"><select id="run"><option value="">all runs</option></select><select id="st"><option value="">all states</option></select>+<select id="mode"><option value="">both modes</option><option>privileged</option><option>standard</option></select>+<input id="q" placeholder="filter task / benchmark"><button id="view">matrix view</button></div>+<div class="wrap" id="table"></div></main>+<aside id="drawer"></aside>+<script>+const ORDER=["running","verifying","setup","pending","aborted","failed","success"],COL={success:"--ok",failed:"--bad",running:"--run",verifying:"--ver",setup:"--warn",pending:"--pend",aborted:"--mut"};+const LABEL={success:"✓ success",failed:"✗ failed",running:"running",verifying:"verifying",setup:"setup",pending:"pending",aborted:"aborted"};+let S=null,matrix=false,open=null;const $=id=>document.getElementById(id);+try{matrix=localStorage.getItem("lb.matrix")==="1"}catch(e){}+const fmtT=n=>n==null?"":n>=1e6?(n/1e6).toFixed(1)+"M":n>=1e3?(n/1e3).toFixed(0)+"k":String(n);+const fmtM=s=>s==null?"":s<3600?Math.floor(s/60)+"m":(s/3600).toFixed(1)+"h";+const esc=s=>String(s??"").replace(/[&<>"]/g,c=>({"&":"&","<":"<",">":">",'"':"""}[c]));+const st=s=>`<span class="st ${s}">${LABEL[s]||s}</span>`;+function filt(){const r=$("run").value,s=$("st").value,m=$("mode").value,q=$("q").value.toLowerCase();+ return S.trials.filter(t=>(!r||t.run===r)&&(!s||t.state===s)&&(!m||t.mode===m)&&(!q||(t.task+" "+t.bench).toLowerCase().includes(q)))}+function cards(){const by={};for(const t of S.trials)(by[t.run]??={})[t.state]=((by[t.run]??{})[t.state]||0)+1;+ $("cards").innerHTML=Object.keys(by).sort().map(r=>{const c=by[r],n=Object.values(c).reduce((a,b)=>a+b,0),done=(c.success||0)+(c.failed||0)+(c.aborted||0);+ return `<div class="card"><h2>${esc(r)}</h2><div><b>${c.success||0}</b> / ${done} succeeded · ${done} / ${n} done</div>+ <div class="bar">${ORDER.slice().reverse().map(s=>c[s]?`<span style="width:${100*c[s]/n}%;background:var(${COL[s]})"></span>`:"").join("")}</div>+ <div class="legend">${ORDER.filter(s=>c[s]).map(s=>`<span><i class="dot" style="background:var(${COL[s]})"></i>${LABEL[s]} ${c[s]}</span>`).join("")}</div></div>`}).join("")||'<div class="mut">no trials yet</div>'}+function selects(){const runs=[...new Set(S.trials.map(t=>t.run))].sort(),r=$("run"),v=r.value;+ r.innerHTML='<option value="">all runs</option>'+runs.map(x=>`<option${x===v?" selected":""}>${esc(x)}</option>`).join("");+ const s=$("st"),sv=s.value;s.innerHTML='<option value="">all states</option>'+ORDER.map(x=>`<option value="${x}"${x===sv?" selected":""}>${LABEL[x]}</option>`).join("")}+function elapsed(t){if(t.agent_start==null)return"";const e=(t.agent_end??S.now)-t.agent_start;+ if(t.state==="running"&&t.timeout){const p=Math.min(100,100*e/t.timeout);return `<span class="prog"><i style="width:${p}%"></i></span>${fmtM(e)} / ${fmtM(t.timeout)}`}return fmtM(e)}+function table(){const rows=filt().sort((a,b)=>ORDER.indexOf(a.state)-ORDER.indexOf(b.state)||a.task.localeCompare(b.task)||a.run.localeCompare(b.run));+ $("view").textContent=matrix?"list view":"matrix view";+ if(matrix){const runs=[...new Set(rows.map(t=>t.run))].sort(),tasks=[...new Set(rows.map(t=>t.task))].sort(),cell={};+ for(const t of rows){const k=t.task+"|"+t.run,o=cell[k];if(!o||ORDER.indexOf(t.state)<ORDER.indexOf(o.state))cell[k]=t}+ $("table").innerHTML=`<table class="mx"><tr><th>task</th>${runs.map(r=>`<th>${esc(r)}</th>`).join("")}</tr>`+tasks.map(k=>`<tr><td>${esc(k)}</td>${runs.map(r=>{const t=cell[k+"|"+r];+ return `<td>${t?`<span class="${t.id?"row":""}" data-id="${esc(t.id||"")}" style="cursor:${t.id?"pointer":"default"}">${st(t.state)}</span>`:""}</td>`}).join("")}</tr>`).join("")+"</table>"}+ else $("table").innerHTML=`<table><tr><th>state</th><th>task</th><th>benchmark</th><th>mode</th><th>run</th><th>agent time</th><th class="num">input tok</th><th class="num">output tok</th><th class="num">requests</th><th class="num">reward</th><th>exception</th></tr>`++ rows.map(t=>`<tr class="${t.id?"row":""}" data-id="${esc(t.id||"")}"><td>${st(t.state)}${t.over_budget?' <span class="st failed">over budget</span>':""}</td><td>${esc(t.task)}</td><td>${esc(t.bench)}</td><td>${esc(t.mode)}</td><td class="mut">${esc(t.run)}</td>+ <td>${elapsed(t)}</td><td class="num">${fmtT(t.tok_in)}</td><td class="num">${fmtT(t.tok_out)}</td><td class="num">${t.requests??""}</td><td class="num">${t.reward??""}</td><td class="mut">${esc((t.exception||"").replace(/Error$/,""))}</td></tr>`).join("")+"</table>";+ for(const el of document.querySelectorAll("[data-id]"))if(el.dataset.id)el.onclick=()=>show(el.dataset.id)}+async function show(id){open=id;const t=S.trials.find(x=>x.id===id);if(!t)return;+ const d=await (await fetch("api/trial?id="+encodeURIComponent(id))).json(),f=p=>"file?p="+encodeURIComponent(t.id.split(":")[0]+":"+p);+ const vids=t.media.filter(m=>m.endsWith(".mp4")),imgs=t.media.filter(m=>m.endsWith(".png"));+ $("drawer").innerHTML=`<button class="close" onclick="closeD()">close</button><h2 style="margin-top:0">${esc(t.task)}</h2>+ <p>${st(t.state)} <span class="mut">${esc(t.run)} · ${esc(t.trial)}</span></p><p>agent time ${elapsed(t)} · tokens ${fmtT(t.tok_in)} in / ${fmtT(t.tok_out)} out${t.requests?` · ${t.requests} requests`:""}${t.summary?` · <a href="${f(t.summary)}" target="_blank">summary.md</a>`:""}</p>+ ${vids.map(v=>`<p class="mut">${esc(v.split("/").pop())}</p><video src="${f(v)}" controls preload="metadata"></video>`).join("")}${!vids.length?imgs.map(v=>`<img src="${f(v)}">`).join(""):""}+ ${d.exception?`<h3>exception</h3><pre>${esc(d.exception)}</pre>`:""}${d.reward&&Object.keys(d.reward).length?`<h3>reward</h3><pre>${esc(JSON.stringify(d.reward,null,1))}</pre>`:""}+ ${d.agent_tail?`<h3>agent output (${esc(d.agent_log)}, last lines)</h3><pre>${esc(d.agent_tail)}</pre>`:""}${d.trial_log_tail?`<h3>trial.log</h3><pre>${esc(d.trial_log_tail)}</pre>`:""}`;+ $("drawer").classList.add("open")}+function closeD(){open=null;$("drawer").classList.remove("open")}+async function tick(){try{S=await (await fetch("api/state")).json();+ const c={};for(const t of S.trials)c[t.state]=(c[t.state]||0)+1;+ $("meta").textContent=`${S.trials.length} trials · ${ORDER.filter(s=>c[s]).map(s=>c[s]+" "+s).join(" · ")} · updated ${new Date(S.now*1000).toLocaleTimeString()}`;+ cards();selects();table();const t=open&&S.trials.find(x=>x.id===open);if(t&&["running","verifying","setup"].includes(t.state))show(open)}catch(e){$("meta").textContent="board unreachable: "+e}}+for(const id of["run","st","mode","q"])$(id).addEventListener("input",table);+$("view").onclick=()=>{matrix=!matrix;try{localStorage.setItem("lb.matrix",matrix?"1":"0")}catch(e){}table()};+document.addEventListener("keydown",e=>{if(e.key==="Escape")closeD()});tick();setInterval(tick,5000);+</script></body></html>"""+++class Handler(BaseHTTPRequestHandler):+ def log_message(self, *a):+ pass++ def send(self, code: int, body: bytes, ctype: str, extra: dict | None = None) -> None:+ self.send_response(code)+ self.send_header("Content-Type", ctype)+ self.send_header("Content-Length", str(len(body)))+ self.send_header("Cache-Control", "no-store")+ for k, v in (extra or {}).items():+ self.send_header(k, v)+ self.end_headers()+ if self.command != "HEAD":+ self.wfile.write(body)++ def do_HEAD(self):+ self.do_GET()++ def do_GET(self):+ host = (self.headers.get("Host") or "").lower()+ host = host[1:].split("]")[0] if host.startswith("[") else host.rsplit(":", 1)[0]+ if LOOPBACK_BIND and host not in LOOPBACK:+ return self.send(403, b"use http://localhost:<port>/ (or 127.0.0.1)", "text/plain")+ u = urllib.parse.urlparse(self.path)+ q = urllib.parse.parse_qs(u.query)+ if u.path == "/":+ return self.send(200, PAGE.encode(), "text/html; charset=utf-8")+ if u.path == "/api/state":+ return self.send(200, json.dumps(state()).encode(), "application/json")+ if u.path == "/api/trial":+ return self.send(200, json.dumps(detail(q.get("id", [""])[0])).encode(), "application/json")+ if u.path == "/file":+ rp = resolve(q.get("p", [""])[0])+ if rp is None or rp[1].suffix not in MEDIA or not rp[1].is_file():+ return self.send(404, b"not found", "text/plain")+ return self.file(rp[1])+ self.send(404, b"not found", "text/plain")++ def file(self, p: pathlib.Path) -> None:+ """The file, or one byte range of it (video seeking), streamed in 1 MB pieces."""+ size = p.stat().st_size+ a, b, partial = 0, size - 1, False+ m = re.match(r"bytes=(\d*)-(\d*)$", self.headers.get("Range", "").strip())+ if m and (m.group(1) or m.group(2)):+ if m.group(1):+ a, b = int(m.group(1)), int(m.group(2)) if m.group(2) else size - 1+ else:+ a = max(0, size - int(m.group(2))) if int(m.group(2)) else size+ b, partial = min(b, size - 1), True+ if a > b:+ return self.send(416, b"", "text/plain", {"Content-Range": f"bytes */{size}"})+ self.send_response(206 if partial else 200)+ self.send_header("Content-Type", MEDIA[p.suffix])+ self.send_header("Content-Length", str(b - a + 1))+ self.send_header("Cache-Control", "no-store")+ self.send_header("Accept-Ranges", "bytes")+ if partial:+ self.send_header("Content-Range", f"bytes {a}-{b}/{size}")+ self.end_headers()+ if self.command == "HEAD":+ return+ with open(p, "rb") as f:+ f.seek(a)+ left = b - a + 1+ while left > 0:+ chunk = f.read(min(left, 1 << 20))+ if not chunk:+ break+ try:+ self.wfile.write(chunk)+ except OSError: # the browser hung up (seeking elsewhere)+ return+ left -= len(chunk)+++def main() -> int:+ global ARGS, LOOPBACK_BIND+ p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--jobs", type=pathlib.Path, action="append", help="a Harbor -o directory (repeatable; default jobs/agents)")+ p.add_argument("--port", type=int, help="default: the first free port from 8765")+ p.add_argument("--bind", default="127.0.0.1")+ p.add_argument("--once", action="store_true", help="print the state as JSON and exit")+ ARGS = p.parse_args()+ ARGS.jobs = [j.resolve() for j in (ARGS.jobs or [REPO / "jobs" / "agents"])]+ LOOPBACK_BIND = ARGS.bind in LOOPBACK or ARGS.bind.startswith("127.")+ if ARGS.once:+ json.dump(collect(), sys.stdout, indent=1)+ return 0+ for port in [ARGS.port] if ARGS.port is not None else range(8765, 8865):+ try:+ srv = ThreadingHTTPServer((ARGS.bind, port), Handler)+ break+ except OSError as e:+ if ARGS.port is not None or port == 8864:+ sys.exit(f"cannot listen on {ARGS.bind}:{port}: {e}")+ srv.daemon_threads = True+ print(f"live board on http://{ARGS.bind}:{srv.server_address[1]}/ for {', '.join(map(str, ARGS.jobs))}", flush=True)+ srv.serve_forever()+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/run_agent_batch.sh b/scripts/run_agent_batch.shindex 01e6a72753..76ed0bc678 100755--- a/scripts/run_agent_batch.sh+++ b/scripts/run_agent_batch.sh@@ -39,7 +39,8 @@# them unchanged (the agent keeps its own credentials) and records each trial's tokens per request in# <trial>/gateway/ (usage.jsonl, totals.json)# GATEWAY_RAW 1 (default, takes effect with GATEWAY=1) = the gateway also keeps every request and response body-# (<trial>/gateway/raw/, gzipped; private: it holds the whole conversation); 0 = token usage only+# (<trial>/gateway/raw.jsonl.xz, deduplicated, see scripts/usage_gateway/rawlog.py; private: it holds the+# whole conversation); 0 = token usage only# GATEWAY_BIND the docker bridge address the trial containers reach the host by (172.17.0.1)# GATEWAY_EXTRA_HOSTS more model API host patterns the gateway forwards to, space-separated (e.g. "*.services.ai.azure.com")# TOKEN_BUDGET_IN / TOKEN_BUDGET_OUT per-trial token budget the gateway enforces (100000000 / 10000000, PROTOCOL@@ -48,7 +49,8 @@# append-only into ${JOBS}_mirror/ (scripts/log_mirror.py; root-owned: neither the agent nor the host user can delete it),# and scripts/mem_guard.py keeps 2 GiB of host memory available. After each job: a key scan with redaction# (scripts/scan_secrets.py), the task results, the summary table $JOBS/<batch>.summary.md, and a lane stop on structured-# model-account failures (scripts/account_failure_check.py). Progress: $JOBS/<batch>.log.+# model-account failures (scripts/account_failure_check.py). Progress: $JOBS/<batch>.log; in a browser, every trial's+# state (pending / running / done) with scripts/live_board.py --jobs $JOBS.set -uR="$(cd "$(dirname "$0")/.." && pwd)"[ $# -ge 2 ] || { sed -n 2,40p "$0"; exit 2; }@@ -94,6 +96,18 @@ LOG="$JOBS/$batch.log"log() { echo "[$(date '+%F %T')] $*" | tee -a "$LOG"; }if [ -n "$ENV_FILE" ] && [ ! -f "$ENV_FILE" ]; then log "no env file $ENV_FILE"; exit 2; filog "runner $$: $AGENT $MODEL effort $EFFORT, run id $RUN_ID, modes $MODES, web search $WEB_SEARCH"+# the batch's plan, for scripts/live_board.py: every job this command runs, "pending" there until it starts+mkdir -p "$JOBS/.plan"+python3 - "$R" "$JOBS/.plan/$batch.json" "$batch" "$RUN_ID" "$SEQ" "$MODES" "${tasks[@]}" <<'PY'+import json, os, pathlib, sys+repo, out, batch, run_id, seq, modes, *tasks = sys.argv[1:]+name = lambda m, t: f"{t}-standard" if m in ("standard", "limited") else f"{t}-privileged"+jobs = [{"job": f"{batch}-{m}-{t}" if seq == "1" else f"{batch}-{m}", "task": name(m, t), "mode": m}+ for t in tasks for m in modes.split() if (pathlib.Path(repo) / "tasks" / name(m, t)).is_dir()]+tmp = pathlib.Path(f"{out}.{os.getpid()}")+tmp.write_text(json.dumps({"batch": batch, "run_id": run_id, "jobs": jobs}, indent=1))+tmp.replace(out)+PY# append-only mirror of every trial's agent/ directory, out of every trial container's reachif [ "$LOG_MIRROR" = 1 ] && ! docker ps --format '{{.Names}}' | grep -qx rcb-log-mirror; then@@ -159,8 +173,9 @@ wait_for() { # <mode>: block until every WAIT_FOR file of this mode existsdone}-finish_job() { # <job>: no key in any saved file (+redact); the task results (D3)+finish_job() { # <job>: the raw logs compacted; no key in any saved file (+redact); the task results (D3)[ -d "$JOBS/$1" ] || return 0+ [ "$GATEWAY" = 1 ] && [ "$GATEWAY_RAW" = 1 ] && python3 "$R/scripts/usage_gateway/rawlog.py" compact "$JOBS/$1" 2>&1 | sed "s/^/[$1] /" | tee -a "$LOG"python3 "$R/scripts/scan_secrets.py" "$JOBS/$1" "${SCAN[@]}" --redact 2>&1 | tail -3 | sed "s/^/[$1] /" | tee -a "$LOG"[ -n "$RESULTS" ] && python3 "$R/scripts/write_task_results.py" "$JOBS" "$1" "$RUN_ID" "$LABEL" --out "$RESULTS/$batch" 2>&1 | tail -3 | sed "s/^/[$1] /" | tee -a "$LOG"}diff --git a/scripts/scan_secrets.py b/scripts/scan_secrets.pyindex f1d4908359..64f19203b4 100755--- a/scripts/scan_secrets.py+++ b/scripts/scan_secrets.py@@ -7,13 +7,42 @@ Loads the secrets INSIDE this process and never prints them: the named variablesagents read, VARS below) from the .env files (default: the repo's ../.env) or, where a file does not set one, from thisprocess's environment, and, with --codex-auth, the tokens of a Codex ChatGPT login (auth.json). Values shorter than 16characters are not treated as secrets. It reports only which files contain one (and how many times). With --redact every-occurrence is replaced with [REDACTED-<NAME>] in place. Binary files (videos, npz) are scanned too, as bytes."""-import argparse, json, os, pathlib, sys+occurrence is replaced with [REDACTED-<NAME>] in place. Binary files (videos, npz) are scanned too, as bytes; .gz and .xz+files (the usage gateway's raw logs) decompressed. A damaged .gz / .xz (truncated, corrupt) is scanned as far as it+decompresses, and as bytes; it is reported as DAMAGED and counted in the last line, and --redact rewrites it with only+its readable part."""+import argparse, gzip, json, lzma, os, pathlib, sys, zlib++MAGIC = {".gz": b"\x1f\x8b", ".xz": b"\xfd7zXZ\x00"}+DECOMPRESSOR = {".gz": lambda: zlib.decompressobj(zlib.MAX_WBITS | 16), ".xz": lzma.LZMADecompressor}VARS = ["OPENROUTER_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "CLAUDE_CODE_OAUTH_TOKEN","AZURE_API_KEY", "AZURE_OPENAI_API_KEY"]+def decompress(data: bytes, suffix: str) -> tuple[bytes, bool]:+ """(what decompresses, whether all of it did): every stream of a multi-stream file (the gateway's live raw log is+ one gzip stream per request), fed in 64 kB pieces; after a damaged stream it resumes at the next stream header."""+ out, rest, whole = [], data, True+ while rest:+ d, i = DECOMPRESSOR[suffix](), 0+ try:+ while not d.eof and i < len(rest):+ out.append(d.decompress(rest[i:i + (1 << 16)]))+ i += 1 << 16+ except (zlib.error, lzma.LZMAError):+ pass+ if not d.eof:+ whole = False+ j = rest.find(MAGIC[suffix], 1)+ if j < 0:+ break+ rest = rest[j:]+ continue+ rest = (d.unused_data + rest[i:]).lstrip(b"\0") # the next stream (zero padding between streams allowed)+ return b"".join(out), whole++def load_env(path: str) -> dict:out = {}for line in pathlib.Path(path).read_text().splitlines():@@ -50,22 +79,31 @@ for f in a.codex_auth:secrets["CODEX_OPENAI_API_KEY"] = d["OPENAI_API_KEY"].encode()if not secrets:sys.exit("no secret loaded (check --env-file / --var / --codex-auth); nothing to scan for")-hits = scanned = 0+hits = scanned = damaged = 0for p in pathlib.Path(a.dir).rglob("*"):if not p.is_file() or p.is_symlink():continuescanned += 1try:- data = p.read_bytes()+ raw = p.read_bytes()except OSError:continue- found = {n: data.count(s) for n, s in secrets.items() if s in data}+ codec, data, whole = None, raw, True+ if p.suffix in MAGIC and raw.startswith(MAGIC[p.suffix]): # otherwise not really compressed: scan it as bytes+ codec = p.suffix+ data, whole = decompress(raw, codec)+ if not whole:+ damaged += 1+ print(f"DAMAGED {codec} (scanned the {len(data)} bytes that decompress, and the file as bytes): {p}")+ found = {n: data.count(s) + (0 if whole else raw.count(s)) for n, s in secrets.items()}+ found = {n: c for n, c in found.items() if c}if found:hits += 1print(f"FOUND {', '.join(f'{n} x{c}' for n, c in found.items())}: {p}")if a.redact:for n in found:data = data.replace(secrets[n], f"[REDACTED-{n}]".encode())- p.write_bytes(data)- print(" redacted")-print(f"scanned {scanned} files under {a.dir} for {len(secrets)} secret(s) ({', '.join(sorted(secrets))}): {hits} contain one")+ p.write_bytes(lzma.compress(data, preset=9) if codec == ".xz" else gzip.compress(data) if codec else data)+ print(" redacted" + ("" if whole else " (only the readable part is kept)"))+print(f"scanned {scanned} files under {a.dir} for {len(secrets)} secret(s) ({', '.join(sorted(secrets))}): "+ f"{hits} contain one" + (f", {damaged} damaged compressed file(s)" if damaged else ""))diff --git a/scripts/usage_gateway/gateway.py b/scripts/usage_gateway/gateway.pyindex 98dd9ce0fb..075001fd04 100755--- a/scripts/usage_gateway/gateway.py+++ b/scripts/usage_gateway/gateway.py@@ -16,8 +16,11 @@ own agent/ and verifier/ (not inside agent/, which the agent's container mountsusage.jsonl one line per request: time, duration, host, path, status, model, normalized and raw usagetotals.json running totals (rewritten after every request): requests, input / cached / output tokens- raw/NNNNNN.json.gz with --raw: the request body and the response body (SSE text) as sent and received;- credentials, cookies and account ids are removed from the recorded headers+ raw.jsonl.gz with --raw: every request body and response body (SSE text) as sent and received, deduplicated+ (rawlog.py: each conversation item is stored once, a request as references to them; lossless);+ credentials, cookies and account ids are removed from the recorded headers. usage.jsonl's "raw"+ is the request's n there. `rawlog.py compact` (run_agent_batch.sh, after each job) turns it into+ raw.jsonl.xz; `rawlog.py cat` prints the requests in fullToken budget (PROTOCOL §5, D10; on by default when the gateway runs): once a trial's totals reach --budget-input inputor --budget-output output tokens, totals.json gets over_token_budget = 1 and every further model request of that trial@@ -47,6 +50,9 @@ import threadingimport timefrom http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))+import rawlog # noqa: E402+ROOT = pathlib.Path(__file__).resolve().parents[2]@@ -196,6 +202,7 @@ class Trials:ARGS: argparse.NamespaceTRIALS: TrialsSEQ = {"n": 0}+RAWLOGS: dict[pathlib.Path, rawlog.Writer] = {} # per trial; a restarted gateway resumes from the filedef log(msg: str) -> None:@@ -214,11 +221,11 @@ def record(trial: pathlib.Path | None, rec: dict, raw: dict | None) -> None:d = trial / "gateway"d.mkdir(exist_ok=True)if raw is not None:- (d / "raw").mkdir(exist_ok=True)- n = sum(1 for _ in (d / "raw").glob("*.json.gz")) + 1- rec["raw_file"] = f"raw/{n:06d}.json.gz"- with gzip.open(d / rec["raw_file"], "wt", encoding="utf-8") as f:- json.dump(raw, f)+ if d not in RAWLOGS:+ if len(RAWLOGS) >= 256: # finished trials: drop the oldest table+ del RAWLOGS[next(iter(RAWLOGS))]+ RAWLOGS[d] = rawlog.Writer(d / rawlog.NAME)+ rec["raw"] = RAWLOGS[d].append(raw)with open(d / "usage.jsonl", "a") as f:f.write(json.dumps(rec) + "\n")tp = d / "totals.json"@@ -366,7 +373,7 @@ def main() -> int:p.add_argument("--jobs", type=pathlib.Path, required=True, help="Harbor's -o directory")p.add_argument("--bind", default="172.17.0.1", help="the docker bridge address containers reach the host by")p.add_argument("--port", type=int, default=0, help="0 = any free port (written to --ready-file)")- p.add_argument("--raw", action="store_true", help="also record request and response bodies (raw/)")+ p.add_argument("--raw", action="store_true", help="also record request and response bodies (raw.jsonl.gz)")p.add_argument("--ready-file", type=pathlib.Path, help="written with the port once the gateway listens")p.add_argument("--timeout", type=float, default=900.0, help="upstream socket timeout, seconds")p.add_argument("--extra-host", action="append", default=[], help="another model API host pattern to forward to (repeatable)")diff --git a/scripts/usage_gateway/rawlog.py b/scripts/usage_gateway/rawlog.pynew file mode 100644index 0000000000..aaedbe5dda--- /dev/null+++ b/scripts/usage_gateway/rawlog.py@@ -0,0 +1,356 @@+#!/usr/bin/env python3+"""Deduplicated raw log of a trial's model requests: <trial>/gateway/raw.jsonl.gz (raw.jsonl.xz once compacted),+written by gateway.py --raw.++An agent resends its whole conversation with every request, so one request is mostly the previous one: in a codex+trial ~98% of a request's `input` items are byte-identical to items it already sent, and recording every request body+in full grows quadratically (one 2,000-request trial: 5.9 GB of raw/NNNNNN.json.gz). Like the harnesses' own session+files (codex's rollout-*.jsonl, Claude Code's session .jsonl), which store every conversation item once, this log stores+every distinct piece of a request body once and each request as references to those pieces. It stays lossless: every+request body and response is reconstructed byte for byte.++Format: one JSON object per line. While the trial runs, the gateway appends them to raw.jsonl.gz as gzip members (one+stream to zcat). A crash mid-write loses only that append: readers stop before an incomplete member, and a restarted+gateway first cuts the file back to its last complete member, so nothing is appended behind a broken one. Each member is+compressed on its own, so after the trial `compact` rewrites the log as one xz stream (raw.jsonl.xz, LZMA's 64 MB window+also finds the repeats between requests, e.g. the request settings every response echoes): one 1,994-request trial is+5.9 GB as raw/, 66 MB as raw.jsonl.gz, 32 MB as raw.jsonl.xz. Both are plain compressed JSON Lines (zcat / xzcat).++ {"item": 7, "v": <JSON value>} a body piece (a list element or a large field), the first time it is sent+ {"n": 12, "request": {...}, "response": {...}}+ request n (1-based, the n of usage.jsonl's "raw"), with headers and the+ response as recorded. Its body is either "body" (text, as before) or+ "body_parts": [[key, "v", value] | [key, "i", item] | [key, "l", ranges],+ ...] in key order plus "body_ascii" (how the JSON was serialized), where+ ranges lists item ids, a run a..b written [a, b].++A JSON body is split into parts only when re-serializing it reproduces the text exactly (compact separators, as codex,+Claude Code and the OpenAI / Anthropic SDKs send); anything else is kept as text.++ rawlog.py cat <gateway dir> [-n N] print the records in full ({"n", "request", "response"}), one per line+ rawlog.py compact <dir>... raw.jsonl.gz -> raw.jsonl.xz for every gateway/ below <dir> (finished trials)+ rawlog.py pack <dir>... [--delete] legacy raw/NNNNNN.json.gz -> raw.jsonl.xz; checks every record round-trips,+ then (--delete) removes raw/+ rawlog.py verify <dir>... decode every log below <dir>; compare with raw/ where it still exists+"""+from __future__ import annotations++import argparse+import gzip+import hashlib+import json+import lzma+import pathlib+import sys+import zlib+from collections.abc import Iterator++NAME = "raw.jsonl.gz" # live: gzip members appended by the gateway+PACKED = "raw.jsonl.xz" # compacted: one xz stream+XZ_PRESET = 9 # 64 MB dictionary; ~0.7 GB of memory while compressing+INLINE_MAX = 256 # a non-list field whose JSON is shorter than this stays inline+CHUNK = 1 << 20+++def dumps(v, ascii: bool = False) -> str:+ return json.dumps(v, separators=(",", ":"), ensure_ascii=ascii)+++def line(obj) -> bytes:+ """One log line. A lone surrogate (a JSON \\udXXX escape) cannot be UTF-8: such a line is written escaped."""+ try:+ return (dumps(obj) + "\n").encode()+ except UnicodeEncodeError:+ return (dumps(obj, True) + "\n").encode()+++def ranges(ids: list[int]) -> list:+ out: list = []+ for i in ids:+ last = out[-1] if out else None+ if isinstance(last, list) and last[1] + 1 == i:+ last[1] = i+ elif isinstance(last, int) and last + 1 == i:+ out[-1] = [last, i]+ else:+ out.append(i)+ return out+++def expand(rs: list) -> Iterator[int]:+ for r in rs:+ if isinstance(r, list):+ yield from range(r[0], r[1] + 1)+ else:+ yield r+++def members(path: pathlib.Path) -> Iterator[tuple[int, list[dict]]]:+ """(end offset, objects) of each complete gzip member of a raw.jsonl.gz (one append each), in order; stops at the+ first incomplete or damaged member (a crash mid-write)."""+ with open(path, "rb") as f:+ start, pending = 0, b""+ while True:+ d = zlib.decompressobj(zlib.MAX_WBITS | 16)+ out, fed = [], 0+ chunk = pending or f.read(CHUNK)+ while chunk:+ fed += len(chunk)+ try:+ out.append(d.decompress(chunk))+ except zlib.error:+ return+ if d.eof:+ break+ chunk = f.read(CHUNK)+ if not d.eof:+ return+ pending = d.unused_data+ end = start + fed - len(pending)+ try:+ objs = [json.loads(ln) for ln in b"".join(out).decode("utf-8").split("\n") if ln]+ except ValueError: # UnicodeDecodeError included+ return+ yield end, objs+ start = end+++def lines(path: pathlib.Path) -> Iterator[dict]:+ """The log's objects; stops quietly at a truncated or damaged append (a crash mid-write)."""+ if ".xz" not in path.name:+ for _, objs in members(path):+ yield from objs+ return+ try:+ with lzma.open(path, "rt", encoding="utf-8") as f:+ for line in f:+ if not line.endswith("\n"):+ return+ yield json.loads(line)+ except (EOFError, lzma.LZMAError, ValueError):+ return+++class Writer:+ """Appends records to one trial's raw.jsonl.gz; resumes the item table from an existing file, cut back to its last+ complete append first (a crash mid-write leaves a partial gzip member, and anything appended behind it would be+ unreadable)."""++ def __init__(self, path: pathlib.Path):+ self.path = path+ self.ids: dict[bytes, int] = {}+ self.n = 0+ if path.exists():+ end = 0+ for end, objs in members(path):+ for obj in objs:+ if "item" in obj:+ self.ids[self._key(obj["v"])] = obj["item"]+ elif "n" in obj:+ self.n = obj["n"]+ size = path.stat().st_size+ if size > end:+ with open(path, "r+b") as f:+ f.truncate(end)+ print(f"rawlog: {path}: dropped {size - end} bytes of an incomplete append", file=sys.stderr)++ @staticmethod+ def _key(v) -> bytes:+ return hashlib.sha256(dumps(v).encode("utf-8", "surrogatepass")).digest()++ def _encode(self, text: str, new: list[bytes]) -> dict | None:+ try:+ obj = json.loads(text)+ except ValueError:+ return None+ if not isinstance(obj, dict):+ return None+ ascii = dumps(obj) != text+ if ascii and dumps(obj, True) != text:+ return None++ def ref(v) -> int:+ k = self._key(v)+ if k not in self.ids:+ self.ids[k] = len(self.ids)+ new.append(line({"item": self.ids[k], "v": v}))+ return self.ids[k]++ parts = []+ for k, v in obj.items():+ if len(dumps(v)) < INLINE_MAX:+ parts.append([k, "v", v])+ elif isinstance(v, list):+ parts.append([k, "l", ranges([ref(x) for x in v])])+ else:+ parts.append([k, "i", ref(v)])+ return {"body_parts": parts, "body_ascii": ascii}++ def append(self, raw: dict) -> int:+ """Record {"request": {..., "body"}, "response": {...}}; returns its n."""+ new: list[bytes] = []+ req = dict(raw["request"])+ enc = self._encode(req["body"], new) if req.get("body_as", "text") == "text" else None+ if enc is not None:+ del req["body"]+ req.update(enc)+ self.n += 1+ new.append(line({"n": self.n, "request": req, "response": raw["response"]}))+ with gzip.open(self.path, "ab") as f:+ f.write(b"".join(new))+ return self.n+++def read(path: pathlib.Path) -> Iterator[dict]:+ """The records of a raw.jsonl.gz in full: {"n", "request": {..., "body"}, "response"}."""+ items: dict[int, object] = {}+ for obj in lines(path):+ if "item" in obj:+ items[obj["item"]] = obj["v"]+ continue+ req = obj["request"]+ parts = req.pop("body_parts", None)+ if parts is not None:+ body = {}+ for k, kind, v in parts:+ body[k] = v if kind == "v" else items[v] if kind == "i" else [items[i] for i in expand(v)]+ ascii = req.pop("body_ascii")+ req["body"] = dumps(body, ascii)+ yield {"n": obj["n"], "request": req, "response": obj["response"]}+++def log_of(d: pathlib.Path) -> pathlib.Path | None:+ for name in (PACKED, NAME):+ if (d / name).exists():+ return d / name+ return None+++def gateway_dirs(paths: list[pathlib.Path]) -> Iterator[pathlib.Path]:+ for p in paths:+ if (p / "raw").is_dir() or log_of(p):+ yield p+ else:+ yield from sorted(q for q in p.rglob("gateway") if q.is_dir())+++def legacy(d: pathlib.Path) -> list[pathlib.Path]:+ return sorted((d / "raw").glob("*.json.gz")) if (d / "raw").is_dir() else []+++def remove(files: list[pathlib.Path]) -> None:+ for f in files:+ f.unlink()+ files[0].parent.rmdir()+++def same(rec: dict, old: dict) -> bool:+ return rec["request"] == old["request"] and rec["response"] == old["response"]+++def to_xz(src: pathlib.Path, out: pathlib.Path) -> None:+ """Rewrite a raw.jsonl.gz as one xz stream (via out.tmp, so a half-written out never exists)."""+ tmp = out.with_name(out.name + ".tmp")+ with lzma.open(tmp, "wb", preset=XZ_PRESET) as f:+ for obj in lines(src):+ f.write(line(obj))+ if sum(1 for _ in lines(src)) != sum(1 for _ in lines(tmp)):+ tmp.unlink()+ raise RuntimeError(f"{tmp}: line count differs")+ tmp.rename(out)+++def compact(d: pathlib.Path) -> str:+ src = d / NAME+ if not src.exists():+ return f"no {NAME}"+ if (d / PACKED).exists():+ return f"{PACKED} exists already, skipped"+ to_xz(src, d / PACKED)+ msg = f"{src.stat().st_size / 1e6:.1f} MB -> {(d / PACKED).stat().st_size / 1e6:.1f} MB"+ src.unlink()+ return msg+++def pack(d: pathlib.Path, delete: bool) -> str:+ files = legacy(d)+ if not files:+ return "no raw/"+ if log_of(d): # packed before: --delete removes raw/ once it matches+ msg = verify(d)+ if delete and "identical" in msg:+ remove(files)+ return msg + ", raw/ removed"+ return msg if "differs" in msg else f"{log_of(d).name} exists already, skipped"+ if [int(f.name.split(".")[0]) for f in files] != list(range(1, len(files) + 1)):+ return "raw/ is not numbered 1..N, skipped"+ tmp = d / (NAME + ".tmp")+ tmp.unlink(missing_ok=True)+ w = Writer(tmp)+ for f in files:+ w.append(json.loads(gzip.decompress(f.read_bytes())))+ to_xz(tmp, d / PACKED)+ tmp.unlink()+ for f, rec in zip(files, read(d / PACKED), strict=True):+ if not same(rec, json.loads(gzip.decompress(f.read_bytes()))):+ (d / PACKED).unlink()+ return f"round trip differs at {f.name}, skipped"+ before = sum(f.stat().st_size for f in files)+ msg = f"{len(files)} requests, {before / 1e6:.1f} MB -> {(d / PACKED).stat().st_size / 1e6:.1f} MB"+ if delete:+ remove(files)+ msg += ", raw/ removed"+ return msg+++def verify(d: pathlib.Path) -> str:+ path = log_of(d)+ if path is None:+ return "no log"+ files = legacy(d)+ n = 0+ for rec in read(path): # streamed: a long trial's records add up to gigabytes+ n += 1+ if rec["n"] != n:+ return f"{path.name}: records out of order"+ if files and n > len(files):+ return f"{path.name}: more records than the {len(files)} raw/ files: differs"+ if files and not same(rec, json.loads(gzip.decompress(files[n - 1].read_bytes()))):+ return f"{path.name}: differs from raw/{files[n - 1].name}"+ if files:+ if len(files) != n:+ return f"{path.name}: {n} records, {len(files)} raw/ files: differs"+ return f"{path.name}: {n} records, identical to raw/"+ return f"{path.name}: {n} records"+++def main() -> int:+ p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ sub = p.add_subparsers(dest="cmd", required=True)+ c = sub.add_parser("cat")+ c.add_argument("dir", type=pathlib.Path)+ c.add_argument("-n", type=int, help="only request n")+ for name in ("compact", "pack", "verify"):+ s = sub.add_parser(name)+ s.add_argument("dirs", type=pathlib.Path, nargs="+")+ if name == "pack":+ s.add_argument("--delete", action="store_true", help="remove raw/ once the packed log round-trips")+ a = p.parse_args()+ if a.cmd == "cat":+ path = log_of(a.dir) if a.dir.is_dir() else a.dir+ if path is None:+ sys.exit(f"{a.dir}: no {PACKED} or {NAME}")+ for rec in read(path):+ if a.n is None or rec["n"] == a.n:+ sys.stdout.write(json.dumps(rec, ensure_ascii=False) + "\n")+ return 0+ bad = 0+ for d in gateway_dirs(a.dirs):+ msg = {"compact": compact, "verify": verify}[a.cmd](d) if a.cmd != "pack" else pack(d, a.delete)+ bad += any(w in msg for w in ("differs", "out of order"))+ print(f"{d}: {msg}", flush=True)+ return 1 if bad else 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/usage_gateway/tests/test_rawlog.py b/scripts/usage_gateway/tests/test_rawlog.pynew file mode 100644index 0000000000..20a742b6d0--- /dev/null+++ b/scripts/usage_gateway/tests/test_rawlog.py@@ -0,0 +1,143 @@+"""Tests for the deduplicated raw log (rawlog.py): every request body and response comes back byte for byte.++ python -m unittest discover -s scripts/usage_gateway/tests # stdlib only+"""+from __future__ import annotations++import gzip+import json+import pathlib+import subprocess+import sys+import tempfile+import unittest++HERE = pathlib.Path(__file__).resolve().parents[1]+sys.path.insert(0, str(HERE))+import rawlog # noqa: E402++BIG = "x" * 2000+++def rec(body: str, how: str = "text", resp: str = "event: done\ndata: {}\n\n") -> dict:+ return {"request": {"method": "POST", "host": "api.example.com", "path": "/v1/responses",+ "headers": {"content-type": "application/json"}, "body": body, "body_as": how},+ "response": {"status": 200, "headers": {"content-type": "text/event-stream"}, "body": resp}}+++def conversation() -> list[dict]:+ """An OpenAI Responses and an Anthropic Messages conversation, interleaved, each request resending the history."""+ out, inp, msgs = [], [], []+ for i in range(6):+ inp.append({"type": "message", "role": "user", "content": [{"type": "input_text", "text": f"{BIG} {i}"}]})+ out.append(rec(json.dumps({"model": "gpt-x", "instructions": BIG, "input": inp, "stream": True},+ separators=(",", ":"), ensure_ascii=False)))+ # Claude Code moves cache_control to the newest message, so the previous last message changes every request+ msgs = [{k: v for k, v in m.items() if k != "cache_control"} for m in msgs]+ msgs.append({"role": "user", "content": f"é {BIG} {i}", "cache_control": {"type": "ephemeral"}})+ out.append(rec(json.dumps({"model": "claude-x", "system": [{"type": "text", "text": BIG}], "messages": msgs,+ "tools": [{"name": "bash", "description": BIG}], "max_tokens": 32000},+ separators=(",", ":"), ensure_ascii=False), resp=f"event: message\ndata: {i}\n\n"))+ return out+++class RawLogTest(unittest.TestCase):+ def setUp(self):+ self.tmp = tempfile.TemporaryDirectory()+ self.d = pathlib.Path(self.tmp.name)++ def tearDown(self):+ self.tmp.cleanup()++ def write(self, recs: list[dict]) -> rawlog.Writer:+ w = rawlog.Writer(self.d / rawlog.NAME)+ for r in recs:+ w.append(r)+ return w++ def check(self, recs: list[dict], path: pathlib.Path | None = None) -> None:+ got = list(rawlog.read(path or self.d / rawlog.NAME))+ self.assertEqual([g["n"] for g in got], list(range(1, len(recs) + 1)))+ for g, r in zip(got, recs, strict=True):+ self.assertEqual(g["request"], r["request"])+ self.assertEqual(g["response"], r["response"])++ def test_round_trip_and_dedup(self):+ recs = conversation()+ self.write(recs)+ self.check(recs)+ full = sum(len(json.dumps(r)) for r in recs)+ stored = len(gzip.decompress((self.d / rawlog.NAME).read_bytes()))+ self.assertLess(stored, full / 2) # the history is stored once, not once per request++ def test_bodies_kept_as_text(self):+ recs = [rec('{"a": 1}'), # spaced JSON: does not re-serialize exactly+ rec(json.dumps({"t": "é" + BIG}, separators=(",", ":"))), # é escapes: ascii serialization+ rec('{"t":"\\ud800' + BIG + '"}'), # lone surrogate+ rec("not json"), rec("[1,2,3]"), rec("AAEC", how="base64"), rec("")]+ self.write(recs)+ self.check(recs)++ def test_resume_after_restart(self):+ recs = conversation()+ self.write(recs[:5])+ w = self.write(recs[5:]) # a new gateway process picks up the item table and n+ self.assertEqual(w.n, len(recs))+ self.check(recs)++ def test_truncated_last_append(self):+ recs = conversation()+ self.write(recs)+ path = self.d / rawlog.NAME+ path.write_bytes(path.read_bytes()[:-20])+ got = list(rawlog.read(path))+ self.assertEqual(len(got), len(recs) - 1)++ def test_restart_after_crash_mid_write(self):+ recs = conversation()+ self.write(recs[:5])+ path = self.d / rawlog.NAME+ path.write_bytes(path.read_bytes()[:-20]) # the gateway died writing request 5+ w = self.write(recs[5:]) # the restarted gateway cuts the partial member, then appends+ self.assertEqual(w.n, len(recs) - 1)+ got = list(rawlog.read(path))+ self.assertEqual([g["n"] for g in got], list(range(1, len(recs))))+ for g, r in zip(got, recs[:4] + recs[5:], strict=True):+ self.assertEqual(g["request"], r["request"])+ self.assertEqual(g["response"], r["response"])+ self.assertIn("->", rawlog.compact(self.d))+ self.assertEqual(len(list(rawlog.read(self.d / rawlog.PACKED))), len(recs) - 1)++ def test_damaged_member_stops_reading(self):+ recs = conversation()+ self.write(recs[:3])+ path = self.d / rawlog.NAME+ path.write_bytes(path.read_bytes() + gzip.compress(b"x" * 5000)[:-30] + gzip.compress(b'{"n": 99}\n'))+ self.assertEqual(len(list(rawlog.read(path))), 3) # no exception; nothing behind the broken member++ def test_compact(self):+ recs = conversation()+ self.write(recs)+ self.assertIn("->", rawlog.compact(self.d))+ self.assertFalse((self.d / rawlog.NAME).exists())+ self.check(recs, self.d / rawlog.PACKED)++ def test_pack_legacy_cli(self):+ recs = conversation()+ (self.d / "raw").mkdir()+ for i, r in enumerate(recs, 1):+ (self.d / "raw" / f"{i:06d}.json.gz").write_bytes(gzip.compress(json.dumps(r).encode()))+ tool = [sys.executable, str(HERE / "rawlog.py")]+ out = subprocess.run(tool + ["pack", str(self.d)], capture_output=True, text=True, check=True).stdout+ self.assertIn(f"{len(recs)} requests", out)+ out = subprocess.run(tool + ["verify", str(self.d)], capture_output=True, text=True, check=True).stdout+ self.assertIn("identical to raw/", out)+ subprocess.run(tool + ["pack", "--delete", str(self.d)], check=True, capture_output=True) # verifies, removes+ self.assertFalse((self.d / "raw").exists())+ last = json.loads(subprocess.run(tool + ["cat", str(self.d), "-n", str(len(recs))], capture_output=True,+ text=True, check=True).stdout)+ self.assertEqual(last["request"], recs[-1]["request"])+++if __name__ == "__main__":+ unittest.main()