Skip to content

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

Merge pull request #37 from JamesKrW/gateway/compact-raw (+1 more)

Kangrui Wang218aa2745eef
diff --git a/docs/RUNNING.md b/docs/RUNNING.md
index 699bb43ff8..6ef9357bc9 100644
--- a/docs/RUNNING.md
+++ b/docs/RUNNING.md
@@ -56,6 +56,14 @@ results site with `scripts/write_task_results.py --out` outside the repository (
next to the checkout, gets `<batch>/<task>/<run id>/`; `RESULTS=` skips it), updates `$JOBS/<batch>.summary.md`, and
stops its lane when a trial shows a model-account failure.
+To watch a batch live: `scripts/live_board.py --jobs <JOBS>` (repeatable; stdlib only) serves a page on 127.0.0.1 that
+refreshes every 5 s. It shows every trial as pending, setup, running (agent time against the task's limit), verifying,
+success or failed, with live tokens when the usage gateway runs, the reward, the videos and the tail of the agent's
+output. Pending trials come from `$JOBS/.plan/<batch>.json`, which the runner writes when it starts. Reach the page from
+another machine with `ssh -N -L 8765:127.0.0.1:<port> <host>`; it shows the agent's output, so keep it off public
+addresses. It has no login (every user of a shared host can open it) and serves a running trial's files before the
+runner's secret scan has redacted them.
+
Runner-side tools (they run on the host, never in a task): `scripts/log_mirror.py` (in the `rcb-log-mirror` container: an
append-only copy of every trial's `agent/` record that neither the agent nor the host user can delete),
`scripts/mem_guard.py` (keeps 2 GiB of host memory available by killing the largest benchmark container first),
@@ -90,7 +98,7 @@ Per trial, next to Harbor's `agent/` and `verifier/` (outside the agent's mounts
```
<trial>/gateway/usage.jsonl one line per request: time, duration, host, path, status, model, input / cached / output tokens
<trial>/gateway/totals.json running totals, rewritten after every request
-<trial>/gateway/raw/ every request and response body (GATEWAY_RAW=1, the default with the gateway on), gzipped JSON (headers without credentials)
+<trial>/gateway/raw.jsonl.xz every request and response body (GATEWAY_RAW=1, the default with the gateway on; headers without credentials)
```
Input counts what the model read (Anthropic: input + cache read + cache creation; OpenAI: `input_tokens`, cached
included); output includes reasoning: the same convention as Harbor's `n_input_tokens` / `n_output_tokens`, which the batch
@@ -100,7 +108,12 @@ before logging (measured 2026-10-01 on 9 benchmarks: the difference is exactly t
(default 100M / 10M; 0 = no limit) gets every further model request refused (HTTP 400, "token budget exceeded"), so
the agent stops; `totals.json` records `over_token_budget = 1`, the batch summary marks it, and `write_task_results.py`
counts it as failed (PROTOCOL §5). Tested 2026-10-02: Codex 0.159.2 ends its turn on the first refusal, without retrying.
+The raw log stores each conversation item once and a request as references to the items it resends, like Codex's and
+Claude Code's own session files, and is lossless: `scripts/usage_gateway/rawlog.py cat <trial>/gateway` prints every
+request and response in full. The gateway appends to `raw.jsonl.gz` while the trial runs; after each job the runner
+compacts it to `raw.jsonl.xz` (one 1,994-request trial: 5.9 GB as one gzip file per request, the format before
+2026-10-02, 32 MB now; `rawlog.py pack --delete <dir>` converts such `raw/` directories).
Switches: `GATEWAY=1` (record usage and, by default, raw bodies), `GATEWAY_RAW=0` (usage only; raw bodies are private: they hold the
whole conversation). Known limits: no WebSocket (Codex's built-in provider falls back to HTTPS after a few refused
-attempts); Codex request bodies are zstd-compressed and are stored as base64 unless `zstandard` is installed; OpenAI
-reasoning arrives encrypted. Requests no trial can be matched to go to `<jobs>/.gateway/unattributed.jsonl`.
+attempts); Codex request bodies are zstd-compressed and are stored as base64, and so not deduplicated, unless
+`zstandard` is installed on the runner host; OpenAI reasoning arrives encrypted. Requests no trial can be matched to go to `<jobs>/.gateway/unattributed.jsonl`.
diff --git a/scripts/live_board.py b/scripts/live_board.py
new file mode 100644
index 0000000000..c403cb5742
--- /dev/null
+++ b/scripts/live_board.py
@@ -0,0 +1,430 @@
+#!/usr/bin/env python3
+"""Live board of batch evaluations: one web page with every trial under Harbor jobs directories and its state, refreshed
+every few seconds while the batch runs. A runner-side tool (stdlib only); it only reads.
+
+ scripts/live_board.py [--jobs jobs/agents] [--jobs jobs] [--port N] [--bind 127.0.0.1]
+ # prints its address (default: the first free port from 8765); from a laptop: ssh -N -L 8765:127.0.0.1:<port> <host>,
+ # then open http://localhost:8765
+
+States: pending (in a batch's plan, not started; scripts/run_agent_batch.sh writes <jobs>/.plan/<batch>.json), setup
+(environment and agent install), running (with the agent's time limit), verifying, then success / failed, or aborted
+(the job ended without finishing the trial). Per trial: tokens (live from the usage gateway's totals.json when it runs,
+else Harbor's count at the end), reward, exception, the replay / live videos, and the tail of the agent's output.
+
+It serves the agent's output and the videos, which can hold the whole conversation: keep it on 127.0.0.1 (the default)
+and reach it through an SSH tunnel; bind elsewhere only on a trusted network. There is no login: on a shared host every
+local user can open it, and files of a running trial are served before the runner's secret scan has redacted them. On
+127.0.0.1 it answers only requests addressed to localhost / 127.0.0.1 / [::1] (a web page cannot reach it through DNS
+rebinding).
+"""
+from __future__ import annotations
+
+import argparse
+import datetime as dt
+import json
+import os
+import pathlib
+import re
+import sys
+import threading
+import time
+import tomllib
+import urllib.parse
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+REPO = pathlib.Path(__file__).resolve().parents[1]
+SKIP = {"agent", "artifacts", "verifier", "gateway", "sessions", "_logs", "node_modules"}
+MEDIA = {".mp4": "video/mp4", ".png": "image/png", ".jpg": "image/jpeg", ".md": "text/plain; charset=utf-8",
+ ".json": "application/json", ".txt": "text/plain; charset=utf-8", ".log": "text/plain; charset=utf-8"}
+ARGS: argparse.Namespace
+LOOPBACK = {"localhost", "127.0.0.1", "::1"}
+LOOPBACK_BIND = True # bound to a loopback address: answer only requests addressed to one (main sets it)
+CACHE: dict = {"t": 0.0, "state": None}
+LOCK = threading.Lock()
+
+
+def jload(p: pathlib.Path) -> dict:
+ try:
+ return json.loads(p.read_text())
+ except (OSError, ValueError):
+ return {}
+
+
+def ts(s: str | None) -> float | None:
+ if not s:
+ return None
+ try:
+ t = dt.datetime.fromisoformat(s.replace("Z", "+00:00"))
+ except ValueError:
+ return None
+ return (t if t.tzinfo else t.astimezone()).timestamp() # job-level times are local, trial-level UTC
+
+
+_TASKS: dict[str, dict] = {}
+
+
+def task_info(task: str) -> dict:
+ """Benchmark (the task name's prefix: robocasa-gr1-..., robotwin_2_0_..., kinder-...) and agent time limit."""
+ if task not in _TASKS:
+ try:
+ t = tomllib.loads((REPO / "tasks" / task / "task.toml").read_text())
+ except (OSError, ValueError):
+ t = {}
+ m = re.match(r"(robocasa-gr1|robocasa365|mujoco_playground|robotwin_2_0|[a-z0-9]+)", task)
+ bench = m.group(1) if m else task
+ _TASKS[task] = {"bench": bench, "timeout": (t.get("agent") or {}).get("timeout_sec")}
+ return _TASKS[task]
+
+
+def run_id(agent: dict) -> str:
+ """<harness>-<model>-<effort> as scripts/run_agent_batch.sh derives it, except that an Azure deployment replica's
+ suffix is dropped (gpt-6-luna-3 -> gpt6_luna; claude-fable-5-1 stays fable_5_1)."""
+ f = lambda s: re.sub(r"^_+|_+$", "", re.sub(r"[^a-z0-9]+", "_", s.lower()))
+ model = (agent.get("model_name") or "").split("/")[-1].lower()
+ model = re.sub(r"^(gpt-\d+(?:\.\d+)?-[a-z]+)-\d+$", r"\1", model)
+ model = re.sub(r"^gpt-(\d+)", r"gpt\1", re.sub(r"^claude-", "", model))
+ effort = (agent.get("kwargs") or {}).get("reasoning_effort") or "default"
+ return f"{f(agent.get('name') or '?')}-{f(model)}-{f(effort)}"
+
+
+def walk(root: pathlib.Path, depth: int = 0):
+ """Job directories (a config.json, no __ in the name) below root."""
+ try:
+ entries = [e for e in os.scandir(root) if e.is_dir(follow_symlinks=False)]
+ except OSError:
+ return
+ for e in entries:
+ if e.name.startswith((".", "_")) or e.name in SKIP: # _superseded/, _logs/: set aside, not shown
+ continue
+ p = pathlib.Path(e.path)
+ if (p / "config.json").exists() and "__" not in e.name:
+ yield p
+ elif depth < 5:
+ yield from walk(p, depth + 1)
+
+
+def newest(paths: list[pathlib.Path]) -> float:
+ m = 0.0
+ for p in paths:
+ try:
+ m = max(m, p.stat().st_mtime)
+ except OSError:
+ pass
+ return m
+
+
+def trial_state(t: pathlib.Path, job: dict, ri: int, root: pathlib.Path, plan_run: str | None) -> dict:
+ cfg = jload(t / "config.json")
+ r = jload(t / "result.json")
+ task = pathlib.Path((cfg.get("task") or {}).get("path") or t.name.split("__")[0]).name
+ info = task_info(task)
+ gw = jload(t / "gateway" / "totals.json")
+ ae = r.get("agent_execution") or {}
+ ver = r.get("verifier") or {}
+ rewards = (r.get("verifier_result") or {}).get("rewards") or {}
+ reward = rewards.get("reward")
+ if reward is None and (t / "verifier" / "reward.txt").exists():
+ try:
+ reward = float((t / "verifier" / "reward.txt").read_text().strip())
+ except ValueError:
+ pass
+ agent_dir = t / "agent"
+ agent_files = list(agent_dir.glob("*.txt")) if agent_dir.is_dir() else []
+ now = time.time()
+ a0, a1 = ts(ae.get("started_at")), ts(ae.get("finished_at"))
+ if r.get("finished_at"):
+ state = "success" if reward is not None and reward >= 1 and not gw.get("over_token_budget") else "failed"
+ elif job.get("finished_at"):
+ state = "aborted"
+ elif ver.get("started_at") or (a1 and not ver.get("finished_at")):
+ state = "verifying"
+ elif a0 or agent_files:
+ state = "running"
+ else:
+ state = "setup"
+ if a0 is None and agent_files and state == "running":
+ a0 = min(p.stat().st_ctime for p in agent_files)
+ ar = r.get("agent_result") or {}
+ tok_in = gw.get("input") if gw else ar.get("n_input_tokens")
+ tok_out = gw.get("output") if gw else ar.get("n_output_tokens")
+ media = []
+ for d in ("verifier", "artifacts/data"):
+ for p in sorted((t / d).glob("*")) if (t / d).is_dir() else []:
+ if p.suffix in (".mp4", ".png") and p.name not in {m.split("/")[-1] for m in media}:
+ media.append(str(p.relative_to(root)))
+ exc = (r.get("exception_info") or {}).get("exception_type") or ""
+ return {
+ "id": f"{ri}:{t.relative_to(root)}", "task": task, "bench": info["bench"],
+ "mode": "standard" if task.endswith("-standard") else "privileged",
+ "run": plan_run or run_id((cfg.get("agent") or {})), "job": job.get("job_name") or t.parent.name,
+ "trial": t.name, "state": state, "reward": reward, "exception": exc,
+ "agent_start": a0, "agent_end": a1 or (now if state == "running" else None), "timeout": info["timeout"],
+ "tok_in": tok_in, "tok_out": tok_out, "requests": gw.get("requests"), "gateway": bool(gw),
+ "over_budget": bool(gw.get("over_token_budget")),
+ "updated": newest([t / "result.json", t / "trial.log", t / "gateway" / "totals.json", *agent_files]),
+ "media": media, "summary": str((t.parent / "summary.md").relative_to(root)) if (t.parent / "summary.md").exists() else None,
+ }
+
+
+def collect() -> dict:
+ trials, plans = [], []
+ for ri, root in enumerate(ARGS.jobs):
+ plan_of: dict[str, str] = {}
+ for pf in sorted((root / ".plan").glob("*.json")) if (root / ".plan").is_dir() else []:
+ p = jload(pf)
+ for j in p.get("jobs") or []:
+ plan_of[j["job"]] = p.get("run_id")
+ plans.append((ri, p, j))
+ for jd in walk(root):
+ jc = jload(jd / "config.json")
+ jr = jload(jd / "result.json")
+ name = jc.get("job_name") or jd.name
+ try:
+ tds = sorted(jd.iterdir())
+ except OSError: # the job directory moved away (archived) during the scan
+ continue
+ for t in tds:
+ try:
+ if t.is_dir() and "__" in t.name and (t / "config.json").exists():
+ trials.append(trial_state(t, {"job_name": name, **jr}, ri, root,
+ plan_of.get(name) or plan_of.get(jd.name)))
+ except OSError: # a file vanished mid-scan: the next refresh picks the trial up
+ continue
+ started = {(t["job"], t["task"]) for t in trials}
+ for ri, p, j in plans:
+ if (j["job"], j["task"]) in started:
+ continue
+ info = task_info(j["task"])
+ trials.append({"id": None, "task": j["task"], "bench": info["bench"], "mode": j.get("mode"),
+ "run": p.get("run_id"), "job": j["job"], "trial": None, "state": "pending",
+ "timeout": info["timeout"], "batch": p.get("batch")})
+ return {"now": time.time(), "roots": [str(r) for r in ARGS.jobs], "trials": trials}
+
+
+def state() -> dict:
+ with LOCK:
+ if CACHE["state"] is None or time.time() - CACHE["t"] > 3:
+ CACHE["state"], CACHE["t"] = collect(), time.time()
+ return CACHE["state"]
+
+
+def resolve(tid: str) -> tuple[pathlib.Path, pathlib.Path] | None:
+ ri, _, rel = tid.partition(":")
+ try:
+ root = ARGS.jobs[int(ri)]
+ except (ValueError, IndexError):
+ return None
+ p = (root / rel).resolve()
+ return (root, p) if p.is_relative_to(root.resolve()) and p.exists() else None
+
+
+def tail(p: pathlib.Path, n: int = 80, maxb: int = 64_000) -> str:
+ try:
+ with open(p, "rb") as f:
+ f.seek(0, 2)
+ f.seek(max(0, f.tell() - maxb))
+ return "\n".join(f.read().decode("utf-8", "replace").splitlines()[-n:])
+ except OSError:
+ return ""
+
+
+def detail(tid: str) -> dict:
+ rp = resolve(tid)
+ if rp is None:
+ return {"error": "no such trial"}
+ _, t = rp
+ r = jload(t / "result.json")
+ out = sorted((t / "agent").glob("*.txt"), key=lambda p: p.stat().st_size, reverse=True) if (t / "agent").is_dir() else []
+ exc = r.get("exception_info") or {}
+ return {"reward": jload(t / "verifier" / "reward.json") or (r.get("verifier_result") or {}).get("rewards"),
+ "gateway": jload(t / "gateway" / "totals.json"),
+ "exception": ((exc.get("exception_type") or "") + ": " + (exc.get("exception_message") or "")[:2000]) if exc else "",
+ "agent_log": out[0].name if out else None, "agent_tail": tail(out[0]) if out else "",
+ "trial_log_tail": tail(t / "trial.log", 25)}
+
+
+PAGE = r"""<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
+<title>Live Board</title><style>
+:root{--bg:#f7f7f5;--fg:#1d1d1b;--mut:#6b6b66;--card:#fff;--line:#e3e2dd;--ok:#1f8a4c;--bad:#c2410c;--run:#2563eb;--ver:#7c3aed;--pend:#9ca3af;--warn:#b45309;--on:#fff}
+@media (prefers-color-scheme:dark){:root:not([data-theme=light]){--bg:#141413;--fg:#ecebe6;--mut:#9b9a93;--card:#1e1e1c;--line:#33332f;--ok:#4ade80;--bad:#fb923c;--run:#60a5fa;--ver:#a78bfa;--pend:#6b7280;--warn:#fbbf24;--on:#111}}
+:root[data-theme=dark]{--bg:#141413;--fg:#ecebe6;--mut:#9b9a93;--card:#1e1e1c;--line:#33332f;--ok:#4ade80;--bad:#fb923c;--run:#60a5fa;--ver:#a78bfa;--pend:#6b7280;--warn:#fbbf24;--on:#111}
+*{box-sizing:border-box}body{margin:0;background:var(--bg);color:var(--fg);font:14px/1.45 system-ui,-apple-system,Segoe UI,sans-serif}
+header{padding:16px;display:flex;flex-wrap:wrap;gap:12px;align-items:baseline}h1{font-size:18px;margin:0}.mut{color:var(--mut)}
+main{padding:0 16px 32px}.cards{display:grid;grid-template-columns:repeat(auto-fill,minmax(260px,1fr));gap:12px;margin-bottom:16px}
+.card{background:var(--card);border:1px solid var(--line);border-radius:10px;padding:12px}.card h2{font-size:14px;margin:0 0 6px;font-family:ui-monospace,monospace}
+.bar{display:flex;height:8px;border-radius:4px;overflow:hidden;background:var(--line);margin:8px 0}.bar span{display:block}
+.legend{display:flex;flex-wrap:wrap;gap:10px;font-size:12px}.dot{display:inline-block;width:8px;height:8px;border-radius:50%;margin-right:4px}
+.tools{display:flex;flex-wrap:wrap;gap:8px;margin-bottom:10px}select,input,button{font:inherit;color:var(--fg);background:var(--card);border:1px solid var(--line);border-radius:6px;padding:4px 8px}
+.wrap{overflow-x:auto;background:var(--card);border:1px solid var(--line);border-radius:10px}table{border-collapse:collapse;width:100%}
+th,td{padding:6px 10px;border-bottom:1px solid var(--line);text-align:left;white-space:nowrap}th{font-size:12px;color:var(--mut);font-weight:600;position:sticky;top:0;background:var(--card)}
+tr.row{cursor:pointer}tr.row:hover{background:color-mix(in srgb,var(--run) 6%,transparent)}td.num{text-align:right;font-variant-numeric:tabular-nums}
+.st{display:inline-block;padding:1px 8px;border-radius:999px;font-size:12px;font-weight:600;color:var(--on)}
+.st.success{background:var(--ok)}.st.failed{background:var(--bad)}.st.running{background:var(--run)}.st.verifying{background:var(--ver)}
+.st.setup{background:var(--warn)}.st.pending{background:var(--pend)}.st.aborted{background:#6b7280}
+.prog{width:90px;height:6px;background:var(--line);border-radius:3px;display:inline-block;vertical-align:middle;margin-right:6px}.prog i{display:block;height:100%;background:var(--run);border-radius:3px}
+.mx td,.mx th{text-align:center}.mx td:first-child,.mx th:first-child{text-align:left}
+#drawer{position:fixed;top:0;right:0;height:100%;width:min(720px,100%);background:var(--card);border-left:1px solid var(--line);transform:translateX(100%);visibility:hidden;transition:transform .2s,visibility .2s;overflow-y:auto;padding:16px;z-index:5}
+#drawer.open{transform:none;visibility:visible;box-shadow:-8px 0 24px #0003}#drawer pre{background:var(--bg);border:1px solid var(--line);border-radius:6px;padding:8px;overflow-x:auto;font-size:12px;white-space:pre-wrap;word-break:break-word;max-height:420px}
+video,img{max-width:100%;border-radius:6px;border:1px solid var(--line)}.close{float:right}
+</style></head><body>
+<header><h1>Live Board</h1><span class="mut" id="meta"></span></header>
+<main><div class="cards" id="cards"></div>
+<div class="tools"><select id="run"><option value="">all runs</option></select><select id="st"><option value="">all states</option></select>
+<select id="mode"><option value="">both modes</option><option>privileged</option><option>standard</option></select>
+<input id="q" placeholder="filter task / benchmark"><button id="view">matrix view</button></div>
+<div class="wrap" id="table"></div></main>
+<aside id="drawer"></aside>
+<script>
+const ORDER=["running","verifying","setup","pending","aborted","failed","success"],COL={success:"--ok",failed:"--bad",running:"--run",verifying:"--ver",setup:"--warn",pending:"--pend",aborted:"--mut"};
+const LABEL={success:"✓ success",failed:"✗ failed",running:"running",verifying:"verifying",setup:"setup",pending:"pending",aborted:"aborted"};
+let S=null,matrix=false,open=null;const $=id=>document.getElementById(id);
+try{matrix=localStorage.getItem("lb.matrix")==="1"}catch(e){}
+const fmtT=n=>n==null?"":n>=1e6?(n/1e6).toFixed(1)+"M":n>=1e3?(n/1e3).toFixed(0)+"k":String(n);
+const fmtM=s=>s==null?"":s<3600?Math.floor(s/60)+"m":(s/3600).toFixed(1)+"h";
+const esc=s=>String(s??"").replace(/[&<>"]/g,c=>({"&":"&amp;","<":"&lt;",">":"&gt;",'"':"&quot;"}[c]));
+const st=s=>`<span class="st ${s}">${LABEL[s]||s}</span>`;
+function filt(){const r=$("run").value,s=$("st").value,m=$("mode").value,q=$("q").value.toLowerCase();
+ return S.trials.filter(t=>(!r||t.run===r)&&(!s||t.state===s)&&(!m||t.mode===m)&&(!q||(t.task+" "+t.bench).toLowerCase().includes(q)))}
+function cards(){const by={};for(const t of S.trials)(by[t.run]??={})[t.state]=((by[t.run]??{})[t.state]||0)+1;
+ $("cards").innerHTML=Object.keys(by).sort().map(r=>{const c=by[r],n=Object.values(c).reduce((a,b)=>a+b,0),done=(c.success||0)+(c.failed||0)+(c.aborted||0);
+ return `<div class="card"><h2>${esc(r)}</h2><div><b>${c.success||0}</b> / ${done} succeeded · ${done} / ${n} done</div>
+ <div class="bar">${ORDER.slice().reverse().map(s=>c[s]?`<span style="width:${100*c[s]/n}%;background:var(${COL[s]})"></span>`:"").join("")}</div>
+ <div class="legend">${ORDER.filter(s=>c[s]).map(s=>`<span><i class="dot" style="background:var(${COL[s]})"></i>${LABEL[s]} ${c[s]}</span>`).join("")}</div></div>`}).join("")||'<div class="mut">no trials yet</div>'}
+function selects(){const runs=[...new Set(S.trials.map(t=>t.run))].sort(),r=$("run"),v=r.value;
+ r.innerHTML='<option value="">all runs</option>'+runs.map(x=>`<option${x===v?" selected":""}>${esc(x)}</option>`).join("");
+ const s=$("st"),sv=s.value;s.innerHTML='<option value="">all states</option>'+ORDER.map(x=>`<option value="${x}"${x===sv?" selected":""}>${LABEL[x]}</option>`).join("")}
+function elapsed(t){if(t.agent_start==null)return"";const e=(t.agent_end??S.now)-t.agent_start;
+ if(t.state==="running"&&t.timeout){const p=Math.min(100,100*e/t.timeout);return `<span class="prog"><i style="width:${p}%"></i></span>${fmtM(e)} / ${fmtM(t.timeout)}`}return fmtM(e)}
+function table(){const rows=filt().sort((a,b)=>ORDER.indexOf(a.state)-ORDER.indexOf(b.state)||a.task.localeCompare(b.task)||a.run.localeCompare(b.run));
+ $("view").textContent=matrix?"list view":"matrix view";
+ if(matrix){const runs=[...new Set(rows.map(t=>t.run))].sort(),tasks=[...new Set(rows.map(t=>t.task))].sort(),cell={};
+ for(const t of rows){const k=t.task+"|"+t.run,o=cell[k];if(!o||ORDER.indexOf(t.state)<ORDER.indexOf(o.state))cell[k]=t}
+ $("table").innerHTML=`<table class="mx"><tr><th>task</th>${runs.map(r=>`<th>${esc(r)}</th>`).join("")}</tr>`+tasks.map(k=>`<tr><td>${esc(k)}</td>${runs.map(r=>{const t=cell[k+"|"+r];
+ return `<td>${t?`<span class="${t.id?"row":""}" data-id="${esc(t.id||"")}" style="cursor:${t.id?"pointer":"default"}">${st(t.state)}</span>`:""}</td>`}).join("")}</tr>`).join("")+"</table>"}
+ else $("table").innerHTML=`<table><tr><th>state</th><th>task</th><th>benchmark</th><th>mode</th><th>run</th><th>agent time</th><th class="num">input tok</th><th class="num">output tok</th><th class="num">requests</th><th class="num">reward</th><th>exception</th></tr>`+
+ rows.map(t=>`<tr class="${t.id?"row":""}" data-id="${esc(t.id||"")}"><td>${st(t.state)}${t.over_budget?' <span class="st failed">over budget</span>':""}</td><td>${esc(t.task)}</td><td>${esc(t.bench)}</td><td>${esc(t.mode)}</td><td class="mut">${esc(t.run)}</td>
+ <td>${elapsed(t)}</td><td class="num">${fmtT(t.tok_in)}</td><td class="num">${fmtT(t.tok_out)}</td><td class="num">${t.requests??""}</td><td class="num">${t.reward??""}</td><td class="mut">${esc((t.exception||"").replace(/Error$/,""))}</td></tr>`).join("")+"</table>";
+ for(const el of document.querySelectorAll("[data-id]"))if(el.dataset.id)el.onclick=()=>show(el.dataset.id)}
+async function show(id){open=id;const t=S.trials.find(x=>x.id===id);if(!t)return;
+ const d=await (await fetch("api/trial?id="+encodeURIComponent(id))).json(),f=p=>"file?p="+encodeURIComponent(t.id.split(":")[0]+":"+p);
+ const vids=t.media.filter(m=>m.endsWith(".mp4")),imgs=t.media.filter(m=>m.endsWith(".png"));
+ $("drawer").innerHTML=`<button class="close" onclick="closeD()">close</button><h2 style="margin-top:0">${esc(t.task)}</h2>
+ <p>${st(t.state)} <span class="mut">${esc(t.run)} · ${esc(t.trial)}</span></p><p>agent time ${elapsed(t)} · tokens ${fmtT(t.tok_in)} in / ${fmtT(t.tok_out)} out${t.requests?` · ${t.requests} requests`:""}${t.summary?` · <a href="${f(t.summary)}" target="_blank">summary.md</a>`:""}</p>
+ ${vids.map(v=>`<p class="mut">${esc(v.split("/").pop())}</p><video src="${f(v)}" controls preload="metadata"></video>`).join("")}${!vids.length?imgs.map(v=>`<img src="${f(v)}">`).join(""):""}
+ ${d.exception?`<h3>exception</h3><pre>${esc(d.exception)}</pre>`:""}${d.reward&&Object.keys(d.reward).length?`<h3>reward</h3><pre>${esc(JSON.stringify(d.reward,null,1))}</pre>`:""}
+ ${d.agent_tail?`<h3>agent output (${esc(d.agent_log)}, last lines)</h3><pre>${esc(d.agent_tail)}</pre>`:""}${d.trial_log_tail?`<h3>trial.log</h3><pre>${esc(d.trial_log_tail)}</pre>`:""}`;
+ $("drawer").classList.add("open")}
+function closeD(){open=null;$("drawer").classList.remove("open")}
+async function tick(){try{S=await (await fetch("api/state")).json();
+ const c={};for(const t of S.trials)c[t.state]=(c[t.state]||0)+1;
+ $("meta").textContent=`${S.trials.length} trials · ${ORDER.filter(s=>c[s]).map(s=>c[s]+" "+s).join(" · ")} · updated ${new Date(S.now*1000).toLocaleTimeString()}`;
+ cards();selects();table();const t=open&&S.trials.find(x=>x.id===open);if(t&&["running","verifying","setup"].includes(t.state))show(open)}catch(e){$("meta").textContent="board unreachable: "+e}}
+for(const id of["run","st","mode","q"])$(id).addEventListener("input",table);
+$("view").onclick=()=>{matrix=!matrix;try{localStorage.setItem("lb.matrix",matrix?"1":"0")}catch(e){}table()};
+document.addEventListener("keydown",e=>{if(e.key==="Escape")closeD()});tick();setInterval(tick,5000);
+</script></body></html>"""
+
+
+class Handler(BaseHTTPRequestHandler):
+ def log_message(self, *a):
+ pass
+
+ def send(self, code: int, body: bytes, ctype: str, extra: dict | None = None) -> None:
+ self.send_response(code)
+ self.send_header("Content-Type", ctype)
+ self.send_header("Content-Length", str(len(body)))
+ self.send_header("Cache-Control", "no-store")
+ for k, v in (extra or {}).items():
+ self.send_header(k, v)
+ self.end_headers()
+ if self.command != "HEAD":
+ self.wfile.write(body)
+
+ def do_HEAD(self):
+ self.do_GET()
+
+ def do_GET(self):
+ host = (self.headers.get("Host") or "").lower()
+ host = host[1:].split("]")[0] if host.startswith("[") else host.rsplit(":", 1)[0]
+ if LOOPBACK_BIND and host not in LOOPBACK:
+ return self.send(403, b"use http://localhost:<port>/ (or 127.0.0.1)", "text/plain")
+ u = urllib.parse.urlparse(self.path)
+ q = urllib.parse.parse_qs(u.query)
+ if u.path == "/":
+ return self.send(200, PAGE.encode(), "text/html; charset=utf-8")
+ if u.path == "/api/state":
+ return self.send(200, json.dumps(state()).encode(), "application/json")
+ if u.path == "/api/trial":
+ return self.send(200, json.dumps(detail(q.get("id", [""])[0])).encode(), "application/json")
+ if u.path == "/file":
+ rp = resolve(q.get("p", [""])[0])
+ if rp is None or rp[1].suffix not in MEDIA or not rp[1].is_file():
+ return self.send(404, b"not found", "text/plain")
+ return self.file(rp[1])
+ self.send(404, b"not found", "text/plain")
+
+ def file(self, p: pathlib.Path) -> None:
+ """The file, or one byte range of it (video seeking), streamed in 1 MB pieces."""
+ size = p.stat().st_size
+ a, b, partial = 0, size - 1, False
+ m = re.match(r"bytes=(\d*)-(\d*)$", self.headers.get("Range", "").strip())
+ if m and (m.group(1) or m.group(2)):
+ if m.group(1):
+ a, b = int(m.group(1)), int(m.group(2)) if m.group(2) else size - 1
+ else:
+ a = max(0, size - int(m.group(2))) if int(m.group(2)) else size
+ b, partial = min(b, size - 1), True
+ if a > b:
+ return self.send(416, b"", "text/plain", {"Content-Range": f"bytes */{size}"})
+ self.send_response(206 if partial else 200)
+ self.send_header("Content-Type", MEDIA[p.suffix])
+ self.send_header("Content-Length", str(b - a + 1))
+ self.send_header("Cache-Control", "no-store")
+ self.send_header("Accept-Ranges", "bytes")
+ if partial:
+ self.send_header("Content-Range", f"bytes {a}-{b}/{size}")
+ self.end_headers()
+ if self.command == "HEAD":
+ return
+ with open(p, "rb") as f:
+ f.seek(a)
+ left = b - a + 1
+ while left > 0:
+ chunk = f.read(min(left, 1 << 20))
+ if not chunk:
+ break
+ try:
+ self.wfile.write(chunk)
+ except OSError: # the browser hung up (seeking elsewhere)
+ return
+ left -= len(chunk)
+
+
+def main() -> int:
+ global ARGS, LOOPBACK_BIND
+ p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--jobs", type=pathlib.Path, action="append", help="a Harbor -o directory (repeatable; default jobs/agents)")
+ p.add_argument("--port", type=int, help="default: the first free port from 8765")
+ p.add_argument("--bind", default="127.0.0.1")
+ p.add_argument("--once", action="store_true", help="print the state as JSON and exit")
+ ARGS = p.parse_args()
+ ARGS.jobs = [j.resolve() for j in (ARGS.jobs or [REPO / "jobs" / "agents"])]
+ LOOPBACK_BIND = ARGS.bind in LOOPBACK or ARGS.bind.startswith("127.")
+ if ARGS.once:
+ json.dump(collect(), sys.stdout, indent=1)
+ return 0
+ for port in [ARGS.port] if ARGS.port is not None else range(8765, 8865):
+ try:
+ srv = ThreadingHTTPServer((ARGS.bind, port), Handler)
+ break
+ except OSError as e:
+ if ARGS.port is not None or port == 8864:
+ sys.exit(f"cannot listen on {ARGS.bind}:{port}: {e}")
+ srv.daemon_threads = True
+ print(f"live board on http://{ARGS.bind}:{srv.server_address[1]}/ for {', '.join(map(str, ARGS.jobs))}", flush=True)
+ srv.serve_forever()
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/run_agent_batch.sh b/scripts/run_agent_batch.sh
index 01e6a72753..76ed0bc678 100755
--- a/scripts/run_agent_batch.sh
+++ b/scripts/run_agent_batch.sh
@@ -39,7 +39,8 @@
# them unchanged (the agent keeps its own credentials) and records each trial's tokens per request in
# <trial>/gateway/ (usage.jsonl, totals.json)
# GATEWAY_RAW 1 (default, takes effect with GATEWAY=1) = the gateway also keeps every request and response body
-# (<trial>/gateway/raw/, gzipped; private: it holds the whole conversation); 0 = token usage only
+# (<trial>/gateway/raw.jsonl.xz, deduplicated, see scripts/usage_gateway/rawlog.py; private: it holds the
+# whole conversation); 0 = token usage only
# GATEWAY_BIND the docker bridge address the trial containers reach the host by (172.17.0.1)
# GATEWAY_EXTRA_HOSTS more model API host patterns the gateway forwards to, space-separated (e.g. "*.services.ai.azure.com")
# TOKEN_BUDGET_IN / TOKEN_BUDGET_OUT per-trial token budget the gateway enforces (100000000 / 10000000, PROTOCOL
@@ -48,7 +49,8 @@
# append-only into ${JOBS}_mirror/ (scripts/log_mirror.py; root-owned: neither the agent nor the host user can delete it),
# and scripts/mem_guard.py keeps 2 GiB of host memory available. After each job: a key scan with redaction
# (scripts/scan_secrets.py), the task results, the summary table $JOBS/<batch>.summary.md, and a lane stop on structured
-# model-account failures (scripts/account_failure_check.py). Progress: $JOBS/<batch>.log.
+# model-account failures (scripts/account_failure_check.py). Progress: $JOBS/<batch>.log; in a browser, every trial's
+# state (pending / running / done) with scripts/live_board.py --jobs $JOBS.
set -u
R="$(cd "$(dirname "$0")/.." && pwd)"
[ $# -ge 2 ] || { sed -n 2,40p "$0"; exit 2; }
@@ -94,6 +96,18 @@ LOG="$JOBS/$batch.log"
log() { echo "[$(date '+%F %T')] $*" | tee -a "$LOG"; }
if [ -n "$ENV_FILE" ] && [ ! -f "$ENV_FILE" ]; then log "no env file $ENV_FILE"; exit 2; fi
log "runner $$: $AGENT $MODEL effort $EFFORT, run id $RUN_ID, modes $MODES, web search $WEB_SEARCH"
+# the batch's plan, for scripts/live_board.py: every job this command runs, "pending" there until it starts
+mkdir -p "$JOBS/.plan"
+python3 - "$R" "$JOBS/.plan/$batch.json" "$batch" "$RUN_ID" "$SEQ" "$MODES" "${tasks[@]}" <<'PY'
+import json, os, pathlib, sys
+repo, out, batch, run_id, seq, modes, *tasks = sys.argv[1:]
+name = lambda m, t: f"{t}-standard" if m in ("standard", "limited") else f"{t}-privileged"
+jobs = [{"job": f"{batch}-{m}-{t}" if seq == "1" else f"{batch}-{m}", "task": name(m, t), "mode": m}
+ for t in tasks for m in modes.split() if (pathlib.Path(repo) / "tasks" / name(m, t)).is_dir()]
+tmp = pathlib.Path(f"{out}.{os.getpid()}")
+tmp.write_text(json.dumps({"batch": batch, "run_id": run_id, "jobs": jobs}, indent=1))
+tmp.replace(out)
+PY
# append-only mirror of every trial's agent/ directory, out of every trial container's reach
if [ "$LOG_MIRROR" = 1 ] && ! docker ps --format '{{.Names}}' | grep -qx rcb-log-mirror; then
@@ -159,8 +173,9 @@ wait_for() { # <mode>: block until every WAIT_FOR file of this mode exists
done
}
-finish_job() { # <job>: no key in any saved file (+redact); the task results (D3)
+finish_job() { # <job>: the raw logs compacted; no key in any saved file (+redact); the task results (D3)
[ -d "$JOBS/$1" ] || return 0
+ [ "$GATEWAY" = 1 ] && [ "$GATEWAY_RAW" = 1 ] && python3 "$R/scripts/usage_gateway/rawlog.py" compact "$JOBS/$1" 2>&1 | sed "s/^/[$1] /" | tee -a "$LOG"
python3 "$R/scripts/scan_secrets.py" "$JOBS/$1" "${SCAN[@]}" --redact 2>&1 | tail -3 | sed "s/^/[$1] /" | tee -a "$LOG"
[ -n "$RESULTS" ] && python3 "$R/scripts/write_task_results.py" "$JOBS" "$1" "$RUN_ID" "$LABEL" --out "$RESULTS/$batch" 2>&1 | tail -3 | sed "s/^/[$1] /" | tee -a "$LOG"
}
diff --git a/scripts/scan_secrets.py b/scripts/scan_secrets.py
index f1d4908359..64f19203b4 100755
--- a/scripts/scan_secrets.py
+++ b/scripts/scan_secrets.py
@@ -7,13 +7,42 @@ Loads the secrets INSIDE this process and never prints them: the named variables
agents read, VARS below) from the .env files (default: the repo's ../.env) or, where a file does not set one, from this
process's environment, and, with --codex-auth, the tokens of a Codex ChatGPT login (auth.json). Values shorter than 16
characters are not treated as secrets. It reports only which files contain one (and how many times). With --redact every
-occurrence is replaced with [REDACTED-<NAME>] in place. Binary files (videos, npz) are scanned too, as bytes."""
-import argparse, json, os, pathlib, sys
+occurrence is replaced with [REDACTED-<NAME>] in place. Binary files (videos, npz) are scanned too, as bytes; .gz and .xz
+files (the usage gateway's raw logs) decompressed. A damaged .gz / .xz (truncated, corrupt) is scanned as far as it
+decompresses, and as bytes; it is reported as DAMAGED and counted in the last line, and --redact rewrites it with only
+its readable part."""
+import argparse, gzip, json, lzma, os, pathlib, sys, zlib
+
+MAGIC = {".gz": b"\x1f\x8b", ".xz": b"\xfd7zXZ\x00"}
+DECOMPRESSOR = {".gz": lambda: zlib.decompressobj(zlib.MAX_WBITS | 16), ".xz": lzma.LZMADecompressor}
VARS = ["OPENROUTER_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "CLAUDE_CODE_OAUTH_TOKEN",
"AZURE_API_KEY", "AZURE_OPENAI_API_KEY"]
+def decompress(data: bytes, suffix: str) -> tuple[bytes, bool]:
+ """(what decompresses, whether all of it did): every stream of a multi-stream file (the gateway's live raw log is
+ one gzip stream per request), fed in 64 kB pieces; after a damaged stream it resumes at the next stream header."""
+ out, rest, whole = [], data, True
+ while rest:
+ d, i = DECOMPRESSOR[suffix](), 0
+ try:
+ while not d.eof and i < len(rest):
+ out.append(d.decompress(rest[i:i + (1 << 16)]))
+ i += 1 << 16
+ except (zlib.error, lzma.LZMAError):
+ pass
+ if not d.eof:
+ whole = False
+ j = rest.find(MAGIC[suffix], 1)
+ if j < 0:
+ break
+ rest = rest[j:]
+ continue
+ rest = (d.unused_data + rest[i:]).lstrip(b"\0") # the next stream (zero padding between streams allowed)
+ return b"".join(out), whole
+
+
def load_env(path: str) -> dict:
out = {}
for line in pathlib.Path(path).read_text().splitlines():
@@ -50,22 +79,31 @@ for f in a.codex_auth:
secrets["CODEX_OPENAI_API_KEY"] = d["OPENAI_API_KEY"].encode()
if not secrets:
sys.exit("no secret loaded (check --env-file / --var / --codex-auth); nothing to scan for")
-hits = scanned = 0
+hits = scanned = damaged = 0
for p in pathlib.Path(a.dir).rglob("*"):
if not p.is_file() or p.is_symlink():
continue
scanned += 1
try:
- data = p.read_bytes()
+ raw = p.read_bytes()
except OSError:
continue
- found = {n: data.count(s) for n, s in secrets.items() if s in data}
+ codec, data, whole = None, raw, True
+ if p.suffix in MAGIC and raw.startswith(MAGIC[p.suffix]): # otherwise not really compressed: scan it as bytes
+ codec = p.suffix
+ data, whole = decompress(raw, codec)
+ if not whole:
+ damaged += 1
+ print(f"DAMAGED {codec} (scanned the {len(data)} bytes that decompress, and the file as bytes): {p}")
+ found = {n: data.count(s) + (0 if whole else raw.count(s)) for n, s in secrets.items()}
+ found = {n: c for n, c in found.items() if c}
if found:
hits += 1
print(f"FOUND {', '.join(f'{n} x{c}' for n, c in found.items())}: {p}")
if a.redact:
for n in found:
data = data.replace(secrets[n], f"[REDACTED-{n}]".encode())
- p.write_bytes(data)
- print(" redacted")
-print(f"scanned {scanned} files under {a.dir} for {len(secrets)} secret(s) ({', '.join(sorted(secrets))}): {hits} contain one")
+ p.write_bytes(lzma.compress(data, preset=9) if codec == ".xz" else gzip.compress(data) if codec else data)
+ print(" redacted" + ("" if whole else " (only the readable part is kept)"))
+print(f"scanned {scanned} files under {a.dir} for {len(secrets)} secret(s) ({', '.join(sorted(secrets))}): "
+ f"{hits} contain one" + (f", {damaged} damaged compressed file(s)" if damaged else ""))
diff --git a/scripts/usage_gateway/gateway.py b/scripts/usage_gateway/gateway.py
index 98dd9ce0fb..075001fd04 100755
--- a/scripts/usage_gateway/gateway.py
+++ b/scripts/usage_gateway/gateway.py
@@ -16,8 +16,11 @@ own agent/ and verifier/ (not inside agent/, which the agent's container mounts
usage.jsonl one line per request: time, duration, host, path, status, model, normalized and raw usage
totals.json running totals (rewritten after every request): requests, input / cached / output tokens
- raw/NNNNNN.json.gz with --raw: the request body and the response body (SSE text) as sent and received;
- credentials, cookies and account ids are removed from the recorded headers
+ raw.jsonl.gz with --raw: every request body and response body (SSE text) as sent and received, deduplicated
+ (rawlog.py: each conversation item is stored once, a request as references to them; lossless);
+ credentials, cookies and account ids are removed from the recorded headers. usage.jsonl's "raw"
+ is the request's n there. `rawlog.py compact` (run_agent_batch.sh, after each job) turns it into
+ raw.jsonl.xz; `rawlog.py cat` prints the requests in full
Token budget (PROTOCOL §5, D10; on by default when the gateway runs): once a trial's totals reach --budget-input input
or --budget-output output tokens, totals.json gets over_token_budget = 1 and every further model request of that trial
@@ -47,6 +50,9 @@ import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+import rawlog # noqa: E402
+
ROOT = pathlib.Path(__file__).resolve().parents[2]
@@ -196,6 +202,7 @@ class Trials:
ARGS: argparse.Namespace
TRIALS: Trials
SEQ = {"n": 0}
+RAWLOGS: dict[pathlib.Path, rawlog.Writer] = {} # per trial; a restarted gateway resumes from the file
def log(msg: str) -> None:
@@ -214,11 +221,11 @@ def record(trial: pathlib.Path | None, rec: dict, raw: dict | None) -> None:
d = trial / "gateway"
d.mkdir(exist_ok=True)
if raw is not None:
- (d / "raw").mkdir(exist_ok=True)
- n = sum(1 for _ in (d / "raw").glob("*.json.gz")) + 1
- rec["raw_file"] = f"raw/{n:06d}.json.gz"
- with gzip.open(d / rec["raw_file"], "wt", encoding="utf-8") as f:
- json.dump(raw, f)
+ if d not in RAWLOGS:
+ if len(RAWLOGS) >= 256: # finished trials: drop the oldest table
+ del RAWLOGS[next(iter(RAWLOGS))]
+ RAWLOGS[d] = rawlog.Writer(d / rawlog.NAME)
+ rec["raw"] = RAWLOGS[d].append(raw)
with open(d / "usage.jsonl", "a") as f:
f.write(json.dumps(rec) + "\n")
tp = d / "totals.json"
@@ -366,7 +373,7 @@ def main() -> int:
p.add_argument("--jobs", type=pathlib.Path, required=True, help="Harbor's -o directory")
p.add_argument("--bind", default="172.17.0.1", help="the docker bridge address containers reach the host by")
p.add_argument("--port", type=int, default=0, help="0 = any free port (written to --ready-file)")
- p.add_argument("--raw", action="store_true", help="also record request and response bodies (raw/)")
+ p.add_argument("--raw", action="store_true", help="also record request and response bodies (raw.jsonl.gz)")
p.add_argument("--ready-file", type=pathlib.Path, help="written with the port once the gateway listens")
p.add_argument("--timeout", type=float, default=900.0, help="upstream socket timeout, seconds")
p.add_argument("--extra-host", action="append", default=[], help="another model API host pattern to forward to (repeatable)")
diff --git a/scripts/usage_gateway/rawlog.py b/scripts/usage_gateway/rawlog.py
new file mode 100644
index 0000000000..aaedbe5dda
--- /dev/null
+++ b/scripts/usage_gateway/rawlog.py
@@ -0,0 +1,356 @@
+#!/usr/bin/env python3
+"""Deduplicated raw log of a trial's model requests: <trial>/gateway/raw.jsonl.gz (raw.jsonl.xz once compacted),
+written by gateway.py --raw.
+
+An agent resends its whole conversation with every request, so one request is mostly the previous one: in a codex
+trial ~98% of a request's `input` items are byte-identical to items it already sent, and recording every request body
+in full grows quadratically (one 2,000-request trial: 5.9 GB of raw/NNNNNN.json.gz). Like the harnesses' own session
+files (codex's rollout-*.jsonl, Claude Code's session .jsonl), which store every conversation item once, this log stores
+every distinct piece of a request body once and each request as references to those pieces. It stays lossless: every
+request body and response is reconstructed byte for byte.
+
+Format: one JSON object per line. While the trial runs, the gateway appends them to raw.jsonl.gz as gzip members (one
+stream to zcat). A crash mid-write loses only that append: readers stop before an incomplete member, and a restarted
+gateway first cuts the file back to its last complete member, so nothing is appended behind a broken one. Each member is
+compressed on its own, so after the trial `compact` rewrites the log as one xz stream (raw.jsonl.xz, LZMA's 64 MB window
+also finds the repeats between requests, e.g. the request settings every response echoes): one 1,994-request trial is
+5.9 GB as raw/, 66 MB as raw.jsonl.gz, 32 MB as raw.jsonl.xz. Both are plain compressed JSON Lines (zcat / xzcat).
+
+ {"item": 7, "v": <JSON value>} a body piece (a list element or a large field), the first time it is sent
+ {"n": 12, "request": {...}, "response": {...}}
+ request n (1-based, the n of usage.jsonl's "raw"), with headers and the
+ response as recorded. Its body is either "body" (text, as before) or
+ "body_parts": [[key, "v", value] | [key, "i", item] | [key, "l", ranges],
+ ...] in key order plus "body_ascii" (how the JSON was serialized), where
+ ranges lists item ids, a run a..b written [a, b].
+
+A JSON body is split into parts only when re-serializing it reproduces the text exactly (compact separators, as codex,
+Claude Code and the OpenAI / Anthropic SDKs send); anything else is kept as text.
+
+ rawlog.py cat <gateway dir> [-n N] print the records in full ({"n", "request", "response"}), one per line
+ rawlog.py compact <dir>... raw.jsonl.gz -> raw.jsonl.xz for every gateway/ below <dir> (finished trials)
+ rawlog.py pack <dir>... [--delete] legacy raw/NNNNNN.json.gz -> raw.jsonl.xz; checks every record round-trips,
+ then (--delete) removes raw/
+ rawlog.py verify <dir>... decode every log below <dir>; compare with raw/ where it still exists
+"""
+from __future__ import annotations
+
+import argparse
+import gzip
+import hashlib
+import json
+import lzma
+import pathlib
+import sys
+import zlib
+from collections.abc import Iterator
+
+NAME = "raw.jsonl.gz" # live: gzip members appended by the gateway
+PACKED = "raw.jsonl.xz" # compacted: one xz stream
+XZ_PRESET = 9 # 64 MB dictionary; ~0.7 GB of memory while compressing
+INLINE_MAX = 256 # a non-list field whose JSON is shorter than this stays inline
+CHUNK = 1 << 20
+
+
+def dumps(v, ascii: bool = False) -> str:
+ return json.dumps(v, separators=(",", ":"), ensure_ascii=ascii)
+
+
+def line(obj) -> bytes:
+ """One log line. A lone surrogate (a JSON \\udXXX escape) cannot be UTF-8: such a line is written escaped."""
+ try:
+ return (dumps(obj) + "\n").encode()
+ except UnicodeEncodeError:
+ return (dumps(obj, True) + "\n").encode()
+
+
+def ranges(ids: list[int]) -> list:
+ out: list = []
+ for i in ids:
+ last = out[-1] if out else None
+ if isinstance(last, list) and last[1] + 1 == i:
+ last[1] = i
+ elif isinstance(last, int) and last + 1 == i:
+ out[-1] = [last, i]
+ else:
+ out.append(i)
+ return out
+
+
+def expand(rs: list) -> Iterator[int]:
+ for r in rs:
+ if isinstance(r, list):
+ yield from range(r[0], r[1] + 1)
+ else:
+ yield r
+
+
+def members(path: pathlib.Path) -> Iterator[tuple[int, list[dict]]]:
+ """(end offset, objects) of each complete gzip member of a raw.jsonl.gz (one append each), in order; stops at the
+ first incomplete or damaged member (a crash mid-write)."""
+ with open(path, "rb") as f:
+ start, pending = 0, b""
+ while True:
+ d = zlib.decompressobj(zlib.MAX_WBITS | 16)
+ out, fed = [], 0
+ chunk = pending or f.read(CHUNK)
+ while chunk:
+ fed += len(chunk)
+ try:
+ out.append(d.decompress(chunk))
+ except zlib.error:
+ return
+ if d.eof:
+ break
+ chunk = f.read(CHUNK)
+ if not d.eof:
+ return
+ pending = d.unused_data
+ end = start + fed - len(pending)
+ try:
+ objs = [json.loads(ln) for ln in b"".join(out).decode("utf-8").split("\n") if ln]
+ except ValueError: # UnicodeDecodeError included
+ return
+ yield end, objs
+ start = end
+
+
+def lines(path: pathlib.Path) -> Iterator[dict]:
+ """The log's objects; stops quietly at a truncated or damaged append (a crash mid-write)."""
+ if ".xz" not in path.name:
+ for _, objs in members(path):
+ yield from objs
+ return
+ try:
+ with lzma.open(path, "rt", encoding="utf-8") as f:
+ for line in f:
+ if not line.endswith("\n"):
+ return
+ yield json.loads(line)
+ except (EOFError, lzma.LZMAError, ValueError):
+ return
+
+
+class Writer:
+ """Appends records to one trial's raw.jsonl.gz; resumes the item table from an existing file, cut back to its last
+ complete append first (a crash mid-write leaves a partial gzip member, and anything appended behind it would be
+ unreadable)."""
+
+ def __init__(self, path: pathlib.Path):
+ self.path = path
+ self.ids: dict[bytes, int] = {}
+ self.n = 0
+ if path.exists():
+ end = 0
+ for end, objs in members(path):
+ for obj in objs:
+ if "item" in obj:
+ self.ids[self._key(obj["v"])] = obj["item"]
+ elif "n" in obj:
+ self.n = obj["n"]
+ size = path.stat().st_size
+ if size > end:
+ with open(path, "r+b") as f:
+ f.truncate(end)
+ print(f"rawlog: {path}: dropped {size - end} bytes of an incomplete append", file=sys.stderr)
+
+ @staticmethod
+ def _key(v) -> bytes:
+ return hashlib.sha256(dumps(v).encode("utf-8", "surrogatepass")).digest()
+
+ def _encode(self, text: str, new: list[bytes]) -> dict | None:
+ try:
+ obj = json.loads(text)
+ except ValueError:
+ return None
+ if not isinstance(obj, dict):
+ return None
+ ascii = dumps(obj) != text
+ if ascii and dumps(obj, True) != text:
+ return None
+
+ def ref(v) -> int:
+ k = self._key(v)
+ if k not in self.ids:
+ self.ids[k] = len(self.ids)
+ new.append(line({"item": self.ids[k], "v": v}))
+ return self.ids[k]
+
+ parts = []
+ for k, v in obj.items():
+ if len(dumps(v)) < INLINE_MAX:
+ parts.append([k, "v", v])
+ elif isinstance(v, list):
+ parts.append([k, "l", ranges([ref(x) for x in v])])
+ else:
+ parts.append([k, "i", ref(v)])
+ return {"body_parts": parts, "body_ascii": ascii}
+
+ def append(self, raw: dict) -> int:
+ """Record {"request": {..., "body"}, "response": {...}}; returns its n."""
+ new: list[bytes] = []
+ req = dict(raw["request"])
+ enc = self._encode(req["body"], new) if req.get("body_as", "text") == "text" else None
+ if enc is not None:
+ del req["body"]
+ req.update(enc)
+ self.n += 1
+ new.append(line({"n": self.n, "request": req, "response": raw["response"]}))
+ with gzip.open(self.path, "ab") as f:
+ f.write(b"".join(new))
+ return self.n
+
+
+def read(path: pathlib.Path) -> Iterator[dict]:
+ """The records of a raw.jsonl.gz in full: {"n", "request": {..., "body"}, "response"}."""
+ items: dict[int, object] = {}
+ for obj in lines(path):
+ if "item" in obj:
+ items[obj["item"]] = obj["v"]
+ continue
+ req = obj["request"]
+ parts = req.pop("body_parts", None)
+ if parts is not None:
+ body = {}
+ for k, kind, v in parts:
+ body[k] = v if kind == "v" else items[v] if kind == "i" else [items[i] for i in expand(v)]
+ ascii = req.pop("body_ascii")
+ req["body"] = dumps(body, ascii)
+ yield {"n": obj["n"], "request": req, "response": obj["response"]}
+
+
+def log_of(d: pathlib.Path) -> pathlib.Path | None:
+ for name in (PACKED, NAME):
+ if (d / name).exists():
+ return d / name
+ return None
+
+
+def gateway_dirs(paths: list[pathlib.Path]) -> Iterator[pathlib.Path]:
+ for p in paths:
+ if (p / "raw").is_dir() or log_of(p):
+ yield p
+ else:
+ yield from sorted(q for q in p.rglob("gateway") if q.is_dir())
+
+
+def legacy(d: pathlib.Path) -> list[pathlib.Path]:
+ return sorted((d / "raw").glob("*.json.gz")) if (d / "raw").is_dir() else []
+
+
+def remove(files: list[pathlib.Path]) -> None:
+ for f in files:
+ f.unlink()
+ files[0].parent.rmdir()
+
+
+def same(rec: dict, old: dict) -> bool:
+ return rec["request"] == old["request"] and rec["response"] == old["response"]
+
+
+def to_xz(src: pathlib.Path, out: pathlib.Path) -> None:
+ """Rewrite a raw.jsonl.gz as one xz stream (via out.tmp, so a half-written out never exists)."""
+ tmp = out.with_name(out.name + ".tmp")
+ with lzma.open(tmp, "wb", preset=XZ_PRESET) as f:
+ for obj in lines(src):
+ f.write(line(obj))
+ if sum(1 for _ in lines(src)) != sum(1 for _ in lines(tmp)):
+ tmp.unlink()
+ raise RuntimeError(f"{tmp}: line count differs")
+ tmp.rename(out)
+
+
+def compact(d: pathlib.Path) -> str:
+ src = d / NAME
+ if not src.exists():
+ return f"no {NAME}"
+ if (d / PACKED).exists():
+ return f"{PACKED} exists already, skipped"
+ to_xz(src, d / PACKED)
+ msg = f"{src.stat().st_size / 1e6:.1f} MB -> {(d / PACKED).stat().st_size / 1e6:.1f} MB"
+ src.unlink()
+ return msg
+
+
+def pack(d: pathlib.Path, delete: bool) -> str:
+ files = legacy(d)
+ if not files:
+ return "no raw/"
+ if log_of(d): # packed before: --delete removes raw/ once it matches
+ msg = verify(d)
+ if delete and "identical" in msg:
+ remove(files)
+ return msg + ", raw/ removed"
+ return msg if "differs" in msg else f"{log_of(d).name} exists already, skipped"
+ if [int(f.name.split(".")[0]) for f in files] != list(range(1, len(files) + 1)):
+ return "raw/ is not numbered 1..N, skipped"
+ tmp = d / (NAME + ".tmp")
+ tmp.unlink(missing_ok=True)
+ w = Writer(tmp)
+ for f in files:
+ w.append(json.loads(gzip.decompress(f.read_bytes())))
+ to_xz(tmp, d / PACKED)
+ tmp.unlink()
+ for f, rec in zip(files, read(d / PACKED), strict=True):
+ if not same(rec, json.loads(gzip.decompress(f.read_bytes()))):
+ (d / PACKED).unlink()
+ return f"round trip differs at {f.name}, skipped"
+ before = sum(f.stat().st_size for f in files)
+ msg = f"{len(files)} requests, {before / 1e6:.1f} MB -> {(d / PACKED).stat().st_size / 1e6:.1f} MB"
+ if delete:
+ remove(files)
+ msg += ", raw/ removed"
+ return msg
+
+
+def verify(d: pathlib.Path) -> str:
+ path = log_of(d)
+ if path is None:
+ return "no log"
+ files = legacy(d)
+ n = 0
+ for rec in read(path): # streamed: a long trial's records add up to gigabytes
+ n += 1
+ if rec["n"] != n:
+ return f"{path.name}: records out of order"
+ if files and n > len(files):
+ return f"{path.name}: more records than the {len(files)} raw/ files: differs"
+ if files and not same(rec, json.loads(gzip.decompress(files[n - 1].read_bytes()))):
+ return f"{path.name}: differs from raw/{files[n - 1].name}"
+ if files:
+ if len(files) != n:
+ return f"{path.name}: {n} records, {len(files)} raw/ files: differs"
+ return f"{path.name}: {n} records, identical to raw/"
+ return f"{path.name}: {n} records"
+
+
+def main() -> int:
+ p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ sub = p.add_subparsers(dest="cmd", required=True)
+ c = sub.add_parser("cat")
+ c.add_argument("dir", type=pathlib.Path)
+ c.add_argument("-n", type=int, help="only request n")
+ for name in ("compact", "pack", "verify"):
+ s = sub.add_parser(name)
+ s.add_argument("dirs", type=pathlib.Path, nargs="+")
+ if name == "pack":
+ s.add_argument("--delete", action="store_true", help="remove raw/ once the packed log round-trips")
+ a = p.parse_args()
+ if a.cmd == "cat":
+ path = log_of(a.dir) if a.dir.is_dir() else a.dir
+ if path is None:
+ sys.exit(f"{a.dir}: no {PACKED} or {NAME}")
+ for rec in read(path):
+ if a.n is None or rec["n"] == a.n:
+ sys.stdout.write(json.dumps(rec, ensure_ascii=False) + "\n")
+ return 0
+ bad = 0
+ for d in gateway_dirs(a.dirs):
+ msg = {"compact": compact, "verify": verify}[a.cmd](d) if a.cmd != "pack" else pack(d, a.delete)
+ bad += any(w in msg for w in ("differs", "out of order"))
+ print(f"{d}: {msg}", flush=True)
+ return 1 if bad else 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/usage_gateway/tests/test_rawlog.py b/scripts/usage_gateway/tests/test_rawlog.py
new file mode 100644
index 0000000000..20a742b6d0
--- /dev/null
+++ b/scripts/usage_gateway/tests/test_rawlog.py
@@ -0,0 +1,143 @@
+"""Tests for the deduplicated raw log (rawlog.py): every request body and response comes back byte for byte.
+
+ python -m unittest discover -s scripts/usage_gateway/tests # stdlib only
+"""
+from __future__ import annotations
+
+import gzip
+import json
+import pathlib
+import subprocess
+import sys
+import tempfile
+import unittest
+
+HERE = pathlib.Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(HERE))
+import rawlog # noqa: E402
+
+BIG = "x" * 2000
+
+
+def rec(body: str, how: str = "text", resp: str = "event: done\ndata: {}\n\n") -> dict:
+ return {"request": {"method": "POST", "host": "api.example.com", "path": "/v1/responses",
+ "headers": {"content-type": "application/json"}, "body": body, "body_as": how},
+ "response": {"status": 200, "headers": {"content-type": "text/event-stream"}, "body": resp}}
+
+
+def conversation() -> list[dict]:
+ """An OpenAI Responses and an Anthropic Messages conversation, interleaved, each request resending the history."""
+ out, inp, msgs = [], [], []
+ for i in range(6):
+ inp.append({"type": "message", "role": "user", "content": [{"type": "input_text", "text": f"{BIG} {i}"}]})
+ out.append(rec(json.dumps({"model": "gpt-x", "instructions": BIG, "input": inp, "stream": True},
+ separators=(",", ":"), ensure_ascii=False)))
+ # Claude Code moves cache_control to the newest message, so the previous last message changes every request
+ msgs = [{k: v for k, v in m.items() if k != "cache_control"} for m in msgs]
+ msgs.append({"role": "user", "content": f"é {BIG} {i}", "cache_control": {"type": "ephemeral"}})
+ out.append(rec(json.dumps({"model": "claude-x", "system": [{"type": "text", "text": BIG}], "messages": msgs,
+ "tools": [{"name": "bash", "description": BIG}], "max_tokens": 32000},
+ separators=(",", ":"), ensure_ascii=False), resp=f"event: message\ndata: {i}\n\n"))
+ return out
+
+
+class RawLogTest(unittest.TestCase):
+ def setUp(self):
+ self.tmp = tempfile.TemporaryDirectory()
+ self.d = pathlib.Path(self.tmp.name)
+
+ def tearDown(self):
+ self.tmp.cleanup()
+
+ def write(self, recs: list[dict]) -> rawlog.Writer:
+ w = rawlog.Writer(self.d / rawlog.NAME)
+ for r in recs:
+ w.append(r)
+ return w
+
+ def check(self, recs: list[dict], path: pathlib.Path | None = None) -> None:
+ got = list(rawlog.read(path or self.d / rawlog.NAME))
+ self.assertEqual([g["n"] for g in got], list(range(1, len(recs) + 1)))
+ for g, r in zip(got, recs, strict=True):
+ self.assertEqual(g["request"], r["request"])
+ self.assertEqual(g["response"], r["response"])
+
+ def test_round_trip_and_dedup(self):
+ recs = conversation()
+ self.write(recs)
+ self.check(recs)
+ full = sum(len(json.dumps(r)) for r in recs)
+ stored = len(gzip.decompress((self.d / rawlog.NAME).read_bytes()))
+ self.assertLess(stored, full / 2) # the history is stored once, not once per request
+
+ def test_bodies_kept_as_text(self):
+ recs = [rec('{"a": 1}'), # spaced JSON: does not re-serialize exactly
+ rec(json.dumps({"t": "é" + BIG}, separators=(",", ":"))), # é escapes: ascii serialization
+ rec('{"t":"\\ud800' + BIG + '"}'), # lone surrogate
+ rec("not json"), rec("[1,2,3]"), rec("AAEC", how="base64"), rec("")]
+ self.write(recs)
+ self.check(recs)
+
+ def test_resume_after_restart(self):
+ recs = conversation()
+ self.write(recs[:5])
+ w = self.write(recs[5:]) # a new gateway process picks up the item table and n
+ self.assertEqual(w.n, len(recs))
+ self.check(recs)
+
+ def test_truncated_last_append(self):
+ recs = conversation()
+ self.write(recs)
+ path = self.d / rawlog.NAME
+ path.write_bytes(path.read_bytes()[:-20])
+ got = list(rawlog.read(path))
+ self.assertEqual(len(got), len(recs) - 1)
+
+ def test_restart_after_crash_mid_write(self):
+ recs = conversation()
+ self.write(recs[:5])
+ path = self.d / rawlog.NAME
+ path.write_bytes(path.read_bytes()[:-20]) # the gateway died writing request 5
+ w = self.write(recs[5:]) # the restarted gateway cuts the partial member, then appends
+ self.assertEqual(w.n, len(recs) - 1)
+ got = list(rawlog.read(path))
+ self.assertEqual([g["n"] for g in got], list(range(1, len(recs))))
+ for g, r in zip(got, recs[:4] + recs[5:], strict=True):
+ self.assertEqual(g["request"], r["request"])
+ self.assertEqual(g["response"], r["response"])
+ self.assertIn("->", rawlog.compact(self.d))
+ self.assertEqual(len(list(rawlog.read(self.d / rawlog.PACKED))), len(recs) - 1)
+
+ def test_damaged_member_stops_reading(self):
+ recs = conversation()
+ self.write(recs[:3])
+ path = self.d / rawlog.NAME
+ path.write_bytes(path.read_bytes() + gzip.compress(b"x" * 5000)[:-30] + gzip.compress(b'{"n": 99}\n'))
+ self.assertEqual(len(list(rawlog.read(path))), 3) # no exception; nothing behind the broken member
+
+ def test_compact(self):
+ recs = conversation()
+ self.write(recs)
+ self.assertIn("->", rawlog.compact(self.d))
+ self.assertFalse((self.d / rawlog.NAME).exists())
+ self.check(recs, self.d / rawlog.PACKED)
+
+ def test_pack_legacy_cli(self):
+ recs = conversation()
+ (self.d / "raw").mkdir()
+ for i, r in enumerate(recs, 1):
+ (self.d / "raw" / f"{i:06d}.json.gz").write_bytes(gzip.compress(json.dumps(r).encode()))
+ tool = [sys.executable, str(HERE / "rawlog.py")]
+ out = subprocess.run(tool + ["pack", str(self.d)], capture_output=True, text=True, check=True).stdout
+ self.assertIn(f"{len(recs)} requests", out)
+ out = subprocess.run(tool + ["verify", str(self.d)], capture_output=True, text=True, check=True).stdout
+ self.assertIn("identical to raw/", out)
+ subprocess.run(tool + ["pack", "--delete", str(self.d)], check=True, capture_output=True) # verifies, removes
+ self.assertFalse((self.d / "raw").exists())
+ last = json.loads(subprocess.run(tool + ["cat", str(self.d), "-n", str(len(recs))], capture_output=True,
+ text=True, check=True).stdout)
+ self.assertEqual(last["request"], recs[-1]["request"])
+
+
+if __name__ == "__main__":
+ unittest.main()