The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
LIBERO-MAX: grade a standard episode once its replay verdict is in, not once its video is rendered; a timeout is a failure, not infra (+1 more) (#110, #111)
Kangrui Wang32035cdcc43c
diff --git a/scripts/libero-max/generate.py b/scripts/libero-max/generate.pyindex 1db133602b..1156cf18d0 100644--- a/scripts/libero-max/generate.py+++ b/scripts/libero-max/generate.py@@ -353,7 +353,7 @@ def make(case: dict, name: str, out: Path) -> tuple[Path, Path]:put(s / "environment/docker-compose.yaml", compose(instruction, episode))put(s / "environment/rcb_client.py", src=SHARED / "standard/rcb_client.py")put(s / "environment/rcb1.py", src=SHARED / "standard/rcb1.py")- for f in ("server.py", "rcb_service.py", "rcb1.py"):+ for f in ("server.py", "rcb_service.py", "rcb1.py", "finalize.sh"):put(s / "environment/sim" / f, src=SHARED / "standard" / f)put(s / "environment/sim/lmx_tool.py", src=SHARED / "lmx_tool.py")put(s / "environment/sim/instance.json", src=fz / "instance.json")diff --git a/scripts/libero-max/shared/standard/finalize.sh b/scripts/libero-max/shared/standard/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/scripts/libero-max/shared/standard/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/scripts/libero-max/shared/standard/grade.py b/scripts/libero-max/shared/standard/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/scripts/libero-max/shared/standard/grade.py+++ b/scripts/libero-max/shared/standard/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/scripts/libero-max/templates/standard/task.toml b/scripts/libero-max/templates/standard/task.tomlindex 76d185067f..278fd884de 100644--- a/scripts/libero-max/templates/standard/task.toml+++ b/scripts/libero-max/templates/standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/scripts/run_agent_batch.sh b/scripts/run_agent_batch.shindex c12fc2dc04..2191785d2f 100755--- a/scripts/run_agent_batch.sh+++ b/scripts/run_agent_batch.sh@@ -43,7 +43,9 @@# (<trial>/gateway/raw.jsonl.xz, deduplicated, see scripts/usage_gateway/rawlog.py; private: it holds the# whole conversation); 0 = token usage only# GATEWAY_BIND the docker bridge address the trial containers reach the host by (172.17.0.1)-# GATEWAY_EXTRA_HOSTS more model API host patterns the gateway forwards to, space-separated (e.g. "*.services.ai.azure.com")+# GATEWAY_EXTRA_HOSTS more model API host patterns the gateway forwards to, space-separated (e.g. "*.services.ai.azure.com");+# a plain-HTTP base URL with a port (ANTHROPIC_BASE_URL / OPENAI_BASE_URL=http://<host>:<port>, a self-hosted+# model server) is forwarded too, to exactly that <host>:<port># GATEWAY_START_WAIT seconds to wait for the gateway to come up (10); raise it on busy shared nodes# TOKEN_BUDGET_IN / TOKEN_BUDGET_OUT per-trial token budget the gateway enforces (100000000 / 10000000, PROTOCOL# §5): past it every further model request is refused and the trial counts as failed; 0 = no limit@@ -129,19 +131,23 @@ fi# transparent usage gateway (one per runner, stopped with it): the agent's base URL points at it, it forwards unchangedif [ "$GATEWAY" = 1 ]; thenmkdir -p "$JOBS/.gateway"; ready="$JOBS/.gateway/port.$$"; rm -f "$ready"- gw_args=(--jobs "$JOBS" --bind "$GATEWAY_BIND" --ready-file "$ready" --budget-input "$TOKEN_BUDGET_IN" --budget-output "$TOKEN_BUDGET_OUT"); for h in ${GATEWAY_EXTRA_HOSTS:-}; do gw_args+=(--extra-host "$h"); done; [ "$GATEWAY_RAW" = 1 ] && gw_args+=(--raw)- setsid python3 "$R/scripts/usage_gateway/gateway.py" "${gw_args[@]}" >>"$JOBS/.gateway/gateway.$$.err" 2>&1 </dev/null &- gw_pid=$!; trap 'kill "$gw_pid" 2>/dev/null' EXIT- for _ in $(seq $((GATEWAY_START_WAIT * 5))); do [ -s "$ready" ] && break; sleep 0.2; done- [ -s "$ready" ] || { log "usage gateway did not start within ${GATEWAY_START_WAIT} s (see $JOBS/.gateway/gateway.$$.err); unset GATEWAY to run without it"; exit 2; }- GW="http://$GATEWAY_BIND:$(cat "$ready")"# the upstream each base URL would have used: this environment, else the env file, else the API's defaultupstream() { # <VAR> <default>local v="${!1:-}"[ -z "$v" ] && [ -n "$ENV_FILE" ] && v=$(sed -n "s/^[[:space:]]*\(export[[:space:]]\+\)\?$1=//p" "$ENV_FILE" | tail -1 | tr -d "\"'")printf '%s' "${v:-$2}"}- via_gw() { sed -E "s#^https://([^/]+)#$GW/u/\1#" <<< "$1"; } # https://host/path -> $GW/u/host/path+ gw_args=(--jobs "$JOBS" --bind "$GATEWAY_BIND" --ready-file "$ready" --budget-input "$TOKEN_BUDGET_IN" --budget-output "$TOKEN_BUDGET_OUT"); for h in ${GATEWAY_EXTRA_HOSTS:-}; do gw_args+=(--extra-host "$h"); done; [ "$GATEWAY_RAW" = 1 ] && gw_args+=(--raw)+ # a plain-HTTP base URL (a self-hosted model server, e.g. sglang): the gateway may forward to exactly that host:port+ [ "$AGENT" = claude-code ] && agent_up=$(upstream ANTHROPIC_BASE_URL https://api.anthropic.com) || agent_up=$(upstream OPENAI_BASE_URL https://api.openai.com/v1)+ [[ "$agent_up" =~ ^http://([A-Za-z0-9.-]+:[0-9]+)(/|$) ]] && gw_args+=(--http-host "${BASH_REMATCH[1]}")+ setsid python3 "$R/scripts/usage_gateway/gateway.py" "${gw_args[@]}" >>"$JOBS/.gateway/gateway.$$.err" 2>&1 </dev/null &+ gw_pid=$!; trap 'kill "$gw_pid" 2>/dev/null' EXIT+ for _ in $(seq $((GATEWAY_START_WAIT * 5))); do [ -s "$ready" ] && break; sleep 0.2; done+ [ -s "$ready" ] || { log "usage gateway did not start within ${GATEWAY_START_WAIT} s (see $JOBS/.gateway/gateway.$$.err); unset GATEWAY to run without it"; exit 2; }+ GW="http://$GATEWAY_BIND:$(cat "$ready")"+ # https://host/path -> $GW/u/host/path; http://host:port/path -> $GW/h/host:port/path (`t`: one rewrite only)+ via_gw() { sed -E -e "s#^https://([^/]+)#$GW/u/\1#; t" -e "s#^http://([A-Za-z0-9.-]+:[0-9]+)#$GW/h/\1#" <<< "$1"; }if [ "$AGENT" = claude-code ]; thenAGENT_ARGS+=(--ae "ANTHROPIC_BASE_URL=$(via_gw "$(upstream ANTHROPIC_BASE_URL https://api.anthropic.com)")")elsediff --git a/scripts/usage_gateway/gateway.py b/scripts/usage_gateway/gateway.pyindex 075001fd04..a13f08456d 100755--- a/scripts/usage_gateway/gateway.py+++ b/scripts/usage_gateway/gateway.py@@ -8,6 +8,8 @@ and never logs or stores.Routes: /u/<host>/<path> -> https://<host>/<path>. <host> must be a model API (MODEL_HOSTS below, the hostsscripts/set_agent_egress.py allows), so the gateway can never carry an agent past its egress allowlist.+/h/<host>:<port>/<path> -> http://<host>:<port>/<path>, only for an exact <host>:<port> the runner named with --http-host+(a self-hosted model server without TLS, e.g. sglang's Anthropic-compatible API): the agent's own configured base URL.Attribution: a request's source IP is the trial's container (or, under an egress allowlist, its egress sidecar). Harbornames a trial's compose project `<trial>__env`, so `docker inspect` maps the IP to the project and the project to the@@ -64,6 +66,7 @@ def _model_hosts() -> list[str]:MODEL_HOSTS = _model_hosts() # extended at start-up by --extra-host (e.g. an Azure AI Foundry *.services.ai.azure.com)+HTTP_HOSTS: set[str] = set() # --http-host: exact <host>:<port> plain-HTTP upstreams (/h/ routes)HOP = {"connection", "keep-alive", "proxy-connection", "te", "trailers", "transfer-encoding", "upgrade", "host","content-length", "accept-encoding"}SECRET = {"authorization", "x-api-key", "api-key", "cookie", "set-cookie", "chatgpt-account-id", "openai-organization",@@ -266,10 +269,10 @@ def trial_totals(trial: pathlib.Path | None) -> dict:return {}-def budget_error(host: str, tot: dict) -> dict:+def budget_error(host: str, path: str, tot: dict) -> dict:msg = (f"token budget exceeded for this trial: input {tot.get('input', 0)} / {ARGS.budget_input}, output "f"{tot.get('output', 0)} / {ARGS.budget_output} (PROTOCOL §5); the run is over")- if "anthropic" in host:+ if "anthropic" in host or path.split("?")[0].endswith("/messages"): # Anthropic Messages API, any hostreturn {"type": "error", "error": {"type": "invalid_request_error", "message": msg}}return {"error": {"message": msg, "type": "invalid_request_error", "code": "token_budget_exceeded", "param": None}}@@ -298,7 +301,10 @@ class Handler(BaseHTTPRequestHandler):if self.path == "/_gateway/health":return self._reply(200, {"ok": True})m = re.match(r"^/u/([A-Za-z0-9.-]+)(/.*)?$", self.path)- if not m or not any(fnmatch.fnmatch(m.group(1).lower(), h) for h in MODEL_HOSTS):+ plain = re.match(r"^/h/([A-Za-z0-9.-]+:[0-9]+)(/.*)?$", self.path) # a --http-host upstream (no TLS)+ if plain and plain.group(1).lower() in HTTP_HOSTS:+ m = plain+ elif not m or not any(fnmatch.fnmatch(m.group(1).lower(), h) for h in MODEL_HOSTS):return self._reply(403, {"error": "usage gateway: not a model API host (scripts/set_agent_egress.py HOSTS)"})if (self.headers.get("Upgrade") or "").lower() == "websocket":return self._reply(405, {"error": "usage gateway: no WebSocket; use HTTPS (supports_websockets = false)"})@@ -313,14 +319,14 @@ class Handler(BaseHTTPRequestHandler):"path": path.split("?")[0], "status": 400, "model": None, "usage": {}, "usage_raw": {},"request_bytes": len(body), "response_bytes": 0, "client_gone": False,"refused": "over_token_budget"}, None)- return self._reply(400, budget_error(host, tot))+ return self._reply(400, budget_error(host, path, tot))hdrs = {k: v for k, v in self.headers.items() if k.lower() not in HOP}hdrs["Host"] = hostif body:hdrs["Content-Length"] = str(len(body))acc, chunks, client_gone = {}, [], Falsetry:- conn = http.client.HTTPSConnection(host, timeout=ARGS.timeout)+ conn = (http.client.HTTPConnection if m is plain else http.client.HTTPSConnection)(host, timeout=ARGS.timeout)conn.request(self.command, path, body=body or None, headers=hdrs)r = conn.getresponse()except Exception as exc: # noqa: BLE001 -- the agent sees an upstream error, as it would without the gateway@@ -377,17 +383,20 @@ def main() -> int:p.add_argument("--ready-file", type=pathlib.Path, help="written with the port once the gateway listens")p.add_argument("--timeout", type=float, default=900.0, help="upstream socket timeout, seconds")p.add_argument("--extra-host", action="append", default=[], help="another model API host pattern to forward to (repeatable)")+ p.add_argument("--http-host", action="append", default=[],+ help="a plain-HTTP model server <host>:<port>, reached at /h/<host>:<port>/... (repeatable, exact match)")p.add_argument("--budget-input", type=int, default=100_000_000, help="per-trial input tokens; 0 = no limit (D10)")p.add_argument("--budget-output", type=int, default=10_000_000, help="per-trial output tokens; 0 = no limit (D10)")ARGS = p.parse_args()ARGS.jobs = ARGS.jobs.resolve()MODEL_HOSTS.extend(h.lower() for h in ARGS.extra_host)+ HTTP_HOSTS.update(h.lower() for h in ARGS.http_host)(ARGS.jobs / ".gateway").mkdir(parents=True, exist_ok=True)TRIALS = Trials(ARGS.jobs)srv = ThreadingHTTPServer((ARGS.bind, ARGS.port), Handler)srv.daemon_threads = Trueport = srv.server_address[1]- log(f"listening on {ARGS.bind}:{port}, jobs {ARGS.jobs}, raw={ARGS.raw}, hosts {MODEL_HOSTS}")+ log(f"listening on {ARGS.bind}:{port}, jobs {ARGS.jobs}, raw={ARGS.raw}, hosts {MODEL_HOSTS}, http {sorted(HTTP_HOSTS)}")if ARGS.ready_file:ARGS.ready_file.write_text(str(port))srv.serve_forever()diff --git a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/task.toml b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/task.tomlindex c74197fbcb..8312cd2a18 100644--- a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/task.toml+++ b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/task.toml b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/task.tomlindex 64b5c73100..7b5cd89e62 100644--- a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/task.toml+++ b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.tomlindex 269d40e86d..f46ee4f3e4 100644--- a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml+++ b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/task.tomlindex 13fd0682c7..bc43dacd69 100644--- a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/task.toml+++ b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/task.tomlindex 024399b8cb..a781b96e27 100644--- a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/task.toml+++ b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/task.tomlindex fbb3cec19c..c2bb484abd 100644--- a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/task.toml+++ b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/task.toml b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/task.tomlindex 1ed05c21ca..5c60e236b0 100644--- a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/task.toml+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/task.toml b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/task.tomlindex 8f63f69208..79ebf43a86 100644--- a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/task.toml+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/tests/grade.py b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/tests/grade.py+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/task.tomlindex 9b61e441e9..dc16862e48 100644--- a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/task.toml+++ b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/task.tomlindex 3fb4c66f74..870811c95c 100644--- a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/task.toml+++ b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/task.tomlindex d1be85ed38..bfddd12265 100644--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/task.toml+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/task.tomlindex 0512673d84..1c294ae55f 100644--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/task.toml+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/task.tomlindex 4a7184ac85..55b3fc5abb 100644--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/task.toml+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/task.tomlindex e60206b1ed..bec1793918 100644--- a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/task.toml+++ b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/task.toml b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/task.tomlindex 3352a0d2b7..cd2d72eade 100644--- a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/task.toml+++ b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/tests/grade.py+++ b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/task.tomlindex 5b91403e5b..528f42d02f 100644--- a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/task.toml+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/task.tomlindex d4a6a820a6..6bb88d334a 100644--- a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/task.toml+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/task.tomlindex 2659b459e4..1a34846889 100644--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/task.toml+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/task.tomlindex 62f1c24e8b..fc8ab2ce12 100644--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/task.toml+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/task.tomlindex 847aefafbb..974cd266db 100644--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/task.toml+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/task.tomlindex 311e2596cb..b87b0b1abf 100644--- a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/task.toml+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/task.toml b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/task.tomlindex 001ec49921..f0aa22eeee 100644--- a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/task.toml+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/tests/grade.py b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/tests/grade.py+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/task.toml b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/task.tomlindex 410855d2b0..dcc7dc5efb 100644--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/task.toml+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/task.toml b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/task.tomlindex 5db119d0f1..081d397129 100644--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/task.toml+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/task.tomlindex 2b989d7315..7cf33399a0 100644--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/task.toml+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/task.toml b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/task.tomlindex d6f6320efb..60497f95d8 100644--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/task.toml+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/tests/grade.py b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/tests/grade.py+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/task.tomlindex 80ea9ce420..b977293975 100644--- a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/task.toml+++ b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/task.toml b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/task.tomlindex a340dc6da0..5b713fd932 100644--- a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/task.toml+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/task.toml b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/task.tomlindex 6e6f3b3d08..f1cdb86ddb 100644--- a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/task.toml+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/task.tomlindex 702cd01356..603329c14e 100644--- a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/task.toml+++ b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/task.toml b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/task.tomlindex 90da56e80b..84d982d549 100644--- a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/task.toml+++ b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/task.tomlindex cdf1d05bf4..73654d8662 100644--- a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/task.toml+++ b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/task.tomlindex 53527ee8dc..aba8de39c1 100644--- a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/task.toml+++ b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/task.tomlindex 88c078df11..f5e0804a6d 100644--- a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/task.toml+++ b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/task.tomlindex 896b4fc46e..321e5ef426 100644--- a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/task.toml+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/task.toml b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/task.tomlindex eb5fa89f27..9f43ce062e 100644--- a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/task.toml+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/tests/grade.py b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/tests/grade.py+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/task.tomlindex c2dff478bb..335d94fb4e 100644--- a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/task.toml+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/task.tomlindex adf5063798..7458c9b7fc 100644--- a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/task.toml+++ b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/task.tomlindex 281d1e2a88..1c975df20e 100644--- a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/task.toml+++ b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/task.toml b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/task.tomlindex 97a9d91be9..253659cb09 100644--- a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/task.toml+++ b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/task.tomlindex de3b7f0cd2..01a6859a47 100644--- a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/task.toml+++ b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/task.toml b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/task.tomlindex 4cbe8a9bce..bf045b4f31 100644--- a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/task.toml+++ b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/task.tomlindex a50e5e059b..c6eab22466 100644--- a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/task.toml+++ b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/task.toml b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/task.tomlindex 9e69ed8e52..8d77dd3fcd 100644--- a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/task.toml+++ b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/task.toml b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/task.tomlindex f709674136..4f0e4fdf20 100644--- a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/task.toml+++ b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/task.tomlindex 8cad9e02dc..02113b522d 100644--- a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/task.toml+++ b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/task.toml b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/task.tomlindex 447a134ca3..9253721eaa 100644--- a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/task.toml+++ b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/task.tomlindex bc8008abdc..06e7141f67 100644--- a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/task.toml+++ b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/tests/grade.py+++ b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/task.tomlindex 4934f965c2..06cb5e68fc 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/task.tomlindex ae4111275b..cffecf2621 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/task.tomlindex 929e7d3786..213c7b8bb1 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/task.tomlindex 056e9a56bd..d2d1e18213 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/task.tomlindex 4a63072127..32f641b158 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/task.tomlindex f723c635cc..1996d5db3c 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.tomlindex 055e5a7a33..e9ddf94272 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/task.tomlindex 194c863808..136f0f4db8 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/task.tomlindex 767236094a..37126c45d6 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/task.tomlindex c9e0d3da1f..30e1810e31 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/task.tomlindex 3e1d956059..517bb64830 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/task.tomlindex 2bf866819d..166d26a577 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/task.tomlindex 796ddf7141..6072969576 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/task.toml+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/task.tomlindex 682821ec3f..4c82ee94fe 100644--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/task.toml+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/task.toml b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/task.tomlindex a62c76197f..02fa2656f3 100644--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/task.toml+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/environment/sim/finalize.shnew file mode 100644index 0000000000..f280f5675e--- /dev/null+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh@@ -0,0 +1,30 @@+#!/bin/sh+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).+# 3. Stop waiting when final.json is finalized, or+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its+# replay says, so the replay is evidence only;+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.+# Prints final.json (the collect hook's output in the trial log).+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"+i=0+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done+t=0; verdict_at=""+while [ "$t" -lt "$WAIT_S" ]; do+ grep -q '"finalized": true' "$F" 2>/dev/null && break+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break+ fi+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null+ sleep 2; t=$((t + 2))+done+cat "$F"diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/task.tomlindex a7e325d415..ade046a6a4 100644--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/task.toml+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/task.toml@@ -42,11 +42,13 @@ service = "main"command = "kill -9 -1 2>/dev/null; exit 0"timeout_sec = 30-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.[[verifier.collect]]service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900+command = "sh /srv/finalize.sh"+timeout_sec = 3300[agent]timeout_sec = 3600.0diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/tests/grade.pyindex d3d7f6bf9e..f08edc9958 100644--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/tests/grade.py+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/tests/grade.py@@ -4,7 +4,14 @@success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, lastframe, trajectory, event log) next to them for humans; the evidence never decides the verdict.-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).++The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its+replay says, so it needs no replay.+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode+never ended, or a live success has no replay verdict (the replay did not finish in time or failed)."""import jsonimport os@@ -33,13 +40,15 @@ def main() -> None:with open(path) as f:d = json.load(f)live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))+ ended = bool(d.get("done"))+ replayed = "deterministic" in d # the fresh replay's verdict is on diskfailed_finalize = bool(d.get("finalize_error"))- unfinished = not d.get("finalized")- infra = bool(d.get("infra_error")) or failed_finalize or unfinished+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)ok = live and replay and det and not infraout = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),- "grader_error": int(failed_finalize),+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,+ "infra_error": int(infra), "grader_error": int(failed_finalize),+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),