Skip to content

The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.

LIBERO-MAX: grade a standard episode once its replay verdict is in, not once its video is rendered; a timeout is a failure, not infra (+1 more) (#110, #111)

Kangrui Wang32035cdcc43c
diff --git a/scripts/libero-max/generate.py b/scripts/libero-max/generate.py
index 1db133602b..1156cf18d0 100644
--- a/scripts/libero-max/generate.py
+++ b/scripts/libero-max/generate.py
@@ -353,7 +353,7 @@ def make(case: dict, name: str, out: Path) -> tuple[Path, Path]:
put(s / "environment/docker-compose.yaml", compose(instruction, episode))
put(s / "environment/rcb_client.py", src=SHARED / "standard/rcb_client.py")
put(s / "environment/rcb1.py", src=SHARED / "standard/rcb1.py")
- for f in ("server.py", "rcb_service.py", "rcb1.py"):
+ for f in ("server.py", "rcb_service.py", "rcb1.py", "finalize.sh"):
put(s / "environment/sim" / f, src=SHARED / "standard" / f)
put(s / "environment/sim/lmx_tool.py", src=SHARED / "lmx_tool.py")
put(s / "environment/sim/instance.json", src=fz / "instance.json")
diff --git a/scripts/libero-max/shared/standard/finalize.sh b/scripts/libero-max/shared/standard/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/scripts/libero-max/shared/standard/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/scripts/libero-max/shared/standard/grade.py b/scripts/libero-max/shared/standard/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/scripts/libero-max/shared/standard/grade.py
+++ b/scripts/libero-max/shared/standard/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/scripts/libero-max/templates/standard/task.toml b/scripts/libero-max/templates/standard/task.toml
index 76d185067f..278fd884de 100644
--- a/scripts/libero-max/templates/standard/task.toml
+++ b/scripts/libero-max/templates/standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/scripts/run_agent_batch.sh b/scripts/run_agent_batch.sh
index c12fc2dc04..2191785d2f 100755
--- a/scripts/run_agent_batch.sh
+++ b/scripts/run_agent_batch.sh
@@ -43,7 +43,9 @@
# (<trial>/gateway/raw.jsonl.xz, deduplicated, see scripts/usage_gateway/rawlog.py; private: it holds the
# whole conversation); 0 = token usage only
# GATEWAY_BIND the docker bridge address the trial containers reach the host by (172.17.0.1)
-# GATEWAY_EXTRA_HOSTS more model API host patterns the gateway forwards to, space-separated (e.g. "*.services.ai.azure.com")
+# GATEWAY_EXTRA_HOSTS more model API host patterns the gateway forwards to, space-separated (e.g. "*.services.ai.azure.com");
+# a plain-HTTP base URL with a port (ANTHROPIC_BASE_URL / OPENAI_BASE_URL=http://<host>:<port>, a self-hosted
+# model server) is forwarded too, to exactly that <host>:<port>
# GATEWAY_START_WAIT seconds to wait for the gateway to come up (10); raise it on busy shared nodes
# TOKEN_BUDGET_IN / TOKEN_BUDGET_OUT per-trial token budget the gateway enforces (100000000 / 10000000, PROTOCOL
# §5): past it every further model request is refused and the trial counts as failed; 0 = no limit
@@ -129,19 +131,23 @@ fi
# transparent usage gateway (one per runner, stopped with it): the agent's base URL points at it, it forwards unchanged
if [ "$GATEWAY" = 1 ]; then
mkdir -p "$JOBS/.gateway"; ready="$JOBS/.gateway/port.$$"; rm -f "$ready"
- gw_args=(--jobs "$JOBS" --bind "$GATEWAY_BIND" --ready-file "$ready" --budget-input "$TOKEN_BUDGET_IN" --budget-output "$TOKEN_BUDGET_OUT"); for h in ${GATEWAY_EXTRA_HOSTS:-}; do gw_args+=(--extra-host "$h"); done; [ "$GATEWAY_RAW" = 1 ] && gw_args+=(--raw)
- setsid python3 "$R/scripts/usage_gateway/gateway.py" "${gw_args[@]}" >>"$JOBS/.gateway/gateway.$$.err" 2>&1 </dev/null &
- gw_pid=$!; trap 'kill "$gw_pid" 2>/dev/null' EXIT
- for _ in $(seq $((GATEWAY_START_WAIT * 5))); do [ -s "$ready" ] && break; sleep 0.2; done
- [ -s "$ready" ] || { log "usage gateway did not start within ${GATEWAY_START_WAIT} s (see $JOBS/.gateway/gateway.$$.err); unset GATEWAY to run without it"; exit 2; }
- GW="http://$GATEWAY_BIND:$(cat "$ready")"
# the upstream each base URL would have used: this environment, else the env file, else the API's default
upstream() { # <VAR> <default>
local v="${!1:-}"
[ -z "$v" ] && [ -n "$ENV_FILE" ] && v=$(sed -n "s/^[[:space:]]*\(export[[:space:]]\+\)\?$1=//p" "$ENV_FILE" | tail -1 | tr -d "\"'")
printf '%s' "${v:-$2}"
}
- via_gw() { sed -E "s#^https://([^/]+)#$GW/u/\1#" <<< "$1"; } # https://host/path -> $GW/u/host/path
+ gw_args=(--jobs "$JOBS" --bind "$GATEWAY_BIND" --ready-file "$ready" --budget-input "$TOKEN_BUDGET_IN" --budget-output "$TOKEN_BUDGET_OUT"); for h in ${GATEWAY_EXTRA_HOSTS:-}; do gw_args+=(--extra-host "$h"); done; [ "$GATEWAY_RAW" = 1 ] && gw_args+=(--raw)
+ # a plain-HTTP base URL (a self-hosted model server, e.g. sglang): the gateway may forward to exactly that host:port
+ [ "$AGENT" = claude-code ] && agent_up=$(upstream ANTHROPIC_BASE_URL https://api.anthropic.com) || agent_up=$(upstream OPENAI_BASE_URL https://api.openai.com/v1)
+ [[ "$agent_up" =~ ^http://([A-Za-z0-9.-]+:[0-9]+)(/|$) ]] && gw_args+=(--http-host "${BASH_REMATCH[1]}")
+ setsid python3 "$R/scripts/usage_gateway/gateway.py" "${gw_args[@]}" >>"$JOBS/.gateway/gateway.$$.err" 2>&1 </dev/null &
+ gw_pid=$!; trap 'kill "$gw_pid" 2>/dev/null' EXIT
+ for _ in $(seq $((GATEWAY_START_WAIT * 5))); do [ -s "$ready" ] && break; sleep 0.2; done
+ [ -s "$ready" ] || { log "usage gateway did not start within ${GATEWAY_START_WAIT} s (see $JOBS/.gateway/gateway.$$.err); unset GATEWAY to run without it"; exit 2; }
+ GW="http://$GATEWAY_BIND:$(cat "$ready")"
+ # https://host/path -> $GW/u/host/path; http://host:port/path -> $GW/h/host:port/path (`t`: one rewrite only)
+ via_gw() { sed -E -e "s#^https://([^/]+)#$GW/u/\1#; t" -e "s#^http://([A-Za-z0-9.-]+:[0-9]+)#$GW/h/\1#" <<< "$1"; }
if [ "$AGENT" = claude-code ]; then
AGENT_ARGS+=(--ae "ANTHROPIC_BASE_URL=$(via_gw "$(upstream ANTHROPIC_BASE_URL https://api.anthropic.com)")")
else
diff --git a/scripts/usage_gateway/gateway.py b/scripts/usage_gateway/gateway.py
index 075001fd04..a13f08456d 100755
--- a/scripts/usage_gateway/gateway.py
+++ b/scripts/usage_gateway/gateway.py
@@ -8,6 +8,8 @@ and never logs or stores.
Routes: /u/<host>/<path> -> https://<host>/<path>. <host> must be a model API (MODEL_HOSTS below, the hosts
scripts/set_agent_egress.py allows), so the gateway can never carry an agent past its egress allowlist.
+/h/<host>:<port>/<path> -> http://<host>:<port>/<path>, only for an exact <host>:<port> the runner named with --http-host
+(a self-hosted model server without TLS, e.g. sglang's Anthropic-compatible API): the agent's own configured base URL.
Attribution: a request's source IP is the trial's container (or, under an egress allowlist, its egress sidecar). Harbor
names a trial's compose project `<trial>__env`, so `docker inspect` maps the IP to the project and the project to the
@@ -64,6 +66,7 @@ def _model_hosts() -> list[str]:
MODEL_HOSTS = _model_hosts() # extended at start-up by --extra-host (e.g. an Azure AI Foundry *.services.ai.azure.com)
+HTTP_HOSTS: set[str] = set() # --http-host: exact <host>:<port> plain-HTTP upstreams (/h/ routes)
HOP = {"connection", "keep-alive", "proxy-connection", "te", "trailers", "transfer-encoding", "upgrade", "host",
"content-length", "accept-encoding"}
SECRET = {"authorization", "x-api-key", "api-key", "cookie", "set-cookie", "chatgpt-account-id", "openai-organization",
@@ -266,10 +269,10 @@ def trial_totals(trial: pathlib.Path | None) -> dict:
return {}
-def budget_error(host: str, tot: dict) -> dict:
+def budget_error(host: str, path: str, tot: dict) -> dict:
msg = (f"token budget exceeded for this trial: input {tot.get('input', 0)} / {ARGS.budget_input}, output "
f"{tot.get('output', 0)} / {ARGS.budget_output} (PROTOCOL §5); the run is over")
- if "anthropic" in host:
+ if "anthropic" in host or path.split("?")[0].endswith("/messages"): # Anthropic Messages API, any host
return {"type": "error", "error": {"type": "invalid_request_error", "message": msg}}
return {"error": {"message": msg, "type": "invalid_request_error", "code": "token_budget_exceeded", "param": None}}
@@ -298,7 +301,10 @@ class Handler(BaseHTTPRequestHandler):
if self.path == "/_gateway/health":
return self._reply(200, {"ok": True})
m = re.match(r"^/u/([A-Za-z0-9.-]+)(/.*)?$", self.path)
- if not m or not any(fnmatch.fnmatch(m.group(1).lower(), h) for h in MODEL_HOSTS):
+ plain = re.match(r"^/h/([A-Za-z0-9.-]+:[0-9]+)(/.*)?$", self.path) # a --http-host upstream (no TLS)
+ if plain and plain.group(1).lower() in HTTP_HOSTS:
+ m = plain
+ elif not m or not any(fnmatch.fnmatch(m.group(1).lower(), h) for h in MODEL_HOSTS):
return self._reply(403, {"error": "usage gateway: not a model API host (scripts/set_agent_egress.py HOSTS)"})
if (self.headers.get("Upgrade") or "").lower() == "websocket":
return self._reply(405, {"error": "usage gateway: no WebSocket; use HTTPS (supports_websockets = false)"})
@@ -313,14 +319,14 @@ class Handler(BaseHTTPRequestHandler):
"path": path.split("?")[0], "status": 400, "model": None, "usage": {}, "usage_raw": {},
"request_bytes": len(body), "response_bytes": 0, "client_gone": False,
"refused": "over_token_budget"}, None)
- return self._reply(400, budget_error(host, tot))
+ return self._reply(400, budget_error(host, path, tot))
hdrs = {k: v for k, v in self.headers.items() if k.lower() not in HOP}
hdrs["Host"] = host
if body:
hdrs["Content-Length"] = str(len(body))
acc, chunks, client_gone = {}, [], False
try:
- conn = http.client.HTTPSConnection(host, timeout=ARGS.timeout)
+ conn = (http.client.HTTPConnection if m is plain else http.client.HTTPSConnection)(host, timeout=ARGS.timeout)
conn.request(self.command, path, body=body or None, headers=hdrs)
r = conn.getresponse()
except Exception as exc: # noqa: BLE001 -- the agent sees an upstream error, as it would without the gateway
@@ -377,17 +383,20 @@ def main() -> int:
p.add_argument("--ready-file", type=pathlib.Path, help="written with the port once the gateway listens")
p.add_argument("--timeout", type=float, default=900.0, help="upstream socket timeout, seconds")
p.add_argument("--extra-host", action="append", default=[], help="another model API host pattern to forward to (repeatable)")
+ p.add_argument("--http-host", action="append", default=[],
+ help="a plain-HTTP model server <host>:<port>, reached at /h/<host>:<port>/... (repeatable, exact match)")
p.add_argument("--budget-input", type=int, default=100_000_000, help="per-trial input tokens; 0 = no limit (D10)")
p.add_argument("--budget-output", type=int, default=10_000_000, help="per-trial output tokens; 0 = no limit (D10)")
ARGS = p.parse_args()
ARGS.jobs = ARGS.jobs.resolve()
MODEL_HOSTS.extend(h.lower() for h in ARGS.extra_host)
+ HTTP_HOSTS.update(h.lower() for h in ARGS.http_host)
(ARGS.jobs / ".gateway").mkdir(parents=True, exist_ok=True)
TRIALS = Trials(ARGS.jobs)
srv = ThreadingHTTPServer((ARGS.bind, ARGS.port), Handler)
srv.daemon_threads = True
port = srv.server_address[1]
- log(f"listening on {ARGS.bind}:{port}, jobs {ARGS.jobs}, raw={ARGS.raw}, hosts {MODEL_HOSTS}")
+ log(f"listening on {ARGS.bind}:{port}, jobs {ARGS.jobs}, raw={ARGS.raw}, hosts {MODEL_HOSTS}, http {sorted(HTTP_HOSTS)}")
if ARGS.ready_file:
ARGS.ready_file.write_text(str(port))
srv.serve_forever()
diff --git a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/task.toml b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/task.toml
index c74197fbcb..8312cd2a18 100644
--- a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-akita-black-bowl-in-wooden-cabinet-top-target-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/task.toml b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/task.toml
index 64b5c73100..7b5cd89e62 100644
--- a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-akita-black-bowl-on-flat-stove-cook-target-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml
index 269d40e86d..f46ee4f3e4 100644
--- a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/task.toml
index 13fd0682c7..bc43dacd69 100644
--- a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/task.toml
+++ b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-akita-black-bowl-on-wooden-cabinet-top-obstacle-insertion-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/task.toml
index 024399b8cb..a781b96e27 100644
--- a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/task.toml
+++ b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-cream-cheese-on-akita-black-bowl-obstacle-insertion-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/task.toml
index fbb3cec19c..c2bb484abd 100644
--- a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/task.toml
+++ b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-flat-stove-off-visual-theme-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/task.toml b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/task.toml
index 1ed05c21ca..5c60e236b0 100644
--- a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/task.toml
+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/task.toml b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/task.toml
index 8f63f69208..79ebf43a86 100644
--- a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/task.toml
+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/tests/grade.py b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/tests/grade.py
+++ b/tasks/libero-max-goal-flat-stove-on-illumination-switch-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/task.toml
index 9b61e441e9..dc16862e48 100644
--- a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/task.toml
+++ b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-flat-stove-on-sensor-noise-onset-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/task.toml
index 3fb4c66f74..870811c95c 100644
--- a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-wine-bottle-on-plate-receptacle-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/task.toml
index d1be85ed38..bfddd12265 100644
--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/task.toml
+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-camera-shift-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/task.toml
index 0512673d84..1c294ae55f 100644
--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/task.toml
+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-obstacle-insertion-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/task.toml
index 4a7184ac85..55b3fc5abb 100644
--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/task.toml
+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-wine-bottle-on-wine-rack-top-visual-theme-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/task.toml
index e60206b1ed..bec1793918 100644
--- a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/task.toml
+++ b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-wine-bottle-on-wooden-cabinet-top-sensor-noise-onset-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/task.toml b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/task.toml
index 3352a0d2b7..cd2d72eade 100644
--- a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/task.toml
+++ b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/tests/grade.py
+++ b/tasks/libero-max-goal-yellow-stove-on-camera-shift-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/task.toml
index 5b91403e5b..528f42d02f 100644
--- a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-receptacle-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/task.toml
index d4a6a820a6..6bb88d334a 100644
--- a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-alphabet-soup-and-cream-cheese-in-basket-target-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/task.toml
index 2659b459e4..1a34846889 100644
--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/task.toml
+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-distractor-burst-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/task.toml
index 62f1c24e8b..fc8ab2ce12 100644
--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-receptacle-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/task.toml
index 847aefafbb..974cd266db 100644
--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/task.toml
+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-alphabet-soup-and-tomato-sauce-in-basket-sensor-noise-onset-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/task.toml
index 311e2596cb..b87b0b1abf 100644
--- a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/task.toml
+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/task.toml b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/task.toml
index 001ec49921..f0aa22eeee 100644
--- a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/task.toml
+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/tests/grade.py b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/tests/grade.py
+++ b/tasks/libero-max-long-black-bowl-in-wooden-cabinet-bottom-closed-obstacle-insertion-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/task.toml b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/task.toml
index 410855d2b0..dcc7dc5efb 100644
--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/task.toml
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-camera-shift-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/task.toml b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/task.toml
index 5db119d0f1..081d397129 100644
--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/task.toml
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-illumination-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/task.toml
index 2b989d7315..7cf33399a0 100644
--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/task.toml
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/task.toml b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/task.toml
index d6f6320efb..60497f95d8 100644
--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/task.toml
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/tests/grade.py b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/tests/grade.py
+++ b/tasks/libero-max-long-porcelain-mug-and-white-yellow-mug-on-plate-visual-theme-switch-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/task.toml
index 80ea9ce420..b977293975 100644
--- a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/task.toml
+++ b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-white-yellow-mug-and-green-chocolate-pudding-on-yellow-plate-sensor-noise-onset-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/task.toml b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/task.toml
index a340dc6da0..5b713fd932 100644
--- a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/task.toml
+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-camera-shift-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/task.toml b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/task.toml
index 6e6f3b3d08..f1cdb86ddb 100644
--- a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/task.toml
+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-long-white-yellow-mug-in-desk-caddy-back-illumination-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/task.toml
index 702cd01356..603329c14e 100644
--- a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-alphabet-soup-in-basket-receptacle-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/task.toml b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/task.toml
index 90da56e80b..84d982d549 100644
--- a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-alphabet-soup-in-basket-target-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/task.toml
index cdf1d05bf4..73654d8662 100644
--- a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/task.toml
+++ b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-alphabet-soup-in-basket-visual-theme-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/task.toml
index 53527ee8dc..aba8de39c1 100644
--- a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/task.toml
+++ b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-butter-in-basket-distractor-burst-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/task.toml
index 88c078df11..f5e0804a6d 100644
--- a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-butter-in-basket-receptacle-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/task.toml
index 896b4fc46e..321e5ef426 100644
--- a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/task.toml
+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/task.toml b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/task.toml
index eb5fa89f27..9f43ce062e 100644
--- a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/task.toml
+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/tests/grade.py b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/tests/grade.py
+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-distractor-burst-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/task.toml
index c2dff478bb..335d94fb4e 100644
--- a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/task.toml
+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-chocolate-pudding-in-basket-sensor-noise-onset-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/task.toml
index adf5063798..7458c9b7fc 100644
--- a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/task.toml
+++ b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-cream-cheese-in-basket-visual-theme-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/task.toml
index 281d1e2a88..1c975df20e 100644
--- a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/task.toml
+++ b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-green-bbq-sauce-in-red-basket-distractor-burst-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/task.toml b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/task.toml
index 97a9d91be9..253659cb09 100644
--- a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/task.toml
+++ b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-green-chocolate-pudding-in-red-basket-camera-shift-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/task.toml
index de3b7f0cd2..01a6859a47 100644
--- a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/task.toml
+++ b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-ketchup-in-basket-distractor-burst-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/task.toml b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/task.toml
index 4cbe8a9bce..bf045b4f31 100644
--- a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/task.toml
+++ b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-milk-in-basket-camera-shift-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/task.toml
index a50e5e059b..c6eab22466 100644
--- a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/task.toml
+++ b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-milk-in-basket-distractor-burst-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/task.toml b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/task.toml
index 9e69ed8e52..8d77dd3fcd 100644
--- a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/task.toml
+++ b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-milk-in-basket-illumination-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/task.toml b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/task.toml
index f709674136..4f0e4fdf20 100644
--- a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-milk-in-basket-target-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/task.toml b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/task.toml
index 8cad9e02dc..02113b522d 100644
--- a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/task.toml
+++ b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/tests/grade.py b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-orange-juice-in-basket-distractor-burst-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/task.toml b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/task.toml
index 447a134ca3..9253721eaa 100644
--- a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/task.toml
+++ b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-orange-juice-in-basket-illumination-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/task.toml
index bc8008abdc..06e7141f67 100644
--- a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/task.toml
+++ b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/tests/grade.py
+++ b/tasks/libero-max-object-salad-dressing-in-basket-sensor-noise-onset-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/task.toml
index 4934f965c2..06cb5e68fc 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/task.toml
index ae4111275b..cffecf2621 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-camera-shift-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/task.toml
index 929e7d3786..213c7b8bb1 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/task.toml
index 056e9a56bd..d2d1e18213 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-illumination-switch-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/task.toml
index 4a63072127..32f641b158 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/task.toml
index f723c635cc..1996d5db3c 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-obstacle-insertion-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml
index 055e5a7a33..e9ddf94272 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/task.toml
index 194c863808..136f0f4db8 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-receptacle-relocation-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/task.toml
index 767236094a..37126c45d6 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/task.toml
index c9e0d3da1f..30e1810e31 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-sensor-noise-onset-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/task.toml
index 3e1d956059..517bb64830 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/task.toml
index 2bf866819d..166d26a577 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-target-relocation-i01-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/task.toml
index 796ddf7141..6072969576 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-akita-black-bowl-on-plate-visual-theme-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/task.toml b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/task.toml
index 682821ec3f..4c82ee94fe 100644
--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/tests/grade.py b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-obstacle-insertion-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/task.toml b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/task.toml
index a62c76197f..02fa2656f3 100644
--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/tests/grade.py b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-target-relocation-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),
diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh
new file mode 100644
index 0000000000..f280f5675e
--- /dev/null
+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/environment/sim/finalize.sh
@@ -0,0 +1,30 @@
+#!/bin/sh
+# [[verifier.collect]] hook, run inside the `sim` sidecar once the agent's container is gone (task.toml).
+# 1. Wait for the service's PID file (after a crash it appears once the service has resumed the episode).
+# 2. SIGUSR1, re-sent every 30 s in case the service was restarting: the service ends the episode ("timeout" unless it
+# ended already), replays it in a fresh simulator, writes the verdict to final.json ("deterministic" appears), then
+# renders the replay video and sets "finalized": true (rcb_service.py _finalize).
+# 3. Stop waiting when final.json is finalized, or
+# - RCB_VIDEO_GRACE_S after the verdict is on disk: the video is for humans and never decides the grade (grade.py);
+# - RCB_FAIL_WAIT_S after the start for an episode that ended without a live success: it is a failure whatever its
+# replay says, so the replay is evidence only;
+# - RCB_COLLECT_WAIT_S overall: a live success whose replay has not finished by then is graded infra_error.
+# The fresh replay renders the scene's cameras every control step (LIBERO's OffScreenRenderEnv), about 0.1-0.25 s per
+# step on a busy 4-CPU sidecar plus 1-3 min to build the scene: 3000 s covers the 9000-step episode cap.
+# Prints final.json (the collect hook's output in the trial log).
+DATA="${RCB_DATA:-/data}"; F="$DATA/final.json"
+WAIT_S="${RCB_COLLECT_WAIT_S:-3000}"; GRACE_S="${RCB_VIDEO_GRACE_S:-300}"; FAIL_S="${RCB_FAIL_WAIT_S:-900}"
+i=0
+while [ ! -f "$DATA/server.pid" ] && [ "$i" -lt 180 ]; do sleep 2; i=$((i + 2)); done
+t=0; verdict_at=""
+while [ "$t" -lt "$WAIT_S" ]; do
+ grep -q '"finalized": true' "$F" 2>/dev/null && break
+ if [ -z "$verdict_at" ] && grep -q '"deterministic"' "$F" 2>/dev/null; then verdict_at=$t; fi
+ [ -n "$verdict_at" ] && [ $((t - verdict_at)) -ge "$GRACE_S" ] && { echo "[finalize] verdict in; stop waiting for the video"; break; }
+ if [ "$t" -ge "$FAIL_S" ] && grep -q '"done": true' "$F" 2>/dev/null && grep -q '"live_success": false' "$F" 2>/dev/null; then
+ echo "[finalize] the episode ended without a live success; stop waiting for its replay"; break
+ fi
+ [ $((t % 30)) -eq 0 ] && kill -USR1 "$(cat "$DATA/server.pid" 2>/dev/null)" 2>/dev/null
+ sleep 2; t=$((t + 2))
+done
+cat "$F"
diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/task.toml b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/task.toml
index a7e325d415..ade046a6a4 100644
--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/task.toml
+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/task.toml
@@ -42,11 +42,13 @@ service = "main"
command = "kill -9 -1 2>/dev/null; exit 0"
timeout_sec = 30
-# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.
+# Then the episode ends: environment/sim/finalize.sh sends SIGUSR1 to the service (re-sent every 30 s, in case it was
+# restarting), which ends the episode, replays it in a fresh simulator (the verdict) and renders the video. The replay of a
+# long episode on a busy host takes many minutes; the hook waits for the verdict, not for the video.
[[verifier.collect]]
service = "sim"
-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"
-timeout_sec = 900
+command = "sh /srv/finalize.sh"
+timeout_sec = 3300
[agent]
timeout_sec = 3600.0
diff --git a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/tests/grade.py b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/tests/grade.py
index d3d7f6bf9e..f08edc9958 100644
--- a/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/tests/grade.py
+++ b/tasks/libero-max-spatial-black-bowl-on-yellow-plate-visual-theme-switch-i00-standard/tests/grade.py
@@ -4,7 +4,14 @@
success = live_success AND replay_success AND deterministic (the replay ends in the live run's final state).
Writes /logs/verifier/reward.txt (1/0) and a numeric-only reward.json, and copies the evidence (replay video, last
frame, trajectory, event log) next to them for humans; the evidence never decides the verdict.
-infra_error = 1: no verdict on the agent (the service crashed past its restarts, never finished, or failed to finalize).
+
+The verdict is in once the episode ended (`done`) and, for a live success, once the replay's verdict is on disk
+(`deterministic` present: rcb_service writes it before it renders the video). `finalized` only says the video is done
+too: a video still rendering when the collect hook stops waiting costs the video, never the verdict (PROTOCOL §2.6).
+An episode that ended without a live success (time up, the step cap, a simulator error) is a failure whatever its
+replay says, so it needs no replay.
+infra_error = 1: no verdict on the agent -- the service crashed past its restarts (its own infra_error), the episode
+never ended, or a live success has no replay verdict (the replay did not finish in time or failed).
"""
import json
import os
@@ -33,13 +40,15 @@ def main() -> None:
with open(path) as f:
d = json.load(f)
live, replay, det = (bool(d.get(k)) for k in ("live_success", "replay_success", "deterministic"))
+ ended = bool(d.get("done"))
+ replayed = "deterministic" in d # the fresh replay's verdict is on disk
failed_finalize = bool(d.get("finalize_error"))
- unfinished = not d.get("finalized")
- infra = bool(d.get("infra_error")) or failed_finalize or unfinished
+ infra = bool(d.get("infra_error")) or not ended or (live and not replayed)
ok = live and replay and det and not infra
out = {"success": int(ok), "live_success": int(live), "replay_success": int(replay), "deterministic": int(det),
- "finalized": int(not unfinished), "missing_log": 0, "infra_error": int(infra),
- "grader_error": int(failed_finalize),
+ "replayed": int(replayed), "finalized": int(bool(d.get("finalized"))), "missing_log": 0,
+ "infra_error": int(infra), "grader_error": int(failed_finalize),
+ "timeout": int(d.get("reason") == "timeout"), "sim_error": int(d.get("reason") == "error"),
"sim_crashed_on_call": int(d.get("reason") == "sim_crashed_on_call"),
"n_actions": int(d.get("n_actions") or 0), "sim_steps": int(d.get("control_steps_total") or 0),
"calls": int(d.get("calls") or 0), "rows": int(d.get("rows") or 0), "resets": int(d.get("resets") or 0),