The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Python ai-agentsbenchmarkcoding-agentsembodied-aievaluationharborllm-agentsrobot-learningroboticssimulation
Shared + HumanoidBench: rename the third mode realtime -> standard-realtime (task protocol v1.1.1, #97) (#98)
Kangrui Wang17cccee87c16
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.mdindex 1fbb566178..f81e505617 100644--- a/.github/pull_request_template.md+++ b/.github/pull_request_template.md@@ -9,7 +9,7 @@<!-- Written by you. For a fix: which agent behaviour it closes, with before/after runs. -->## Contract-- Mode(s): privileged / standard / realtime — protocol (standard: `eai-standard/2.1`/`2.2`; realtime: `eai-standard/2.3`): `eai-standard/2.__`+- Mode(s): privileged / standard / standard-realtime — protocol (standard: `eai-standard/2.1`/`2.2`/`2.4`; standard-realtime: `eai-standard/2.3`): `eai-standard/2.__`- Oracle: full / partial / none — evidence: ___- Base image(s): `eai-<sim>:<ver>` (published / BUILD_ONLY)- Agent time budget: ___ s (default 3600; reason if longer)@@ -21,7 +21,7 @@- [ ] New simulator or family: two fresh replays give the same state hash- [ ] Oracle `partial`/`none`: per-family table of positive examples (PROTOCOL §2.7)- [ ] Runs with Harbor's native agents (no custom runner, gateway or adapter)-- [ ] Names `<base>-privileged` / `<base>-standard` / `<base>-realtime`; README in the standard layout; no `results/`+- [ ] Names `<base>-privileged` / `<base>-standard` / `<base>-standard-realtime`; README in the standard layout; no `results/`- [ ] No credentials, personal paths or files > 2 MB; `tasks/dataset.toml` refreshed## Evidencediff --git a/AGENTS.md b/AGENTS.mdindex 7ba0307309..4e12123309 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -3,8 +3,8 @@Read `README.md`, `docs/PROTOCOL.md` (the contract, v1.1) and `CONTRIBUTING.md` first.- Terminology (protocol v1.0): **privileged** mode (formerly unlimited) and **standard** mode (formerly limited,- wire protocol `eai-standard/2.x`, formerly `rcb-limited/2.x`); v1.1 adds a third mode, **realtime** (standard mode on a- clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b). Task directories end in `-privileged`, `-standard` or `-realtime`;+ wire protocol `eai-standard/2.x`, formerly `rcb-limited/2.x`); v1.1 adds a third mode, **standard-realtime** (standard mode on a+ clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b). Task directories end in `-privileged`, `-standard` or `-standard-realtime`;base images are `ghcr.io/mll-lab-nu/eai-<name>`, one image per benchmark (PROTOCOL D5; formerly `rcb-<sim>`). No `results/` in task dirs.- Generators under `scripts/` end with `scripts/task_conventions.py` (`normalize`): fields, protocol spelling, test.sh, graders,README layout, no `results/`. `python3 scripts/task_conventions.py --check tasks/<task>` reports a task it would change.diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 7d0f26f5a0..54580c09fe 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -29,7 +29,7 @@ the expected size of the pull request.## Authoring a task1. Copy the closest existing pair (`tasks/robotwin_2_0_stack_blocks_two-privileged/` and `-standard/`) or generate tasks- with a script under `scripts/<bench>/`. Name them `<base>-privileged` / `<base>-standard` (and `<base>-realtime` for the realtime mode, PROTOCOL §2.2, §4b) and write+ with a script under `scripts/<bench>/`. Name them `<base>-privileged` / `<base>-standard` (and `<base>-standard-realtime` for the standard-realtime mode, PROTOCOL §2.2, §4b) and writethe README in the standard layout (§2.1); there is no `results/` directory.2. Fill in `task.toml` (PROTOCOL §2.3), `instruction.md` (§2.5, written by a human), the verifier under `tests/`(§2.6, separate sandbox) and `solution/` (§2.7).diff --git a/README.md b/README.mdindex 3916529fd3..fc3286fbb0 100644--- a/README.md+++ b/README.md@@ -26,13 +26,13 @@ Privileged mode measures whether an agent can program a robot when it has full ameasures the same thing under real-robot constraints. The standard-mode wire protocol is specified in[docs/LIMITED_MODE_PROTOCOL.md](docs/LIMITED_MODE_PROTOCOL.md).-A third mode, **realtime** (protocol v1.1, `eai-standard/2.3`, PROTOCOL §4b), is standard mode on a clock that does not+A third mode, **standard-realtime** (protocol v1.1, `eai-standard/2.3`, PROTOCOL §4b), is standard mode on a clock that does notstop: once an episode starts, the robot runs in real time and holds its last targets while the agent thinks, so the agentdrives it from a controller program of its own. It is offered where a benchmark's tasks call for it (HumanoidBench), as-`<base>-realtime` next to `<base>-privileged` and `<base>-standard`; its results are reported separately.+`<base>-standard-realtime` next to `<base>-privileged` and `<base>-standard`; its results are reported separately.> **Names.** Protocol v1.0 renamed *unlimited/limited* to *privileged/standard* and `rcb-limited/2.1` to-> `eai-standard/2.1`. Every task directory ends in its mode, `-privileged` or `-standard` (since v1.1 also `-realtime`). Base images are+> `eai-standard/2.1`. Every task directory ends in its mode, `-privileged` or `-standard` (since v1.1 also `-standard-realtime`). Base images are> `ghcr.io/mll-lab-nu/eai-<name>` (formerly `rcb-<sim>` under several registries), one per benchmark (PROTOCOL D5).## Benchmarks@@ -43,7 +43,7 @@ drives it from a controller program of its own. It is offered where a benchmark'| DexToolBench (`dextoolbench_<cat>_<obj>_<task>`) | MuJoCo 3.3.7 · KUKA iiwa14 + Sharpa HA4 hand | 21 of 24 | full on 6, partial on 15 || MuJoCo Playground (`mujoco_playground_<env>`) | MuJoCo · ALOHA, Franka Panda; quadrupeds: Unitree Go1, Boston Dynamics Spot, Google Barkour vB | 4 manipulation + 9 quadruped | full (manipulation: scripted IK; quadrupeds: Playground's Go1 policy or policies trained on its rewards) || KinDER (`kinder-<family>-i00`) | MuJoCo 3.3 / PyBullet · TidyBot++ mobile manipulator | 11 | full on 1, none on 10 (human review) |-| HumanoidBench (`humanoidbench-<family>-i00`) | MuJoCo · Unitree H1 humanoid | 24; + 10 realtime (`-realtime`: time does not stop) | full on 1, none on 23 (human review); realtime: full on 1, none on 9 |+| HumanoidBench (`humanoidbench-<family>-i00`) | MuJoCo · Unitree H1 humanoid | 24; + 10 standard-realtime (`-standard-realtime`: time does not stop) | full on 1, none on 23 (human review); standard-realtime: full on 1, none on 9 || MetaWorld+ (`metaworldplus-<env>-i00`) | MuJoCo (Meta-World v3) · Rethink Sawyer | 50 | full || RoboWits (`robowits-<family>-i00`) | Genesis · Marvin bimanual | 15 | partial on 2, none on 13 (human review) || RoboPaint (`robopaint-<style>-<target>-i00`) | ManiSkill 3 / SAPIEN + libmypaint · Franka Panda with a brush; the picture is written or painted with strokes, not coloured in (process rules) | 82 | partial |@@ -69,7 +69,7 @@ instance, oracle, budget, image), then Task, Difficulty, Reference solution, Ver- **RoboTwin generators:** `scripts/robotwin_expert_sweep.py`, `robotwin_mk_task.py`, `robotwin_mk_limited.py`.- **DexToolBench generators:** `scripts/dtb_mk_task.py`, `dtb_mk_oracle.py`.- **MuJoCo standard-mode twins:** generated by `scripts/mj_mk_limited.py`.-- **KinDER, HumanoidBench:** both modes (HumanoidBench: also realtime) rendered by `scripts/kinder/gen_tasks.py` and `scripts/humanoidbench/gen_tasks.py` (`render`, `check`).+- **KinDER, HumanoidBench:** both modes (HumanoidBench: also standard-realtime) rendered by `scripts/kinder/gen_tasks.py` and `scripts/humanoidbench/gen_tasks.py` (`render`, `check`).- **Other benchmarks:** each has its generator under `scripts/<bench>/` (RoboWits, RoboLab, BEHAVIOR-1K, RoboTangle: `mk_tasks.py --check`; VLABench, RoboCasa, RoboCasa365, RoboCasa-GR1: `generate.py --check`; MyoSuite: `scripts/myosuite/generate.py --check`; MetaWorld+: `generate.py`; RoboPaint: `scripts/robopaint_strict/mk_tasks.py --check`, which runs `scripts/robopaint_brush_mk_task.py --refresh` and adds the process rules). CI runs every check mode (`ci_checks/check-generated-tasks.sh`). Per-benchmark notes: [docs/benchmarks/](docs/benchmarks/).- **Conventions:** every generator ends with `scripts/task_conventions.py` (`--check` reports a task it would change).- **MuJoCo base image:** run `images/mujoco/prepare_assets.sh` before building it.diff --git a/ci_checks/check-instance-copies.sh b/ci_checks/check-instance-copies.shindex 9754963280..2eed434947 100755--- a/ci_checks/check-instance-copies.sh+++ b/ci_checks/check-instance-copies.sh@@ -8,7 +8,7 @@set -uo pipefailt="${1%/}"; n="$(basename "$t")"; rc=0case "$n" in- *-realtime) ref="${n%-realtime}-privileged/tests"; copy="environment/sim" ;; # realtime mode: a sim sidecar too+ *-standard-realtime) ref="${n%-standard-realtime}-privileged/tests"; copy="environment/sim" ;; # a sim sidecar too*-standard) ref="${n%-standard}-privileged/tests"; copy="environment/sim" ;;robopaint-*) ref="$n/tests"; copy="environment/scorer" ;;*) ref="$n/tests"; copy="environment" ;;diff --git a/ci_checks/check-protocol-v1.py b/ci_checks/check-protocol-v1.pyindex 75d206d5b0..5a5979c11c 100755--- a/ci_checks/check-protocol-v1.py+++ b/ci_checks/check-protocol-v1.py@@ -19,7 +19,7 @@ from pathlib import PathROOT = Path(__file__).resolve().parent.parentREGISTRY = "ghcr.io/mll-lab-nu/eai-"PROTOCOLS = {"eai-standard/2.1", "eai-standard/2.2", "eai-standard/2.4"} # standard mode (2.4: the operations capability, issue #89)-REALTIME_PROTOCOLS = {"eai-standard/2.3"} # realtime mode (PROTOCOL §4b): 2.3 is the realtime capability+REALTIME_PROTOCOLS = {"eai-standard/2.3"} # standard-realtime mode (PROTOCOL §4b): 2.3 = the realtime capabilityLEGACY_PROTOCOLS = {"rcb-limited/2.1"}PERSONAL_PATH = re.compile(rb"/(home|Users)/[A-Za-z0-9_.-]+/")TEXT_SUFFIXES = {".md", ".toml", ".py", ".sh", ".yaml", ".yml", ".json", ".jsonl", ".txt", ".cfg", ".xml", ".urdf", ""}@@ -60,18 +60,21 @@ def check_task(task: Path) -> None:if name.endswith("-limited") or mode == "limited":warn("mode name `limited` / `-limited` (use `standard` / `-standard`)", name)if mode is None:- warn("missing `mode` (privileged | standard | realtime)", name)+ warn("missing `mode` (privileged | standard | standard-realtime)", name)if "tier" in meta:warn("deprecated `tier` field (use `mode`)", name)- realtime = mode == "realtime" or name.endswith("-realtime")+ # whole suffixes only: `-standard-realtime` does not end in `-standard`, and `mode` is compared exactly+ realtime = mode == "standard-realtime" or name.endswith("-standard-realtime")standard = not realtime and (mode in ("standard", "limited") or name.endswith(("-limited", "-standard")))+ if name.endswith("-realtime") and not name.endswith("-standard-realtime") or mode == "realtime":+ warn("mode name `realtime` / `-realtime` (the third mode is `standard-realtime`, PROTOCOL D12)", name)proto = meta.get("protocol")if standard and proto in LEGACY_PROTOCOLS:warn("protocol spelled `rcb-limited/2.1` (use `eai-standard/2.1`)", name)elif standard and proto not in PROTOCOLS:warn(f"standard-mode `protocol` not in {sorted(PROTOCOLS)}", f"{name} ({proto})")if realtime and proto not in REALTIME_PROTOCOLS:- warn(f"realtime-mode `protocol` not in {sorted(REALTIME_PROTOCOLS)}", f"{name} ({proto})")+ warn(f"standard-realtime `protocol` not in {sorted(REALTIME_PROTOCOLS)}", f"{name} ({proto})")if not standard and not realtime and proto is not None:warn("`protocol` set on a privileged task", name)@@ -128,11 +131,11 @@ def main() -> int:if not args:check_repository()if not hits:- print(f"protocol v1.1: {len(tasks)} task(s) conform")+ print(f"protocol v1.1.1: {len(tasks)} task(s) conform")return 0strict = os.environ.get("RCB_PROTOCOL_STRICT") == "1"hard = {r: w for r, w in hits.items() if not r.startswith("legacy")}- print(f"protocol v1.1: {'violations' if hard or strict else 'warnings'} "+ print(f"protocol v1.1.1: {'violations' if hard or strict else 'warnings'} ""(`legacy:` rules are accepted until the image move, docs/PROTOCOL.md §7; all others fail):")for rule, where in sorted(hits.items(), key=lambda kv: (not kv[0].startswith("legacy"), -len(kv[1]))):examples = ", ".join(where[:3]) + (", ..." if len(where) > 3 else "")diff --git a/ci_checks/check-standard-agent-image.sh b/ci_checks/check-standard-agent-image.shindex 8eb8b0d8a2..1da357eb1f 100755--- a/ci_checks/check-standard-agent-image.sh+++ b/ci_checks/check-standard-agent-image.sh@@ -1,12 +1,12 @@#!/usr/bin/env bash-# Standard and realtime mode (PROTOCOL §4.1, §4b: no simulator or simulation code in the agent container): the agent's image is built+# Standard and standard-realtime mode (PROTOCOL §4.1, §4b: no simulator or simulation code in the agent container): the agent's image is built# from environment/, but never takes environment/sim/, which compose mounts into the sim sidecar only (and which may# hold the scoring code). Every COPY / ADD source of environment/Dockerfile is therefore a named path outside sim/:# not `.` or `..`, not a wildcard. Sources from another stage or image (--from) and ADD URLs are not from environment/.# Usage: check-standard-agent-image.sh tasks/<task-dir>set -uo pipefailt="${1%/}"-case "$(basename "$t")" in *-standard|*-realtime) ;; *) exit 0 ;; esac+case "$(basename "$t")" in *-standard|*-standard-realtime) ;; *) exit 0 ;; esacpython3 - "$t" <<'PY'import json, pathlib, posixpath, re, syst = pathlib.Path(sys.argv[1]); rc = 0diff --git a/ci_checks/check-standard-client.sh b/ci_checks/check-standard-client.shindex 078d9c002b..37abf073e9 100755--- a/ci_checks/check-standard-client.sh+++ b/ci_checks/check-standard-client.sh@@ -1,8 +1,8 @@#!/usr/bin/env bash# PROTOCOL §2.4: the agent container and the verifier of every standard task share ONE client base image, pinned by# digest, and pinned pip versions. The image and the pins are scripts/eai_standard/client.env (CLIENT_IMAGE) and-# scripts/eai_standard/client-requirements.txt. For each standard or realtime task (mode standard / limited / realtime, or-# a -standard / -limited / -realtime name), environment/Dockerfile and tests/Dockerfile:+# scripts/eai_standard/client-requirements.txt. For each standard or standard-realtime task (mode standard / limited /+# standard-realtime, or a -standard / -limited / -standard-realtime name), environment/Dockerfile and tests/Dockerfile:# - every FROM is CLIENT_IMAGE;# - every `pip install` installs the pins: `-r <file>` with a byte-identical copy of client-requirements.txt in the# build context, or the full pin list inline (name==version for every pin, nothing else).@@ -75,7 +75,7 @@ for t in sorted(os.listdir(tasks)):if not os.path.isfile(tt):continuemeta = tomllib.load(open(tt, "rb")).get("metadata", {})- if not (meta.get("mode") in ("standard", "limited", "realtime") or t.endswith(("-standard", "-limited", "-realtime"))):+ if not (meta.get("mode") in ("standard", "limited", "standard-realtime") or t.endswith(("-standard", "-limited", "-standard-realtime"))):continueprobs, froms = [], []for sub in ("environment", "tests"):diff --git a/ci_checks/check-task-fields.sh b/ci_checks/check-task-fields.shindex 04c316b455..dcd9c63c8d 100755--- a/ci_checks/check-task-fields.sh+++ b/ci_checks/check-task-fields.sh@@ -10,12 +10,12 @@ m = d.get("metadata", {})for k in ["author_name", "task_type", "simulator", "robot", "family", "expert_time_estimate_hours"]:if k not in m: fail(f"[metadata].{k} missing")if m.get("task_type") not in ("trajectory", "policy"): fail("task_type must be trajectory|policy")-# Protocol v1.1 (§2.2, §2.3, D12): the directory ends in its mode (-privileged, -standard, -realtime) and `mode` says the-# same; no `tier`.-MODES = ("privileged", "standard", "realtime")-suffix = t.name.rsplit("-", 1)[-1]-if suffix not in MODES: fail("directory name must end in -privileged, -standard or -realtime")-if m.get("mode") not in MODES: fail("[metadata].mode must be privileged|standard|realtime")+# Protocol v1.1.1 (§2.2, §2.3, D12): the directory ends in its mode (-privileged, -standard, -standard-realtime) and+# `mode` says the same; no `tier`. The suffix is the whole mode name, longest first: -standard-realtime is not -realtime.+MODES = ("standard-realtime", "privileged", "standard")+suffix = next((x for x in MODES if t.name.endswith("-" + x)), t.name.rsplit("-", 1)[-1])+if suffix not in MODES: fail("directory name must end in -privileged, -standard or -standard-realtime")+if m.get("mode") not in MODES: fail("[metadata].mode must be privileged|standard|standard-realtime")elif m["mode"] != suffix: fail(f"[metadata].mode = {m['mode']!r} but the directory ends in -{suffix}")if "tier" in m: fail("[metadata].tier was removed in protocol v1.0 (use mode)")if "/app/output/" not in d.get("artifacts", []): fail('top-level artifacts must include "/app/output/"')diff --git a/ci_checks/check-tooling-identity.sh b/ci_checks/check-tooling-identity.shindex e55d761716..16d0537604 100755--- a/ci_checks/check-tooling-identity.sh+++ b/ci_checks/check-tooling-identity.sh@@ -77,7 +77,8 @@ for t in sorted(os.listdir(tasks)):if bench not in MODULE:continuedirs[bench] += 1- mode = "realtime" if t.endswith("-realtime") else "standard" if t.endswith(("-standard", "-limited")) else "privileged"+ mode = ("standard-realtime" if t.endswith("-standard-realtime") else "standard" if t.endswith(("-standard", "-limited"))+ else "privileged")for dp, dns, fns in os.walk(os.path.join(tasks, t)):dns[:] = [d for d in dns if d not in ("results", "__pycache__")]for f in fns:diff --git a/docs/LIMITED_MODE_PROTOCOL.md b/docs/LIMITED_MODE_PROTOCOL.mdindex aca6f3a799..ee51a2059b 100644--- a/docs/LIMITED_MODE_PROTOCOL.md+++ b/docs/LIMITED_MODE_PROTOCOL.md@@ -161,9 +161,9 @@ same frozen state, `N` times in all; `/status` and `meta` add `episode`, `resetssuccess or once the agent's time is up, `/reset` returns 409, as it does when the budget is spent. Only the lastepisode is graded. Results with `N > 0` are reported separately and never ranked with `N = 0` (PROTOCOL D1, §4.5).-## Realtime mode: `eai-standard/2.3`-Time does not stop. The wire protocol of the realtime mode (PROTOCOL §4b, v1.1; protocol-change issue #86), served by-realtime tasks only (`<base>-realtime`, `mode = "realtime"`, `protocol = "eai-standard/2.3"`; implemented by HumanoidBench's,+## Standard-realtime mode: `eai-standard/2.3`+Time does not stop. The wire protocol of the standard-realtime mode (PROTOCOL §4b, v1.1; protocol-change issue #86), served by+standard-realtime tasks only (`<base>-standard-realtime`, `mode = "standard-realtime"`, `protocol = "eai-standard/2.3"`; implemented by HumanoidBench's,`scripts/humanoidbench/shared/realtime/rcb_realtime.py`). 2.3 is 2.2 plus the realtime capability below; every 2.2addition stays a MAY, so a 2.1 service with this capability is a 2.3 service. Same endpoints, RCB1 frames, crashrecovery and grading as 2.1. `/spec.timing` declares it: `realtime: true`, `slowdown` S (`RCB_SLOWDOWN`, default 1 = real@@ -202,4 +202,4 @@ agent latency counts, the agent's own program may act) and uses its `slowdown` n`/act`, `/wait`, `/ready`, seats, reviews or metering proxy; native agents only (PROTOCOL D11).## Versioning-`protocol` in `/spec`. 1.0 (`/step` with one action and base64 images) is retired; 2.0 = `/submit` + `/observe` + RCB1; 2.1 = 2.0 + measured proprioception (82 columns) + optional depth noise + crash recovery; `eai-standard/2.1` is the v1.0 name of 2.1. Minor versions only add; the 2.x client accepts every 2.x version under both spellings. Accepted, pending: `eai-standard/2.2` (PROTOCOL §6). `eai-standard/2.3` (2026-10-08) = 2.2 + the realtime capability, the realtime mode's protocol (above). `eai-standard/2.4` (2026-10-08) = 2.3 + the operations capability (operation ids, request deadlines, `GET /operations/<id>`, call watchdog, tick ledger; [STANDARD_MODE_2_4.md](STANDARD_MODE_2_4.md)), for standard tasks.+`protocol` in `/spec`. 1.0 (`/step` with one action and base64 images) is retired; 2.0 = `/submit` + `/observe` + RCB1; 2.1 = 2.0 + measured proprioception (82 columns) + optional depth noise + crash recovery; `eai-standard/2.1` is the v1.0 name of 2.1. Minor versions only add; the 2.x client accepts every 2.x version under both spellings. Accepted, pending: `eai-standard/2.2` (PROTOCOL §6). `eai-standard/2.3` (2026-10-08) = 2.2 + the realtime capability, the standard-realtime mode's protocol (above). `eai-standard/2.4` (2026-10-08) = 2.3 + the operations capability (operation ids, request deadlines, `GET /operations/<id>`, call watchdog, tick ledger; [STANDARD_MODE_2_4.md](STANDARD_MODE_2_4.md)), for standard tasks.diff --git a/docs/PROTOCOL.md b/docs/PROTOCOL.mdindex e9d76b5182..38194f4a93 100644--- a/docs/PROTOCOL.md+++ b/docs/PROTOCOL.md@@ -1,15 +1,16 @@-# Robot Coding Bench — Task Protocol v1.1+# Robot Coding Bench — Task Protocol v1.1.1Status: **final** (2026-09-30; v1.0.1 2026-10-01: both modes carry a suffix, no results in the repository, standard-task README; **v1.1 2026-10-08: a third mode, realtime (§4b), served by `eai-standard/2.3`; protocol-change issue #86**).+task README; **v1.1 2026-10-08: a third mode (§4b), served by `eai-standard/2.3`; protocol-change issue #86; v1.1.1 2026-10-08: the third mode is named+`standard-realtime`, issue #97**).The tasks in the repository follow it except for the open items in §7. The wire format of standard mode is specified in-[`LIMITED_MODE_PROTOCOL.md`](LIMITED_MODE_PROTOCOL.md) (`eai-standard/2.1`, and the realtime mode's `eai-standard/2.3`);+[`LIMITED_MODE_PROTOCOL.md`](LIMITED_MODE_PROTOCOL.md) (`eai-standard/2.1`, and the standard-realtime mode's `eai-standard/2.3`);its 2.2 extension in [`STANDARD_MODE_2_2.md`](STANDARD_MODE_2_2.md), and the 2.4 operations capability in[`STANDARD_MODE_2_4.md`](STANDARD_MODE_2_4.md) (2026-10-08, issue #89).> **Names.** v1.0 renamed the two modes (unlimited → **privileged**, limited → **standard**) and the standard-mode> protocol (`rcb-limited/2.x` → **`eai-standard/2.x`**); task directories end in `-privileged` or `-standard`. Base images-> moved to `ghcr.io/mll-lab-nu/eai-<name>` on 2026-10-02 (§7). v1.1 added a third mode, **realtime** (`-realtime`): standard+> moved to `ghcr.io/mll-lab-nu/eai-<name>` on 2026-10-02 (§7). v1.1 added a third mode, **standard-realtime** (`-standard-realtime`): standard> mode's robot as a service, on a clock that does not stop.## 0. Conventions@@ -34,7 +35,7 @@ is the task's own business.|---|---|---|| **privileged mode** | Agent and simulator share a container; the agent may read privileged state, reset freely, call planning primitives. The deliverable is a trajectory. | unlimited || **standard mode** | Robot as a service: the simulator runs in a sidecar the agent cannot open; the agent only sends actions and receives images and proprioception through a client; no reset, unless the benchmark grants a reset budget (§4.5). | limited |-| **realtime mode** | Standard mode on a clock that does not stop: once an episode starts, the robot runs in real time (divided by the slowdown S) whatever the agent does, holds its last targets while the agent thinks, and the agent drives it from a program of its own (§4b, `eai-standard/2.3`). | — (v1.1) |+| **standard-realtime mode** | Standard mode on a clock that does not stop: once an episode starts, the robot runs in real time (divided by the slowdown S) whatever the agent does, holds its last targets while the agent thinks, and the agent drives it from a program of its own (§4b, `eai-standard/2.3`). | — (v1.1) || **`eai-standard/2.x`** | The standard-mode wire protocol (§4). | `rcb-limited/2.x` || **`eai-<name>`** | Base image (a simulator stack, usually one per benchmark) under `ghcr.io/mll-lab-nu/` (§2.4). | `rcb-<sim>` under several registries; `eai-<name>` under `embodied-agent-interface-v2-internal` until 2026-10-02 |@@ -45,7 +46,7 @@ Unchanged: the task name prefix `robot-coding-bench/`, the client and tool file| # | Topic | Decision ||---|---|---|-| D1 | Standard-mode protocol | Only `eai-standard/2.x`. Additive extensions are accepted through a protocol-change PR (2.2 is accepted once its services implement crash recovery). Reset is only an optional capability (`budget.reset = N`, default 0; results with N > 0 are reported separately). Documented per-benchmark exception (owner, 2026-10-07): HumanoidBench standard defaults to N = 50 (§4.5). The realtime mode's `eai-standard/2.3` (§4b) is 2.2 plus the realtime capability. `eai-standard/2.4` (issue #89) is 2.3 plus the operations capability; standard tasks may use it. No step caps. |+| D1 | Standard-mode protocol | Only `eai-standard/2.x`. Additive extensions are accepted through a protocol-change PR (2.2 is accepted once its services implement crash recovery). Reset is only an optional capability (`budget.reset = N`, default 0; results with N > 0 are reported separately). Documented per-benchmark exception (owner, 2026-10-07): HumanoidBench standard defaults to N = 50 (§4.5). The standard-realtime mode's `eai-standard/2.3` (§4b) is 2.2 plus the realtime capability. `eai-standard/2.4` (issue #89) is 2.3 plus the operations capability; standard tasks may use it. No step caps. || D2 | Tasks without an oracle | Allowed, with mandatory human review (§2.7). || D3 | Results | Not in the repository: agent results (per-task READMEs, videos, timelines, batch tables) are published on the results site and the Hugging Face dataset; a task directory has no `results/`. || D4 | Run id | `<harness>-<model>-<effort>`, model token with underscores (`codex-gpt6_luna-xhigh`); provider, tier and settings go in metadata. |@@ -57,7 +58,7 @@ Unchanged: the task name prefix `robot-coding-bench/`, the client and tool file| D10 | Token budget | Per trial: **100M input tokens, 10M output tokens** by default; exceeding either fails the trial. || D11 | Harness | Official results use Harbor's native agents only; no agent adapters, and no gateway that holds credentials or changes requests. A transparent usage gateway that forwards requests unchanged and only records token usage (`scripts/usage_gateway/`) is allowed. || The harness's web search is off in official runs: Codex `web_search = "disabled"` (Harbor `--ak web_search=disabled`), Claude Code `WebSearch` and `WebFetch` disallowed (`--ak disallowed_tools=WebSearch,WebFetch`). The search runs at the model provider, so no egress allowlist stops it; `scripts/run_agent_batch.sh` sets both by default (`WEB_SEARCH=disabled`) | MUST | [new] (2026-10-02: tested with Codex 0.159.2 and gpt-6-luna — with ChatGPT login the default searches GitHub and quotes upstream experts, `disabled` stops it; with an Azure API key Codex offers no search, but the API runs one for a request that carries the `web_search` tool, which only the usage gateway's raw log can reveal) |-| D12 | Mode naming | `mode ∈ {privileged, standard, realtime}` (realtime since v1.1, owner 2026-10-08, #86); the `tier` field is removed; every task directory ends in its mode: `-privileged`, `-standard` or `-realtime`. |+| D12 | Mode naming | `mode ∈ {privileged, standard, standard-realtime}` (the third mode since v1.1, #86; named `standard-realtime` since v1.1.1, owner 2026-10-08, #97: it is standard mode's robot as a service on a clock that does not stop); the `tier` field is removed; every task directory ends in its mode: `-privileged`, `-standard` or `-standard-realtime`. |## 1. Checklist (every task)@@ -73,10 +74,10 @@ Unchanged: the task name prefix `robot-coding-bench/`, the client and tool file10. Reference solutions only under `solution/`; nothing the agent can reach contains or points to one. [current]11. `nop` scores 0; with an oracle, the oracle scores 1; without one, human review (§2.7). [new]12. A published task's `environment/` and `tests/` inputs never change; a change is a new task. [current]-13. The directory name ends in `-privileged`, `-standard` or `-realtime`, matching `mode` (§2.2); no `results/` directory (D3). [CI]+13. The directory name ends in `-privileged`, `-standard` or `-standard-realtime`, matching `mode` (§2.2); no `results/` directory (D3). [CI]14. `README.md` in the standard layout (§2.1). [CI] (sections and key rows; `scripts/task_conventions.py --check` for the rest)15. Runs with Harbor's native agents; no custom runner or adapter, and no gateway beyond the recording-only usage gateway (D11). [new]-16. Standard-mode tasks also satisfy every MUST in §4; realtime-mode tasks every MUST in §4 and §4b.+16. Standard-mode tasks also satisfy every MUST in §4; standard-realtime tasks every MUST in §4 and §4b.## 2. Task contract@@ -97,7 +98,7 @@ Unchanged: the task name prefix `robot-coding-bench/`, the client and tool file|---|---|---|| Privileged tasks: `<base>-privileged`, where `<base>` is `<benchmark>-<family>-i<NN>` (`metaworldplus-<env>-i00`, `kinder-<family>-i00`, `humanoidbench-<category>-<task>-i00`) or an upstream name kept as is (`robotwin_2_0_<task>`, `dextoolbench_<cat>_<obj>_<task>`, `mujoco_playground_<env>`) | MUST | suffix [CI] || Standard tasks: `<base>-standard`, sharing the privileged twin's frozen instance and `instance_seed` | MUST | suffix [CI]; shared instance [current] |-| Realtime tasks: `<base>-realtime`, sharing the privileged twin's frozen instance and `instance_seed`; its twin is `<base>-privileged` (a benchmark that has realtime tasks usually has `<base>-standard` too) | MAY (the mode) / MUST (the name) | suffix and instance [CI] [new] (v1.1) |+| Standard-realtime tasks: `<base>-standard-realtime`, sharing the privileged twin's frozen instance and `instance_seed`; its twin is `<base>-privileged` (a benchmark that has standard-realtime tasks usually has `<base>-standard` too) | MAY (the mode) / MUST (the name) | suffix and instance [CI] [new] (v1.1) || Versions or durations in task names (`-v3-`, `-60min-`) | MUST NOT | [new] — versions are dataset digests |### 2.3 `task.toml`@@ -105,9 +106,9 @@ Unchanged: the task name prefix `robot-coding-bench/`, the client and tool file|---|---|---|| `schema_version = "1.3"`; `[task].name = "robot-coding-bench/<folder>"` | MUST | [current]; name [CI] || `[task].description`, `authors`, `keywords` | SHOULD | [current] |-| `[metadata]`: `author_name`, `task_type ∈ {trajectory, policy}`, `simulator`, `robot`, `family`, `mode ∈ {privileged, standard, realtime}`, `expert_time_estimate_hours` | MUST | [CI] |+| `[metadata]`: `author_name`, `task_type ∈ {trajectory, policy}`, `simulator`, `robot`, `family`, `mode ∈ {privileged, standard, standard-realtime}`, `expert_time_estimate_hours` | MUST | [CI] || `tier` | MUST NOT | [CI] (`privileged` → `mode = "privileged"`, `perception` → `mode = "standard"`) |-| Standard tasks: `protocol = "eai-standard/2.1"`, `"eai-standard/2.2"` or `"eai-standard/2.4"`; realtime tasks: `protocol = "eai-standard/2.3"`; privileged tasks: no `protocol` | MUST | [CI] |+| Standard tasks: `protocol = "eai-standard/2.1"`, `"eai-standard/2.2"` or `"eai-standard/2.4"`; standard-realtime tasks: `protocol = "eai-standard/2.3"`; privileged tasks: no `protocol` | MUST | [CI] || `oracle ∈ {full, partial, none}`; `oracle_evidence` when not `full` | MUST | [CI] || `simulator`, `robot` from the vocabularies in [`TAXONOMY.md`](TAXONOMY.md) (extend by PR) | MUST | [current]; CI check [new] || `instance_seed`; benchmark-specific fields registered in TAXONOMY | MAY | [current] |@@ -246,10 +247,10 @@ Wire details (endpoints, RCB1 frames, `meta`, error codes, crash recovery) are i| rcb-limited/2.1 | 2026-09-27 | 82-column proprioception, optional depth noise; crash recovery added 2026-09-29 (no version bump) || **eai-standard/2.1** | 2026-09-30 | rename only; identical to rcb-limited/2.1 including crash recovery. **Current.** || eai-standard/2.2 | 2026-10-01 | additive extension (§6, [`STANDARD_MODE_2_2.md`](STANDARD_MODE_2_2.md)) with crash recovery; served by BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, MolmoSpaces, MyoSuite and DexJoCo |-| eai-standard/2.3 | 2026-10-08 | 2.2 + the realtime capability (§4b; [`LIMITED_MODE_PROTOCOL.md`](LIMITED_MODE_PROTOCOL.md), "Realtime mode"); every 2.2 addition stays a MAY. Served by realtime-mode tasks only (HumanoidBench); standard tasks stay on 2.1 / 2.2. Issue #86 |+| eai-standard/2.3 | 2026-10-08 | 2.2 + the realtime capability (§4b; [`LIMITED_MODE_PROTOCOL.md`](LIMITED_MODE_PROTOCOL.md), "Standard-realtime mode"); every 2.2 addition stays a MAY. Served by standard-realtime tasks only (HumanoidBench); standard tasks stay on 2.1 / 2.2. Issue #86 || eai-standard/2.4 | 2026-10-08 | 2.3 + the operations capability ([`STANDARD_MODE_2_4.md`](STANDARD_MODE_2_4.md)): operation ids and request deadlines (408 / 409), `GET /operations/<id>`, a service-owned call watchdog, a physics-tick ledger, fsynced journals; every earlier addition, the realtime capability included, stays a MAY. Served by standard tasks (RoboTangle); reference implementation `scripts/eai_standard/v2_4/`. Issue #89 |-Standard services report `eai-standard/2.1`, `eai-standard/2.2` or `eai-standard/2.4`, realtime services `eai-standard/2.3`; the shipped clients+Standard services report `eai-standard/2.1`, `eai-standard/2.2` or `eai-standard/2.4`, standard-realtime services `eai-standard/2.3`; the shipped clientsaccept every 2.x version and the former `rcb-limited/2.x` spelling as well.**Core rules.** The simulator runs in a sidecar the agent cannot open; four endpoints; RCB1 binary responses; no@@ -355,11 +356,11 @@ results must state the reset budget and are compared only with results under the(implemented for RoboTwin). If recorded: rendered from the replay by default, frames stamped with the call, neveraffecting the verdict.-## 4b. Realtime mode (`mode = "realtime"`): `eai-standard/2.3`+## 4b. Standard-realtime mode (`mode = "standard-realtime"`): `eai-standard/2.3`Standard mode on a clock that does not stop (v1.1; owner, 2026-10-08; issue #86). Everything in §4 holds (sidecartopology, endpoints, RCB1, actions, privileged information, crash recovery, evidence, replay grading) except where this-section says otherwise. Wire details: [`LIMITED_MODE_PROTOCOL.md`](LIMITED_MODE_PROTOCOL.md), "Realtime mode". The time model+section says otherwise. Wire details: [`LIMITED_MODE_PROTOCOL.md`](LIMITED_MODE_PROTOCOL.md), "Standard-realtime mode". The time modelis the one proposed in #41 (the world runs at 1/S of real speed, a command takes effect at the next control step, allagent latency counts, the agent's own program may act), on the 2.x endpoints and without its proxy (D11).@@ -373,7 +374,7 @@ agent latency counts, the agent's own program may act), on the 2.x endpoints and| Reset budget: **`budget.reset = 50` by default** (the reset capability of §4.5; `RCB_RESET_BUDGET` overrides it; results state the budget and are compared only under the same budget) | MUST (default) | [new] (owner, 2026-10-08: the same as HumanoidBench standard) || A simulator crash is an infrastructure failure, never the agent's: past `RCB_MAX_RESUMES` restarts the trial ends `sim_crashed` (no call spans a crash: `/submit` answers at once), a failed resume `resume_failed`, and a simulator error that ends the graded episode (`reason = "error"`) — all with `infra_error = 1` | MUST | [new] (owner, 2026-10-08) || The agent's controller runs in the agent's own container and streams targets through the client; no agent code runs next to the simulator | MUST | [new] |-| A realtime task shares its privileged twin's frozen instance and success rule (its verifier grades the sidecar's record, as in standard mode); its instruction says that time does not stop | MUST | [new] |+| A standard-realtime task shares its privileged twin's frozen instance and success rule (its verifier grades the sidecar's record, as in standard mode); its instruction says that time does not stop | MUST | [new] |## 5. Results, cost, token budget| Item | Level | Basis |@@ -384,7 +385,7 @@ agent latency counts, the agent's own program may act), on the 2.x endpoints and| Align reasoning effort, `max_turns`, token budget and `[agent].timeout_sec` before comparing agents | SHOULD | [current] || Run id `<harness>-<model>-<effort>` | MUST | [new] || Results are not committed to the repository: they are published on the results site and the Hugging Face dataset, keyed by run id; no `results/` directory in a task directory, no batch reports under `docs/` | MUST | [CI] for task directories |-| Privileged, standard and realtime results reported separately; standard and realtime results state the protocol version and reset budget, realtime results also the slowdown S (S ≠ 1 reported separately from S = 1) | MUST | [new] (realtime: v1.1) |+| Privileged, standard and standard-realtime results reported separately; standard and standard-realtime results state the protocol version and reset budget, standard-realtime results also the slowdown S (S ≠ 1 reported separately from S = 1) | MUST | [new] (realtime: v1.1) |Harbor 0.23 has no token cap. **With the usage gateway on** (`GATEWAY=1` in `scripts/run_agent_batch.sh`, D11), thebudget is enforced during the run: once a trial's recorded input or output reaches its budget, the gateway refuses every@@ -408,11 +409,11 @@ mounting, `robot_spec`, `world`, a stateless `POST /kinematics`, and a per-call`meta.stopped`; the episode's only budget stays the wall clock) — all MAY capabilities. Its services implement the crashrecovery of §4.7 (with the per-simulator resume rule); the shared implementation is `scripts/eai_standard/`.-**`eai-standard/2.3`** (v1.1, 2026-10-08, issue #86; [`LIMITED_MODE_PROTOCOL.md`](LIMITED_MODE_PROTOCOL.md), "Realtime mode").+**`eai-standard/2.3`** (v1.1, 2026-10-08, issue #86; [`LIMITED_MODE_PROTOCOL.md`](LIMITED_MODE_PROTOCOL.md), "Standard-realtime mode").2.2 plus the realtime capability: `/spec.timing`, `/submit` with `at` answering at once, `cameras: []` for no images, the`meta` / `/status` fields `running`, `planned_steps`, `lag_s` (and `start_step`, `late_steps`, `plan_end_step` on asubmit), and the evidence of §4b. Every 2.2 addition stays a MAY capability, so a 2.1 service with the realtime capability-is a 2.3 service. Only realtime-mode tasks serve it (§4b); 2.x clients connect to it unchanged, and the timing semantics+is a 2.3 service. Only standard-realtime tasks serve it (§4b); 2.x clients connect to it unchanged, and the timing semanticsare declared in `/spec.timing` and the task's instruction.**`eai-standard/2.4`** (2026-10-08, issue #89; [`STANDARD_MODE_2_4.md`](STANDARD_MODE_2_4.md)). 2.3 plus the operations@@ -421,7 +422,7 @@ capability: `meta.operation_id` and `meta.request_deadline_unix_s` on `/submit``interrupted` with `execution_uncertain` after a crash), a service-owned watchdog that ends a hung native call as an infraerror, a physics-tick ledger that resumes replay with the ticks each row took, and fsynced journals. Every earlier addition(2.2's, and 2.3's realtime capability) stays a MAY, so a 2.2 service with the operations capability is a 2.4 service;-standard tasks may declare it, realtime tasks stay on 2.3. The reference implementation is `scripts/eai_standard/v2_4/`+standard tasks may declare it, standard-realtime tasks stay on 2.3. The reference implementation is `scripts/eai_standard/v2_4/`(`rcb_limited.py`, `rcb_client.py`; `rcb1.py` and `run.sh` are 2.2's), copied unchanged into every 2.4 task. Firstserved by RoboTangle, whose release named it "2.3" before that number went to the realtime capability.diff --git a/docs/REVIEWING.md b/docs/REVIEWING.mdindex 966a79b14a..16b3f9e158 100644--- a/docs/REVIEWING.md+++ b/docs/REVIEWING.md@@ -29,7 +29,7 @@ Guide for maintainers. The contract is [PROTOCOL.md](PROTOCOL.md); contributors- Does the task run with Harbor's native agents, without custom adapters or runners, and without any gateway other thanthe recording-only usage gateway (PROTOCOL D11)?- Are images pinned, published (or marked `BUILD_ONLY`), and shared with the other tasks of the same simulator?-- Names end in their mode, `-privileged` / `-standard` / `-realtime`; README in the standard layout; no `results/` in task directories.+- Names end in their mode, `-privileged` / `-standard` / `-standard-realtime`; README in the standard layout; no `results/` in task directories.## Human review (oracle `partial` or `none`)Review per family (same generator and predicate), at least two instances per family:diff --git a/docs/RUNNING.md b/docs/RUNNING.mdindex 06658f6502..86bef56d7f 100644--- a/docs/RUNNING.md+++ b/docs/RUNNING.md@@ -2,7 +2,7 @@Operational notes for running batches and reading results. The contract is [PROTOCOL.md](PROTOCOL.md); thestandard-mode wire protocol is [LIMITED_MODE_PROTOCOL.md](LIMITED_MODE_PROTOCOL.md). Task directories end in `-privileged`,-`-standard` or `-realtime` (PROTOCOL §4b: realtime results state the reset budget and the slowdown `RCB_SLOWDOWN`, default+`-standard` or `-standard-realtime` (PROTOCOL §4b: standard-realtime results state the reset budget and the slowdown `RCB_SLOWDOWN`, default1; results with another slowdown are reported separately).## Agent logs and timelines@@ -38,7 +38,7 @@ separate verifiers. Results (per-task and batch reports) go to the results site## Unattended batches (`scripts/run_agent_batch.sh`)`scripts/run_agent_batch.sh <batch> <base> ...` runs one Harbor-native agent over `<base>-privileged` and `<base>-standard`-(and `<base>-realtime` with `MODES="privileged standard realtime"`) for each task base (a missing twin is skipped), one Harbor job per task and mode. Starting the same command again adds a+(and `<base>-standard-realtime` with `MODES="privileged standard standard-realtime"`) for each task base (a missing twin is skipped), one Harbor job per task and mode. Starting the same command again adds alane: every job is claimed by exactly one runner (`$JOBS/.claims/<job>`). The run id is derived as in TAXONOMY (D4) andchecked. Configuration is Harbor's own:diff --git a/docs/STANDARD_MODE_2_4.md b/docs/STANDARD_MODE_2_4.mdindex bf8d0bf548..b61831c28b 100644--- a/docs/STANDARD_MODE_2_4.md+++ b/docs/STANDARD_MODE_2_4.md@@ -10,8 +10,8 @@ implementation: [`scripts/eai_standard/v2_4/`](../scripts/eai_standard/v2_4/), w`eai-standard/2.4` = 2.3 + the **operations capability** below.- Every earlier addition stays a MAY capability. That covers 2.2's depth, `frame_time_s`, extrinsics, `POST /kinematics`and the per-call wall bound, and 2.3's realtime capability ([`LIMITED_MODE_PROTOCOL.md`](LIMITED_MODE_PROTOCOL.md),- "Realtime mode"). So a 2.2 service that adds the operations capability is a 2.4 service.-- Standard-mode tasks may declare `protocol = "eai-standard/2.4"` (PROTOCOL §2.3). Realtime-mode tasks stay on 2.3+ "Standard-realtime mode"). So a 2.2 service that adds the operations capability is a 2.4 service.+- Standard-mode tasks may declare `protocol = "eai-standard/2.4"` (PROTOCOL §2.3). Standard-realtime tasks stay on 2.3(PROTOCOL §4b).- Everything in PROTOCOL §4 holds unchanged: four endpoints (plus `/kinematics` and the one below), RCB1, no reset,crash recovery that resumes the same episode, and grading by a fresh replay.diff --git a/docs/TAXONOMY.md b/docs/TAXONOMY.mdindex 98f245ed58..99f9bb8031 100644--- a/docs/TAXONOMY.md+++ b/docs/TAXONOMY.md@@ -6,8 +6,8 @@ Known values; a new value is added by pull request (CI validation of these vocab|---|---|| `author_name` | the author's name (an organization may go in `author_org`) || `task_type` | `trajectory` · `policy` |-| `mode` | `privileged` · `standard` (formerly `unlimited` / `limited`) · `realtime` (protocol v1.1, PROTOCOL §4b) |-| `protocol` | standard and realtime mode only. Realtime: `eai-standard/2.3`. Standard: `eai-standard/2.1` · `eai-standard/2.2` (BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, ManiGuard, MolmoSpaces, MyoSuite, DexJoCo, MuJoCo Playground's quadruped tasks; [STANDARD_MODE_2_2.md](STANDARD_MODE_2_2.md)) · `eai-standard/2.4` ([STANDARD_MODE_2_4.md](STANDARD_MODE_2_4.md), the operations capability: RoboTangle) |+| `mode` | `privileged` · `standard` (formerly `unlimited` / `limited`) · `standard-realtime` (protocol v1.1; named so since v1.1.1, PROTOCOL §4b) |+| `protocol` | standard and standard-realtime mode only. Standard-realtime: `eai-standard/2.3`. Standard: `eai-standard/2.1` · `eai-standard/2.2` (BEHAVIOR-1K, RoboLab, RoboWits, RoboPaint, ManiGuard, MolmoSpaces, MyoSuite, DexJoCo, MuJoCo Playground's quadruped tasks; [STANDARD_MODE_2_2.md](STANDARD_MODE_2_2.md)) · `eai-standard/2.4` ([STANDARD_MODE_2_4.md](STANDARD_MODE_2_4.md), the operations capability: RoboTangle) || `simulator` | `robosuite` · `robotwin` (RoboTwin 2.0 = SAPIEN 3 + CuRobo) · `mujoco` · `kinder` (KinDER: MuJoCo and PyBullet) · `humanoidbench` (HumanoidBench: MuJoCo + dm_control) · `robowits` (RoboWits on Genesis) · `robolab` (NVIDIA RoboLab on Isaac Lab) · `behavior` (BEHAVIOR-1K: OmniGibson on Isaac Sim) · `maniskill` (ManiSkill 3 / SAPIEN: RoboPaint) · `vlabench` (VLABench: MuJoCo + dm_control) · `maniguard` (ManiGuard: BEHAVIOR-1K 3.7.2, OmniGibson on Isaac Sim 4.5) · `molmospaces` (Ai2 MolmoSpaces: MuJoCo) · `myosuite` (MyoSuite: musculoskeletal models on MuJoCo) · `dexjoco` (DexJoCo: MuJoCo 3.4 + its gymnasium envs) · `robotangle` (RoboTangle, formerly TopoGym: Genesis 1.2.1 rigid bodies and libuipc IPC / PBD deformables, Nyx RGB) — the simulator, not the image: benchmarks on one simulator may use different images and versions (PROTOCOL D5) || `robot` | `franka-panda` · `aloha-agilex` · `aloha-2` (ALOHA 2, MuJoCo Menagerie: MuJoCo Playground, RoboTangle) · `kuka-iiwa14-sharpa-ha4` · `tidybot-plus-plus` · `unitree-h1` · `unitree-h1hand` (H1 with two Shadow hands) · `sawyer` (Rethink Robotics Sawyer, as in Meta-World) · `marvin-bimanual` (RoboWits) · `franka-robotiq-2f85` (a Franka arm with a Robotiq 2F-85, the DROID setup: RoboLab, MolmoSpaces' FR3, RoboTangle's Panda) · `r1pro` (BEHAVIOR-1K) · `franka-panda-omron` (RoboCasa's PandaOmron: Franka Panda on an Omron mobile base with a lift torso) · `fourier-gr1` (Fourier GR1 humanoid: two arms, waist and two Fourier dexterous hands, fixed base; RoboCasa-GR1) · `franka-panda-longfinger` (ManiGuard: Franka Panda with long fin-ray fingers) · `rainbow-rby1` (Rainbow Robotics RB-Y1: holonomic base, 6-joint torso, two 7-joint arms with parallel grippers; MolmoSpaces) · `myohand` (MyoSuite's MyoHand: a musculoskeletal human forearm and hand, 39 muscles) · `myoarm` (MyoSuite's MyoArm: a musculoskeletal human shoulder, arm and hand, 34 or 63 muscles) · `myoarm-torso` (MyoSuite's table-tennis player: torso, right arm and hand on a sliding pelvis) · `myoarm-mpl` (MyoSuite's MyoArm next to the Modular Prosthetic Limb, a motorized arm) · `myolegs` (MyoSuite's MyoLegs: a free-standing musculoskeletal torso and both legs) · `myoleg-osl` (MyoSuite's MyoLeg with an Open-Source Leg knee-ankle prosthesis) · `unitree-go1` (Unitree Go1 quadruped; MuJoCo Playground) · `boston-dynamics-spot` (Boston Dynamics Spot quadruped; MuJoCo Playground) · `google-barkour-vb` (Google Barkour vB quadruped; MuJoCo Playground) · `franka-allegro` (Franka Panda with a 16-DoF Allegro hand; DexJoCo) · `franka-allegro-bimanual` (two of them side by side; DexJoCo) || `family` | free-form slug naming the scene/goal, shared by its instances (e.g. `stack_blocks_two`, `dextoolbench/brush/blue_brush/sweep_forward`) |@@ -34,8 +34,8 @@ Benchmark-specific fields (MAY, PROTOCOL §2.3):| RoboTangle | `success_rule`, `tooling_version` | the task-native success rule and its terminal revision; the tooling generation (text) |## Task names-- `<base>-privileged`, `<base>-standard` and `<base>-realtime`, the modes of one frozen instance (realtime: PROTOCOL §4b;- HumanoidBench has it for every family with a standard twin; a realtime task's twin is `<base>-privileged`). `<base>` is+- `<base>-privileged`, `<base>-standard` and `<base>-standard-realtime`, the modes of one frozen instance (standard-realtime: PROTOCOL §4b;+ HumanoidBench has it for every family with a standard twin; a standard-realtime task's twin is `<base>-privileged`). `<base>` is`<simulator>-<family-with-dashes>-i<NN>`, or the benchmark's own task name: `robotwin_2_0_<robotwin_task>`,`dextoolbench_<category>_<object>_<task>`, `mujoco_playground_<env>`, `kinder-<family>-i<NN>`,`humanoidbench-<category>-<family>-i<NN>`, `myosuite-<family>-i<NN>`, `dexjoco-<family-with-dashes>-i<NN>`.diff --git a/docs/benchmarks/humanoidbench.md b/docs/benchmarks/humanoidbench.mdindex af66abc603..11edec2401 100644--- a/docs/benchmarks/humanoidbench.md+++ b/docs/benchmarks/humanoidbench.md@@ -8,9 +8,9 @@ Whole-body humanoid control: locomotion and manipulation on a Unitree H1, scored| simulator | MuJoCo 3.1.6 + dm_control, CPU || robots | `h1`: Unitree H1, 19 actuators · `h1hand`: H1 + two Shadow hands, 61 actuators || score | an episode's summed reward (≤ 1000 steps); pass = at least upstream's success bar |-| tasks | 10 × 3 modes: `tasks/humanoidbench-<category>-<task>-i00-privileged`, `-standard` (`eai-standard/2.1`) and `-realtime` (the realtime mode, `eai-standard/2.3`: standard mode on a clock that does not stop, below) |+| tasks | 10 × 3 modes: `tasks/humanoidbench-<category>-<task>-i00-privileged`, `-standard` (`eai-standard/2.1`) and `-standard-realtime` (the standard-realtime mode, `eai-standard/2.3`: standard mode on a clock that does not stop, below) || image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` (`images/humanoidbench/`) |-| oracle | `sit_simple` (the fixture): full, all three modes; the 9 scored tasks: none, human review (`oracle_evidence` in each task.toml; realtime: approved by the owner in PR #83, 2026-10-08) |+| oracle | `sit_simple` (the fixture): full, all three modes; the 9 scored tasks: none, human review (`oracle_evidence` in each task.toml; standard-realtime: approved by the owner in PR #83, 2026-10-08) |## Tasks: upstream's 32, ours ✅ (9 scored + the `sit_simple` fixture)@@ -62,7 +62,7 @@ Why these (no model's results enter the choice):|---|---|---|---|| privileged | the simulator and its full state; no reward | unlimited | its `trajectory.npz`, replayed twice in a fresh process || standard | head cameras, joint angles/velocities, a pelvis IMU, a fixed room camera; no reward | 50 (below; `RCB_RESET_BUDGET=0`: none) | the last episode, replayed in a fresh simulator |-| realtime | the same as standard, but time does not stop: the robot runs at 50 Hz in real time while the agent thinks (below) | 50 (the realtime mode's default, PROTOCOL §4b) | the last episode: the targets the robot executed at each control step, replayed in a fresh simulator |+| standard-realtime | the same as standard, but time does not stop: the robot runs at 50 Hz in real time while the agent thinks (below) | 50 (the standard-realtime mode's default, PROTOCOL §4b) | the last episode: the targets the robot executed at each control step, replayed in a fresh simulator |### Standard mode's reset budget: 50@@ -93,14 +93,14 @@ D1's default of 0 (§4.5); results state the reset budget they ran under. What agrants N. Results under a budget other than 50 are reported separately, and HumanoidBench standard-mode resultsrecorded before this change (budget 0) are a different configuration from today's.-### Realtime mode: time does not stop+### Standard-realtime mode: time does not stopIn standard mode the robot waits, frozen, between the agent's calls: a model that thinks for a minute between two-chunks loses nothing. A real H1 does not wait. The realtime mode (PROTOCOL §4b, protocol v1.1;-`tasks/humanoidbench-*-i00-realtime`, 10 tasks, one per family, `mode = "realtime"`) is standard mode with the same-instance, sensors, success rule, reset budget (50, the realtime mode's default), wall clock (1 h) and client, on a+chunks loses nothing. A real H1 does not wait. The standard-realtime mode (PROTOCOL §4b, protocol v1.1;+`tasks/humanoidbench-*-i00-standard-realtime`, 10 tasks, one per family, `mode = "standard-realtime"`) is standard mode with the same+instance, sensors, success rule, reset budget (50, the standard-realtime mode's default), wall clock (1 h) and client, on aclock that does not stop (`scripts/humanoidbench/shared/realtime/rcb_realtime.py`, which extends `rcb_service.py`; the-wire protocol is `eai-standard/2.3`, 2.2 plus the realtime capability, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"):+wire protocol is `eai-standard/2.3`, 2.2 plus the realtime capability, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"):- **Clock.** An episode starts paused at the frozen start, its servos holding their starting targets. Its clock startsat its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 20 ms of wall time@@ -133,7 +133,7 @@ that reads proprioception and streams targets at the control rate through the cl`cameras=[]` costs a few ms over the shared network namespace), run in the background, and improved between episodes.The agent's code still runs only in its own container, never next to the simulator (standard mode's isolation iskept); the 50 resets are what make that loop of writing, running and revising a controller possible at all, which is-why the realtime mode defaults to 50 resets rather than D1's 0.+why the standard-realtime mode defaults to 50 resets rather than D1's 0.World-frame information (PROTOCOL D7): privileged mode gives the full state; standard mode gives none -- no root pose,no object or target pose; the room camera's pose is given in the robot's starting frame.@@ -163,8 +163,8 @@ its model-API key in its container, so it could call a search-enabled model thro```bashharbor run -c scripts/humanoidbench/configs/codex-openrouter.yaml --env-file .env -p tasks/humanoidbench-loco-hurdle-i00-standardharbor run -p tasks/humanoidbench-loco-sit-simple-i00-standard -a oracle # or -a nop: scores 0-harbor run -c scripts/humanoidbench/configs/codex-openrouter.yaml --env-file .env -p tasks/humanoidbench-loco-walk-i00-realtime-harbor run -p tasks/humanoidbench-loco-sit-simple-i00-realtime -a oracle # streams the same 999 rows+harbor run -c scripts/humanoidbench/configs/codex-openrouter.yaml --env-file .env -p tasks/humanoidbench-loco-walk-i00-standard-realtime+harbor run -p tasks/humanoidbench-loco-sit-simple-i00-standard-realtime -a oracle # streams the same 999 rows````codex-openrouter.yaml` is Harbor's own Codex agent with OpenRouter as its model provider (run id`codex-gpt6_luna-medium`). Standard tasks run with 50 resets by default; `RCB_RESET_BUDGET=0 harbor run ...` runs them@@ -178,6 +178,6 @@ Agent results are not kept in this repository (PROTOCOL D3): per-run READMEs (ve`reward.json`, the replay's last frame and videos are published on the results site, keyed by run id. Standard-moderesults state their reset budget: the standard configuration is 50 resets (since 2026-10-07); runs with 0 -- everyHumanoidBench standard-mode run recorded before that date -- are reported separately, and runs recorded before-protocol v1.0 are marked as such. Realtime results (`-realtime`) are their own mode and are never pooled with+protocol v1.0 are marked as such. Standard-realtime results (`-standard-realtime`) are their own mode and are never pooled withstandard ones; they state the reset budget and `slowdown`. Official runs use `slowdown` 1 (the default); resultswith `RCB_SLOWDOWN` ≠ 1 are reported separately from them (PROTOCOL §4b).diff --git a/scripts/humanoidbench/README.md b/scripts/humanoidbench/README.mdindex e6996d13c4..f6b0e246de 100644--- a/scripts/humanoidbench/README.md+++ b/scripts/humanoidbench/README.md@@ -15,9 +15,9 @@ results are published on the results site, not kept here (PROTOCOL D3).| `measure_baselines.py` | the `nop` / `hold` floors for all 32 upstream families, including the ones outside the subset — the evidence for which families deserve a slot || `templates/` | everything the tasks have in common, with `{{placeholders}}` || `templates/standard/`, `shared/standard/` | the same for each task's standard-mode twin (`tasks/*-standard/`, `eai-standard/2.1`): `rcb_service.py` (the simulator-agnostic service: endpoints, RCB1, episode, crash recovery, finalize), `server.py` (the HumanoidBench adapter), `rcb_client.py` and `rcb1.py` (the agent's client; its docstring is the API reference), `grade.py` (the verifier). Per-task prose is the task's `[task.standard]` table in `subset.toml` |-| `templates/realtime/`, `shared/realtime/` | each task's realtime-mode task (`tasks/*-realtime/`, `eai-standard/2.3`, PROTOCOL §4b): standard mode on a clock that does not stop. `rcb_realtime.py` (the service: extends `rcb_service.py` with the clock and the plan; the sidecar runs it instead of `server.py`, with the same adapter) and `rcb_client.py` (the realtime client: `submit(..., at=)`, `cameras=[]`); the rest is the standard twin's |+| `templates/realtime/`, `shared/realtime/` | each task's standard-realtime task (`tasks/*-standard-realtime/`, `eai-standard/2.3`, PROTOCOL §4b): standard mode on a clock that does not stop. `rcb_realtime.py` (the service: extends `rcb_service.py` with the clock and the plan; the sidecar runs it instead of `server.py`, with the same adapter) and `rcb_client.py` (the realtime client: `submit(..., at=)`, `cameras=[]`); the rest is the standard twin's || `standard_probe.py` | what `measure-standard` runs against a twin's sim service: contract checks, the all-zero return against `facts.json`, a crash it must survive, the t=0 camera images, and the per-step cost the instruction quotes |-| `oracles/<family>/<mode>/` | a task's reference solution, where it has one (only the `sit_simple` fixture: privileged, standard and realtime; files directly under `oracles/<family>/` go to all); `render` copies it to `solution/` |+| `oracles/<family>/<mode>/` | a task's reference solution, where it has one (only the `sit_simple` fixture: privileged, standard and standard-realtime; files directly under `oracles/<family>/` go to all); `render` copies it to `solution/` || `configs/codex-openrouter.yaml` | the Harbor job config the recorded runs use: Harbor's own Codex agent on OpenRouter || `tests/test_standard_service.py` | the standard-mode service, client and verifier against a toy simulator (`python -m unittest discover -s scripts/humanoidbench/tests`; numpy, requests) || `tests/test_realtime_service.py` | the realtime service and client over real HTTP against the same toy simulator: the paused start, the clock, the plan (`at`, late rows, holds), watches, resets, the tick log, a restart |diff --git a/scripts/humanoidbench/gen_tasks.py b/scripts/humanoidbench/gen_tasks.pyindex b242635dcc..019913b83c 100755--- a/scripts/humanoidbench/gen_tasks.py+++ b/scripts/humanoidbench/gen_tasks.py@@ -19,7 +19,7 @@ everything the tasks have in common, `shared/` the one real copy of the tooling./gen_tasks.py measure-standard [family ...] # check + time the standard twins' sim service (needs a GPU)Every task with a [task.standard] table is rendered three times, once per mode (docs/PROTOCOL.md D12):-`<base>-privileged`, `<base>-standard` and `<base>-realtime` (the realtime mode, §4b: standard mode on a clock that does+`<base>-privileged`, `<base>-standard` and `<base>-standard-realtime` (the standard-realtime mode, §4b: standard mode on a clock that doesnot stop, eai-standard/2.3; templates/realtime/, shared/realtime/).`render` writes everything in tasks/ but the instances, which only `instance` draws, and the results/ of agent runs;@@ -459,18 +459,18 @@ def render_standard(t: dict, defaults: dict, facts: dict, tasks_dir: Path = TASKreturn out-# --------------------------------------------------------------------------- realtime mode-# The realtime task (`<base>-realtime`, mode = "realtime", eai-standard/2.3; docs/PROTOCOL.md §4b): the standard twin's+# --------------------------------------------------------------------------- standard-realtime mode+# The standard-realtime task (`<base>-standard-realtime`, mode = "standard-realtime", eai-standard/2.3; docs/PROTOCOL.md §4b): the standard twin's# sidecar, client and verifier, except that the service runs the episode on a clock that does not stop# (shared/realtime/rcb_realtime.py, which extends rcb_service.py unchanged) and the agent gets the realtime client# (`submit(..., at=)`, `cameras=[]`). Same instance, same success rule; reset budget `realtime_reset_budget` (50, the-# realtime mode's default); the instruction tells the agent to drive the robot with a controller program.+# standard-realtime mode's default); the instruction tells the agent to drive the robot with a controller program.REALTIME_TEMPLATES = HERE / "templates" / "realtime"REALTIME = HERE / "shared" / "realtime"def realtime_dir_name(t: dict, instance_tag: str) -> str:- return base_dir_name(t, instance_tag) + "-realtime"+ return base_dir_name(t, instance_tag) + "-standard-realtime"# A recorded episode (a trajectory) is solvability evidence here too: planned from step 0 (`at`), its rows run at the@@ -485,7 +485,7 @@ def realtime_context(t: dict, defaults: dict, facts: dict) -> dict:ev += REALTIME_EVIDENCE if "trajectory" in ev else "" # a recorded episode, not "none known" or an RL score# the realtime tasks' own human review: the pull request that added them (realtime_review_pr; 0: not opened yet)review = {**defaults, "review_pr": defaults.get("realtime_review_pr", 0)}- oracle = oracle_fields({**t, "oracle_evidence": ev}, review, "realtime")+ oracle = oracle_fields({**t, "oracle_evidence": ev}, review, "standard-realtime")return ctx | {"base_dir": base_dir_name(t, defaults["instance_tag"]),"control_ms": f"{1000 / facts['standard']['control_hz']:g}",@@ -511,7 +511,7 @@ def render_realtime(t: dict, defaults: dict, facts: dict, tasks_dir: Path = TASK(STANDARD / "rcb1.py", "environment/rcb1.py"), (STANDARD / "grade.py", "tests/grade.py")]:shutil.copyfile(src, out / rel)shutil.copyfile(CLIENT_REQUIREMENTS, out / "environment/client-requirements.txt")- copy_oracle(t, "realtime", out)+ copy_oracle(t, "standard-realtime", out)normalize(out)return out@@ -701,8 +701,8 @@ FORMULA = re.compile(r"\b(?:bounds|margin|value_at_margin)\s*=|\brewards\.toleradef leaks(task: Path, families: set[str]) -> list[str]:toml = tomllib.loads((task / "task.toml").read_text())- # standard and realtime tasks score in the sim sidecar (environment/sim/), privileged ones in tests/- own, standard = toml["metadata"]["family"], toml["metadata"]["mode"] in ("standard", "realtime")+ # standard and standard-realtime tasks score in the sim sidecar (environment/sim/), privileged ones in tests/+ own, standard = toml["metadata"]["family"], toml["metadata"]["mode"] in ("standard", "standard-realtime")home = task / ("environment/sim" if standard else "tests") # where the formula has to beagent_side = [f for f in sorted((task / "environment").rglob("*"))if f.is_file() and not (standard and home in f.parents)]diff --git a/scripts/humanoidbench/oracles/sit_simple/realtime/oracle_stream.py b/scripts/humanoidbench/oracles/sit_simple/standard-realtime/oracle_stream.pysimilarity index 90%rename from scripts/humanoidbench/oracles/sit_simple/realtime/oracle_stream.pyrename to scripts/humanoidbench/oracles/sit_simple/standard-realtime/oracle_stream.pyindex 339ee2c597..92357e2c7d 100644--- a/scripts/humanoidbench/oracles/sit_simple/realtime/oracle_stream.py+++ b/scripts/humanoidbench/oracles/sit_simple/standard-realtime/oracle_stream.py@@ -1,4 +1,4 @@-"""Oracle (realtime mode): the reference actions (sit_simple_actions.npy, the privileged oracle's 999 rows of+"""Oracle (standard-realtime mode): the reference actions (sit_simple_actions.npy, the privileged oracle's 999 rows ofnormalised servo targets) streamed through the client into the running episode's plan, one row per control step(dt = control_dt, hold): the first chunk starts the clock at step 0, the second is planned from the step where thefirst ends, long before that step comes, so every row runs at its own control step. The service ends the episode thediff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/solution/solve.sh b/scripts/humanoidbench/oracles/sit_simple/standard-realtime/solve.shsimilarity index 68%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/solution/solve.shrename to scripts/humanoidbench/oracles/sit_simple/standard-realtime/solve.shindex e2239bea1f..898a00c1f6 100755--- a/tasks/humanoidbench-loco-sit-simple-i00-realtime/solution/solve.sh+++ b/scripts/humanoidbench/oracles/sit_simple/standard-realtime/solve.sh@@ -1,5 +1,5 @@#!/bin/bash-# Oracle (realtime mode): the privileged oracle's recorded actions (solution/sit_simple_actions.npy) streamed+# Oracle (standard-realtime mode): the privileged oracle's recorded actions (solution/sit_simple_actions.npy) streamed# through the client into the episode's plan, one row per control step. No privileged information at run time: only# the recorded actions.set -ediff --git a/scripts/humanoidbench/shared/realtime/rcb_client.py b/scripts/humanoidbench/shared/realtime/rcb_client.pyindex 8e057bb6e9..f2b0559bf9 100644--- a/scripts/humanoidbench/shared/realtime/rcb_client.py+++ b/scripts/humanoidbench/shared/realtime/rcb_client.py@@ -1,4 +1,4 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, onPYTHONPATH).from rcb_client import Robot, EpisodeOverdiff --git a/scripts/humanoidbench/shared/realtime/rcb_realtime.py b/scripts/humanoidbench/shared/realtime/rcb_realtime.pyindex 410284bf27..e23f8cbd2c 100644--- a/scripts/humanoidbench/shared/realtime/rcb_realtime.py+++ b/scripts/humanoidbench/shared/realtime/rcb_realtime.py@@ -1,7 +1,8 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recoveryand finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:@@ -55,7 +56,7 @@ import numpy as npimport rcb_servicefrom rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request-PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching upCHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)diff --git a/scripts/humanoidbench/subset.toml b/scripts/humanoidbench/subset.tomlindex eaa4de7142..9b3c4320bb 100644--- a/scripts/humanoidbench/subset.toml+++ b/scripts/humanoidbench/subset.toml@@ -100,10 +100,10 @@ review_pr = 9 # the pull request whose human review covers the tas# own, so with no reset its whole run is one blind attempt. A per-benchmark exception to PROTOCOL D1's default of 0# (docs/benchmarks/humanoidbench.md); `RCB_RESET_BUDGET=0 harbor run ...` still runs the single-episode configuration.standard_reset_budget = 50-# Realtime mode's reset budget (docs/PROTOCOL.md §4b): 50 by default, the owner's decision of 2026-10-08 (the same as+# Standard-realtime mode's reset budget (docs/PROTOCOL.md §4b): 50 by default, the owner's decision of 2026-10-08 (the same as# standard's); `RCB_RESET_BUDGET` overrides it. RCB_SLOWDOWN defaults to 1 (real time); other values are reported apart.realtime_reset_budget = 50-realtime_review_pr = 83 # the pull request that added the realtime tasks; the owner approved them there (2026-10-08)+realtime_review_pr = 83 # the pull request that added the standard-realtime tasks; the owner approved them there (2026-10-08)# =====================================================================================================# C1 · flat-ground periodic gaitdiff --git a/scripts/humanoidbench/templates/realtime/README.md b/scripts/humanoidbench/templates/realtime/README.mdindex f0ce285d8d..9d3d3085c6 100644--- a/scripts/humanoidbench/templates/realtime/README.md+++ b/scripts/humanoidbench/templates/realtime/README.md@@ -1,6 +1,6 @@-# {{base_dir}}-realtime+# {{base_dir}}-standard-realtime-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of `{{task_dir}}`:+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of `{{task_dir}}`:the same frozen instance and the same success rule as it and as `{{base_dir}}-standard`, but time does not stop. Once anepisode's clock starts (its first `submit`), the robot runs at {{control_hz}} Hz in real time and holds its last servotargets while the agent thinks, so the agent drives it with a controller program of its own that streams targets@@ -11,10 +11,10 @@ grades the sidecar's record in a container of its own (`tests/Dockerfile`, no ne| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; {{standard_tier}}, no reward; {{reset_budget}} resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; {{standard_tier}}, no reward; {{reset_budget}} resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every {{control_dt}} s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: {{nu}} normalised position targets, {{control_hz}} Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x {{control_dt}} s, results reported apart from S = 1), `RCB_RESET_BUDGET` ({{reset_budget}}: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x {{control_dt}} s, results reported apart from S = 1), `RCB_RESET_BUDGET` ({{reset_budget}}: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |Note: {{standard_camera_note}}diff --git a/scripts/humanoidbench/templates/realtime/docker-compose.yaml b/scripts/humanoidbench/templates/realtime/docker-compose.yamlindex 041a66e5eb..8ed0b68722 100644--- a/scripts/humanoidbench/templates/realtime/docker-compose.yaml+++ b/scripts/humanoidbench/templates/realtime/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-{{reset_budget}}}" # realtime mode's default: {{reset_budget}} resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-{{reset_budget}}}" # standard-realtime mode's default: {{reset_budget}} resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/scripts/humanoidbench/templates/realtime/instruction.md b/scripts/humanoidbench/templates/realtime/instruction.mdindex 6a6882f480..a15bf7bdbd 100644--- a/scripts/humanoidbench/templates/realtime/instruction.md+++ b/scripts/humanoidbench/templates/realtime/instruction.md@@ -1,4 +1,4 @@-# {{family}} — instance {{instance_tag}} — REALTIME MODE (robot as a service; time does not stop)+# {{family}} — instance {{instance_tag}} — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated {{robot_label}} (HumanoidBench / MuJoCo) **as if it were real hardware**: thesimulator runs in a service you cannot open, and you reach it only through a client. Like real hardware, it does not@@ -26,7 +26,7 @@ HumanoidBench scores the *whole episode*, not a final state: it passes iff the sa fresh replay of the recorded episode, the servo targets the robot actually executed at each control step, must reachit as well and end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference (`python -c "import rcb_client;help(rcb_client)"`); the calls and the endpoints they use:diff --git a/scripts/humanoidbench/templates/realtime/task.toml b/scripts/humanoidbench/templates/realtime/task.tomlindex 3a2c544d5d..17f178dc32 100644--- a/scripts/humanoidbench/templates/realtime/task.toml+++ b/scripts/humanoidbench/templates/realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of {{task_dir}} (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of {{task_dir}} (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,10 +16,10 @@ artifacts = [][task]-name = "robot-coding-bench/{{base_dir}}-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench {{family}} on the same frozen instance as {{task_dir}}. Time does not stop: once an episode starts, the robot runs at {{control_hz}} Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to {{reset_budget}} resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/{{base_dir}}-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench {{family}} on the same frozen instance as {{task_dir}}. Time does not stop: once an episode starts, the robot runs at {{control_hz}} Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to {{reset_budget}} resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "{{robot}}", "{{paper_category_long}}", "realtime-mode", "trajectory", "{{family}}"]+keywords = ["robotics", "humanoidbench", "mujoco", "{{robot}}", "{{paper_category_long}}", "standard-realtime-mode", "trajectory", "{{family}}"][metadata]author_name = "pingyue"@@ -27,7 +27,7 @@ task_type = "trajectory"simulator = "humanoidbench"robot = "{{robot_slug}}"family = "{{family}}"-mode = "realtime"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = {{seed}}expert_time_estimate_hours = {{standard_expert_hours}}diff --git a/scripts/live_board.py b/scripts/live_board.pyindex c36dcc3e24..248e4dc0ad 100644--- a/scripts/live_board.py+++ b/scripts/live_board.py@@ -155,7 +155,7 @@ def trial_state(t: pathlib.Path, job: dict, ri: int, root: pathlib.Path, plan_ruexc = (r.get("exception_info") or {}).get("exception_type") or ""return {"id": f"{ri}:{t.relative_to(root)}", "task": task, "bench": info["bench"],- "mode": next((m for m in ("standard", "realtime") if task.endswith(f"-{m}")), "privileged"),+ "mode": next((m for m in ("standard-realtime", "standard") if task.endswith(f"-{m}")), "privileged"),"run": plan_run or run_id((cfg.get("agent") or {})), "job": job.get("job_name") or t.parent.name,"trial": t.name, "state": state, "reward": reward, "exception": exc,"agent_start": a0, "agent_end": a1 or (now if state == "running" else None), "timeout": info["timeout"],@@ -270,7 +270,7 @@ video,img{max-width:100%;border-radius:6px;border:1px solid var(--line)}.close{f<header><h1>Live Board</h1><span class="mut" id="meta"></span></header><main><div class="cards" id="cards"></div><div class="tools"><select id="run"><option value="">all runs</option></select><select id="st"><option value="">all states</option></select>-<select id="mode"><option value="">all modes</option><option>privileged</option><option>standard</option><option>realtime</option></select>+<select id="mode"><option value="">all modes</option><option>privileged</option><option>standard</option><option>standard-realtime</option></select><input id="q" placeholder="filter task / benchmark"><button id="view">matrix view</button></div><div class="wrap" id="table"></div></main><aside id="drawer"></aside>diff --git a/scripts/regrade_trials.sh b/scripts/regrade_trials.shindex cf3e6688af..961c8e77db 100755--- a/scripts/regrade_trials.sh+++ b/scripts/regrade_trials.sh@@ -10,7 +10,7 @@ for tr in "$JOB"/robotwin_2_0_*/; do[ -f "$rj" ] || continue; [ -f "$traj" ] || continuege=$(python3 -c "import json;d=json.load(open('$rj'));print(int(d.get('grader_error',0)))"); vr=$(python3 -c "import json;d=json.load(open('$rj'));print(int(d.get('video_rendered',0)))")[ "$ge" = 1 ] || [ "$vr" = 0 ] || continue- task=$(python3 -c "import json;print(json.load(open('$tr/result.json'))['task_name'].split('/')[-1])"); case "$task" in *-standard|*-realtime|*-limited) continue;; esac # standard and realtime mode: the sidecar grades+ task=$(python3 -c "import json;print(json.load(open('$tr/result.json'))['task_name'].split('/')[-1])"); case "$task" in *-standard|*-standard-realtime|*-limited) continue;; esac # standard and standard-realtime: the sidecar gradescase "$task" in *-privileged) ;; *) task="$task-privileged";; esac # pre-v1.0 jobsfam=${task#robotwin_2_0_}; fam=${fam%-privileged}[ -d "$R/tasks/$task/tests" ] || { echo "skip $task: no tasks/$task/tests"; continue; }diff --git a/scripts/run_agent_batch.sh b/scripts/run_agent_batch.shindex 069e0d9918..c12fc2dc04 100755--- a/scripts/run_agent_batch.sh+++ b/scripts/run_agent_batch.sh@@ -19,13 +19,14 @@# ChatGPT login (codex) CODEX_FORCE_AUTH_JSON=1 (Harbor uploads ~/.codex/auth.json into each trial)# Claude subscription CLAUDE_FORCE_OAUTH=1 and CLAUDE_CODE_OAUTH_TOKEN (`claude setup-token`)# Azure OpenAI (codex) AGENT_CONFIG=<config.toml with the Azure provider> and its key variable-# MODES "privileged standard" K attempts per task (1); add `realtime` for <base>-realtime where it exists+# MODES "privileged standard" K attempts per task (1); add `standard-realtime` for <base>-standard-realtime where it exists# SEQ 1 = one trial at a time per runner (default): task by task, each mode in MODES order, one Harbor job per# task and mode ($JOBS/<batch>-<mode>-<task>/). Start the same command again to add a lane: every job is# claimed (mkdir $JOBS/.claims/<job>) by exactly one runner, so K runners run K trials at a time. A job that# already has its result.json (Harbor writes it when the job starts) is skipped; to rerun an interrupted# job, move its directory and its claim away first.-# 0 = one job per mode ($JOBS/<batch>-<mode>/), the modes side by side, N_P / N_S concurrent trials (2 / 2)+# 0 = one job per mode ($JOBS/<batch>-<mode>/), the modes side by side, N_P / N_S concurrent trials (2 / 2;+# N_S also for standard-realtime)# WAIT_FOR a file that must exist before a mode starts, e.g. "standard:<path>" (default: none)# JOBS <repo>/jobs/agents LABEL the run's title for the results README# RUN_ID the run id (PROTOCOL D4, docs/TAXONOMY.md): <harness>-<model>-<effort>, derived from AGENT / MODEL /@@ -102,7 +103,7 @@ mkdir -p "$JOBS/.plan"python3 - "$R" "$JOBS/.plan/$batch.json" "$batch" "$RUN_ID" "$SEQ" "$MODES" "${tasks[@]}" <<'PY'import json, os, pathlib, sysrepo, out, batch, run_id, seq, modes, *tasks = sys.argv[1:]-name = lambda m, t: {"standard": f"{t}-standard", "limited": f"{t}-standard", "realtime": f"{t}-realtime"}.get(m, f"{t}-privileged")+name = lambda m, t: {"standard": f"{t}-standard", "limited": f"{t}-standard", "standard-realtime": f"{t}-standard-realtime"}.get(m, f"{t}-privileged")jobs = [{"job": f"{batch}-{m}-{t}" if seq == "1" else f"{batch}-{m}", "task": name(m, t), "mode": m}for t in tasks for m in modes.split() if (pathlib.Path(repo) / "tasks" / name(m, t)).is_dir()]tmp = pathlib.Path(f"{out}.{os.getpid()}")@@ -203,7 +204,7 @@ for trial in jobs.glob(f"{batch}-*/*__*"):except Exception:wall = ""task = (r.get("task_name") or trial.name.split("__")[0]).split("/")[-1]- mode = "realtime" if task.endswith("-realtime") else "standard" if task.endswith(("-standard", "-limited")) else "privileged"+ mode = "standard-realtime" if task.endswith("-standard-realtime") else "standard" if task.endswith(("-standard", "-limited")) else "privileged"keys = {k: rw[k] for k in ("deterministic", "live_success", "replay_success", "instance_ok", "missing_trajectory", "grader_error","infra_error", "resumes") if k in rw}rows.append((r.get("started_at") or "~", f"| {task} | {mode} | {rw.get('success', '-')} | {json.dumps(keys)} | "@@ -240,10 +241,10 @@ run_job() { # <job> <concurrency> <harbor -i args...>log "done $job (harbor rc=$?)"}-name() { # <mode> <base> -> task dir name: <base>-privileged / <base>-standard / <base>-realtime (PROTOCOL §2.2)+name() { # <mode> <base> -> task dir name: <base>-privileged / <base>-standard / <base>-standard-realtime (PROTOCOL §2.2)case "$1" instandard|limited) echo "$2-standard" ;;- realtime) echo "$2-realtime" ;;+ standard-realtime) echo "$2-standard-realtime" ;;*) echo "$2-privileged" ;;esac}@@ -274,7 +275,7 @@ elsefor m in $MODES; doinc=(); for t in "${tasks[@]}"; do [ -d "$R/tasks/$(name "$m" "$t")" ] && inc+=(-i "$(name "$m" "$t")"); done[ ${#inc[@]} -eq 0 ] && continue- n="$N_P"; case "$m" in standard|limited) n="$N_S" ;; esac+ n="$N_P"; case "$m" in standard|limited|standard-realtime) n="$N_S" ;; esac # sidecar modes(wait_for "$m"; run_job "$batch-$m" "$n" "${inc[@]}") &pids+=($!)donediff --git a/scripts/task_conventions.py b/scripts/task_conventions.pyindex 9bcb57346c..5ee253e9cc 100644--- a/scripts/task_conventions.py+++ b/scripts/task_conventions.py@@ -8,8 +8,8 @@ pre-v1.0 task (old field names, protocol spelling, README layout); renaming direpython3 scripts/task_conventions.py --check tasks/<task> # exit 1 if normalizing would change anythingWhat it does:-- `task.toml`: `mode` from the directory suffix (`-privileged` / `-standard` / `-realtime`), no `tier`, `protocol = "eai-standard/2.1"`- for standard tasks (2.2 / 2.4 and realtime tasks' `eai-standard/2.3` are set by their generators), `oracle` (+ `oracle_evidence`) for benchmarks whose oracle level is known from their files.+- `task.toml`: `mode` from the directory suffix (`-privileged` / `-standard` / `-standard-realtime`), no `tier`, `protocol = "eai-standard/2.1"`+ for standard tasks (2.2 / 2.4 and standard-realtime tasks' `eai-standard/2.3` are set by their generators), `oracle` (+ `oracle_evidence`) for benchmarks whose oracle level is known from their files.- protocol spelling `rcb-limited/2.x` -> `eai-standard/2.x` and "limited mode" -> "standard mode" in the task's text andservices; the client accepts both spellings.- `tests/test.sh` clears `/logs/verifier` first; graders write `success` (1/0) to `reward.txt` and only numbers to@@ -68,20 +68,23 @@ def replace_words(s: str) -> str:return s-MODES = ("privileged", "standard", "realtime") # PROTOCOL D12: the three modes, each the suffix of its task directories-SIDECAR_MODES = ("standard", "realtime") # robot as a service: the simulator runs in the `sim` sidecar (§4, §4b)+# PROTOCOL D12: the three modes, each the whole suffix of its task directories. A mode is recognised by its complete+# suffix only: `-standard-realtime` is never `-standard` (it does not end in it), and `-standard` is never standard-realtime.+MODES = ("privileged", "standard", "standard-realtime")+SIDECAR_MODES = ("standard", "standard-realtime") # robot as a service: the simulator runs in the `sim` sidecar (§4, §4b)def mode_of(task: Path) -> str:- for mode in MODES:+ for mode in sorted(MODES, key=len, reverse=True): # longest suffix firstif task.name.endswith(f"-{mode}"):return mode- raise ValueError(f"{task.name}: task directories end in -privileged, -standard or -realtime (docs/PROTOCOL.md §2.2)")+ raise ValueError(f"{task.name}: task directories end in -privileged, -standard or -standard-realtime "+ "(docs/PROTOCOL.md §2.2)")def base_of(task: Path) -> str:- """The name without its mode suffix (`-privileged`, `-standard`, `-realtime`; PROTOCOL §2.2)."""- return re.sub(r"-(privileged|standard|realtime)$", "", task.name)+ """The name without its mode suffix (`-privileged`, `-standard`, `-standard-realtime`; PROTOCOL §2.2)."""+ return re.sub(r"-(standard-realtime|privileged|standard)$", "", task.name)def bench_of(task: Path) -> str:diff --git a/scripts/task_names.py b/scripts/task_names.pyindex ec546eae17..a9a37237e2 100644--- a/scripts/task_names.py+++ b/scripts/task_names.py@@ -1,8 +1,14 @@"""Task names across protocol versions: jobs run before protocol v1.0 used `<task>` (privileged) and `<task>-limited`(standard); since v1.0 every task directory ends in its mode, `-privileged` or `-standard`, and since v1.1 also-`-realtime` (docs/PROTOCOL.md §2.2)."""+`-standard-realtime` (docs/PROTOCOL.md §2.2, D12). A mode is recognised by its whole suffix only."""-MODES = ("privileged", "standard", "realtime")+MODES = ("standard-realtime", "privileged", "standard") # longest suffix first+++def mode(name: str) -> str:+ """privileged, standard or standard-realtime."""+ n = canonical(name)+ return next(m for m in MODES if n.endswith(f"-{m}"))def canonical(name: str) -> str:@@ -17,17 +23,12 @@ def canonical(name: str) -> str:def base(name: str) -> str:"""The name without its mode suffix."""n = canonical(name)- return n.rsplit("-", 1)[0]---def mode(name: str) -> str:- """privileged, standard or realtime."""- return canonical(name).rsplit("-", 1)[1]+ return n[: -len(mode(n)) - 1]def is_standard(name: str) -> bool:return mode(name) == "standard"-def is_realtime(name: str) -> bool:- return mode(name) == "realtime"+def is_standard_realtime(name: str) -> bool:+ return mode(name) == "standard-realtime"diff --git a/scripts/write_task_results.py b/scripts/write_task_results.pyindex 15f249d4cf..bfcc467875 100644--- a/scripts/write_task_results.py+++ b/scripts/write_task_results.py@@ -3,7 +3,7 @@(+ replay.mp4, replay.last.png). One folder per agent+model; the privileged and standard twins each get their own.usage: write_task_results.py <jobs_dir> <job_name> <run-id> "<agent label>" --out <dir> [--analysis <make_two_mode_report_zh.py>]"""import json, pathlib, shutil, sys, re, glob, datetime as dt, ast-sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)); from task_names import canonical # noqa: E402+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)); from task_names import base, canonical, mode as mode_of # noqa: E402jobs, job, slug, label = pathlib.Path(sys.argv[1]), sys.argv[2], sys.argv[3], sys.argv[4]root = pathlib.Path(__file__).resolve().parents[1]if "--out" not in sys.argv: sys.exit("--out <dir> is required: results are published on the results site, not committed (PROTOCOL D3)")@@ -30,7 +30,7 @@ for trial in sorted((jobs / job).glob("*__*")):tdir = root / "tasks" / taskif not tdir.is_dir(): continuerw = json.load(open(trial / "verifier/reward.json")) if (trial / "verifier/reward.json").exists() else {}- ar = r.get("agent_result") or {}; mode = task.rsplit("-", 1)[1]; limited = mode != "privileged"; fam = task.replace("robotwin_2_0_", "").rsplit("-", 1)[0]+ ar = r.get("agent_result") or {}; mode = mode_of(task); limited = mode != "privileged"; fam = base(task).replace("robotwin_2_0_", "")out = out_root / task / slug; out.mkdir(parents=True, exist_ok=True)for f in ("replay.mp4", "replay.last.png"):if (trial / "verifier" / f).exists(): shutil.copy(trial / "verifier" / f, out / f)@@ -39,7 +39,7 @@ for trial in sorted((jobs / job).glob("*__*")):ok = rw.get("success") == 1 and not overan = analysis.get(fam, {}).get("L" if limited else "U", "")lines = [f"# {label} — {task}\n",- f"Run: Harbor job `{job}`, trial `{trial.name}`, {(r.get('started_at') or '')[:10]}. Mode: **{ {'standard': 'standard (robot as a service)', 'realtime': 'realtime (robot as a service, time does not stop)'}.get(mode, 'privileged')}**.\n",+ f"Run: Harbor job `{job}`, trial `{trial.name}`, {(r.get('started_at') or '')[:10]}. Mode: **{ {'standard': 'standard (robot as a service)', 'standard-realtime': 'standard-realtime (robot as a service, time does not stop)'}.get(mode, 'privileged')}**.\n","| | |", "|---|---|",f"| result | {'**1 (success)**' if ok else '**0**' + (' — over the token budget' if over else ' — no trajectory handed in' if rw.get('missing_trajectory') else '')} |",f"| agent time | {span(r.get('agent_execution') or {})} s |",diff --git a/tasks/dataset.toml b/tasks/dataset.tomlindex c0326e5d9b..2129b4b060 100644--- a/tasks/dataset.toml+++ b/tasks/dataset.toml@@ -4857,46 +4857,6 @@ digest = "sha256:bcd3454da1d025561cd49379c330a500c627553a5f8356022de9444ca47ad05name = "robot-coding-bench/behavior-wash-a-baseball-cap-i00-standard"digest = "sha256:8e41c22f5a7f3b0c1137874d2841fc3e074a3ef4645faa1182b110f919b4d795"-[[tasks]]-name = "robot-coding-bench/humanoidbench-loco-balance-hard-i00-realtime"-digest = "sha256:3bc849e4978c8d460622149681ef7de68acea397b865fb25349ecab14de2e2b6"--[[tasks]]-name = "robot-coding-bench/humanoidbench-loco-hurdle-i00-realtime"-digest = "sha256:07a0e363f302e169045cb1657b1d0aa8dbfe2fba7277cad7b385a753b9b516c2"--[[tasks]]-name = "robot-coding-bench/humanoidbench-loco-sit-simple-i00-realtime"-digest = "sha256:61e0d7446aeb1166c0e88a0169f40e37bae48d666af3a065bd6e6b0f0c20e5cd"--[[tasks]]-name = "robot-coding-bench/humanoidbench-loco-walk-i00-realtime"-digest = "sha256:64807537534bcc06e61d31883112af37db4b099ca815731fea99c1650fa79e9f"--[[tasks]]-name = "robot-coding-bench/humanoidbench-manip-bookshelf-simple-i00-realtime"-digest = "sha256:9d0079f945ab17e1fe913f25e6117a484e70af701ba12a436a9a89611c1d84e9"--[[tasks]]-name = "robot-coding-bench/humanoidbench-manip-cube-i00-realtime"-digest = "sha256:f029c3a894980d4f8dc0743de18db414cb5e08d45d3e73b26e8dba43c276f4b5"--[[tasks]]-name = "robot-coding-bench/humanoidbench-manip-door-i00-realtime"-digest = "sha256:5da640207ccf505e9a3e5f11bfbb16da506c366f17c8587c7ad59d1f3083dc8e"--[[tasks]]-name = "robot-coding-bench/humanoidbench-manip-powerlift-i00-realtime"-digest = "sha256:9d638d785822688ecbd508caeaca7813dcb2115e713a945d535b3358950fae16"--[[tasks]]-name = "robot-coding-bench/humanoidbench-manip-room-i00-realtime"-digest = "sha256:0f69d1f68d980bee70e3a3c386035aeb6ef60a8e185e2fc3f6b3f0870aae9979"--[[tasks]]-name = "robot-coding-bench/humanoidbench-manip-window-i00-realtime"-digest = "sha256:bb9f5d6ec8905c2e5192acf52ff935dd89214a0dda96d4a8b04543fdd1b2cf6c"-[[tasks]]name = "robot-coding-bench/robotangle-blanket-flatten-easy-i00-privileged"digest = "sha256:5eef1c1a53064e5f43ee98286d67fdd5f64b18e7c63fde8b768458f054e31681"@@ -5313,3 +5273,43 @@ digest = "sha256:ce0ffbbac335105e6ee0929b4cca074de772b380df32e28c03067f0ed11898fname = "robot-coding-bench/robotangle-rope-threading-medium-i00-standard"digest = "sha256:61a76e1dc4a129237492c8d6009b20e35c4f2c0848b581a265445b531dab291e"+[[tasks]]+name = "robot-coding-bench/humanoidbench-loco-balance-hard-i00-standard-realtime"+digest = "sha256:fb83727ee84b95669150682c085685f6d6b411daa32df8e13ca1117d7505a7bd"++[[tasks]]+name = "robot-coding-bench/humanoidbench-loco-hurdle-i00-standard-realtime"+digest = "sha256:632b9a97b1de2444cc06c3831191ac230cc850e0d8199b85f7219c26bb759a19"++[[tasks]]+name = "robot-coding-bench/humanoidbench-loco-sit-simple-i00-standard-realtime"+digest = "sha256:d9a419de81ce89d4abe8ea0bf9f766a02ab654e0aa89bd78ee6c72a0a63a80c9"++[[tasks]]+name = "robot-coding-bench/humanoidbench-loco-walk-i00-standard-realtime"+digest = "sha256:a36f2681e7febf5f23aa76ed80f87305c217ddd58a9f5755307cd59dd51e5c5c"++[[tasks]]+name = "robot-coding-bench/humanoidbench-manip-bookshelf-simple-i00-standard-realtime"+digest = "sha256:8e40cd364013690468d3321116902e133b064cc3d00cccda1777e05672f25dad"++[[tasks]]+name = "robot-coding-bench/humanoidbench-manip-cube-i00-standard-realtime"+digest = "sha256:d0acf7ad4e6fa851896dd9171e8244af38de4eeaa27a9cedaca4eb4a07baa850"++[[tasks]]+name = "robot-coding-bench/humanoidbench-manip-door-i00-standard-realtime"+digest = "sha256:329c3664690de2aad723b2ca84e60c148326537d96606cfefa5264fafaaa0099"++[[tasks]]+name = "robot-coding-bench/humanoidbench-manip-powerlift-i00-standard-realtime"+digest = "sha256:3c0bb3b4937bd83fe8a7ddb5c948d603dbf70430aef5cbc481c5b8f0d918ad21"++[[tasks]]+name = "robot-coding-bench/humanoidbench-manip-room-i00-standard-realtime"+digest = "sha256:0141c76441205aa6bb46cbe11b1382d66ab7b139f2d01d65d217b69d0de45633"++[[tasks]]+name = "robot-coding-bench/humanoidbench-manip-window-i00-standard-realtime"+digest = "sha256:644ea74934e769224704ca3a02b7ac51d6c04f15394d9a3364b70a5816366dc9"+diff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/task.toml b/tasks/humanoidbench-loco-balance-hard-i00-realtime/task.tomldeleted file mode 100644index 03a9fdc40c..0000000000--- a/tasks/humanoidbench-loco-balance-hard-i00-realtime/task.toml+++ /dev/null@@ -1,68 +0,0 @@-# Realtime-mode task of humanoidbench-loco-balance-hard-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit-# subset.toml or the template, not this.-schema_version = "1.3"-artifacts = [- { source = "/data/final.json", service = "sim" },- { source = "/data/episode.jsonl", service = "sim" },- { source = "/data/trajectory.npz", service = "sim" },- { source = "/data/trajectory_live.f64", service = "sim" },- { source = "/data/replay.mp4", service = "sim" },- { source = "/data/replay.last.png", service = "sim" },- { source = "/data/ticks.f64", service = "sim" },- { source = "/data/calls.jsonl", service = "sim" },- { source = "/data/submitted.f64", service = "sim" },- { source = "/data/startups.jsonl", service = "sim" },- "/app/output/",-]--[task]-name = "robot-coding-bench/humanoidbench-loco-balance-hard-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench balance_hard on the same frozen instance as humanoidbench-loco-balance-hard-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."-authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1", "locomotion", "realtime-mode", "trajectory", "balance_hard"]--[metadata]-author_name = "pingyue"-task_type = "trajectory"-simulator = "humanoidbench"-robot = "unitree-h1"-family = "balance_hard"-mode = "realtime"-protocol = "eai-standard/2.3"-instance_seed = 0-expert_time_estimate_hours = 8.0-oracle = "none"-oracle_evidence = "none known: no run has passed it, and published RL reaches 31% of the bar (paper 57, FastTD3 244, bar 800); human review in PR #83"--[verifier]-timeout_sec = 900.0-environment_mode = "separate"-network_mode = "no-network"--# The agent's time is up: stop whatever it left running in its container. Harbor stops that container only after-# collecting from it (~12 s), and a script still running there would keep driving the robot past the deadline.-[[verifier.collect]]-service = "main"-command = "kill -9 -1 2>/dev/null; exit 0"-timeout_sec = 30--# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.-[[verifier.collect]]-service = "sim"-command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"-timeout_sec = 900--[agent]-timeout_sec = 3600.0-# The agent container holds no simulator and no reward, but upstream is a public commit: like every HumanoidBench-# task, the agent reaches the model APIs and nothing else (scripts/set_agent_egress.py HOSTS).-network_mode = "allowlist"-allowed_hosts = ["chatgpt.com", "*.chatgpt.com", "auth.openai.com", "api.openai.com", "*.openai.com", "api.anthropic.com", "*.anthropic.com", "openrouter.ai", "*.openai.azure.com", "*.cognitiveservices.azure.com"]--[environment]-build_timeout_sec = 1800.0-cpus = 4-memory_mb = 16384--[verifier.environment]-build_timeout_sec = 600.0diff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/README.md b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/README.mdsimilarity index 82%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/README.mdrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/README.mdindex 1477b7d421..8739f46134 100644--- a/tasks/humanoidbench-loco-balance-hard-i00-realtime/README.md+++ b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-loco-balance-hard-i00-realtime+# humanoidbench-loco-balance-hard-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-loco-balance-hard-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1` || Family / instance | `balance_hard` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-loco-balance-hard-i00-privileged`: the same frozen instance and the same success rule as it andas `humanoidbench-loco-balance-hard-i00-standard`, but time does not stop. Once an episode's clock starts (itsfirst `submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the agent thinks,@@ -56,8 +56,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no board or roller pose, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no board or roller pose, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 19 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 89%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/docker-compose.yamlindex e51aeb11e5..209fdf0cf9 100644--- a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/rcb_client.pysimilarity index 98%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/rcb_client.pyrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/rcb_client.pyindex 8e057bb6e9..f2b0559bf9 100644--- a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/rcb_client.py+++ b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/rcb_client.py@@ -1,4 +1,4 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, onPYTHONPATH).from rcb_client import Robot, EpisodeOverdiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb_realtime.pysimilarity index 97%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb_realtime.pyrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb_realtime.pyindex 410284bf27..e23f8cbd2c 100644--- a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb_realtime.py+++ b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -1,7 +1,8 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recoveryand finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:@@ -55,7 +56,7 @@ import numpy as npimport rcb_servicefrom rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request-PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching upCHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)diff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/instruction.md b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/instruction.mdrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/instruction.mdindex 01927eef3a..856c4e8689 100644--- a/tasks/humanoidbench-loco-balance-hard-i00-realtime/instruction.md+++ b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# balance_hard — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# balance_hard — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid (HumanoidBench / MuJoCo) **as if it were real hardware**: thesimulator runs in a service you cannot open, and you reach it only through a client. Like real hardware, it@@ -75,7 +75,7 @@ and the board, and every step after that is worth nothing. The service ends themark is reached (`reason = "success"`), and a fresh replay of the recorded episode, the servo targets therobot actually executed at each control step, must reach it as well and end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/task.toml b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/task.tomlsimilarity index 77%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/task.tomlrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/task.tomlindex 87a92484d7..8281917fe9 100644--- a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/task.toml+++ b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of humanoidbench-manip-bookshelf-simple-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of humanoidbench-loco-balance-hard-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,23 +16,23 @@ artifacts = [][task]-name = "robot-coding-bench/humanoidbench-manip-bookshelf-simple-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench bookshelf_simple on the same frozen instance as humanoidbench-manip-bookshelf-simple-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/humanoidbench-loco-balance-hard-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench balance_hard on the same frozen instance as humanoidbench-loco-balance-hard-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "realtime-mode", "trajectory", "bookshelf_simple"]+keywords = ["robotics", "humanoidbench", "mujoco", "h1", "locomotion", "standard-realtime-mode", "trajectory", "balance_hard"][metadata]author_name = "pingyue"task_type = "trajectory"simulator = "humanoidbench"-robot = "unitree-h1hand"-family = "bookshelf_simple"-mode = "realtime"+robot = "unitree-h1"+family = "balance_hard"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = 0-expert_time_estimate_hours = 10.0+expert_time_estimate_hours = 8.0oracle = "none"-oracle_evidence = "none known: no run has passed it, and published RL reaches 41% of the bar (paper 763, FastTD3 822, bar 2000); human review in PR #83"+oracle_evidence = "none known: no run has passed it, and published RL reaches 31% of the bar (paper 57, FastTD3 244, bar 800); human review in PR #83"[verifier]timeout_sec = 900.0diff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/tests/grade.py b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-loco-balance-hard-i00-realtime/tests/test.sh b/tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-loco-balance-hard-i00-realtime/tests/test.shrename to tasks/humanoidbench-loco-balance-hard-i00-standard-realtime/tests/test.shdiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/README.md b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/README.mdsimilarity index 82%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/README.mdrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/README.mdindex 7da1de4f79..01e4e1dda5 100644--- a/tasks/humanoidbench-loco-hurdle-i00-realtime/README.md+++ b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-loco-hurdle-i00-realtime+# humanoidbench-loco-hurdle-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-loco-hurdle-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1` || Family / instance | `hurdle` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-loco-hurdle-i00-privileged`: the same frozen instance and the same success rule as it and as`humanoidbench-loco-hurdle-i00-standard`, but time does not stop. Once an episode's clock starts (its first`submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the agent thinks, so@@ -56,8 +56,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no hurdle or wall positions, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no hurdle or wall positions, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 19 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 89%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/docker-compose.yamlindex f602bcec1c..9b664973c3 100644--- a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/rcb_client.pysimilarity index 98%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/rcb_client.pyrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/rcb_client.pyindex 8e057bb6e9..f2b0559bf9 100644--- a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/rcb_client.py+++ b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/rcb_client.py@@ -1,4 +1,4 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, onPYTHONPATH).from rcb_client import Robot, EpisodeOverdiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb_realtime.pysimilarity index 97%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb_realtime.pyrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb_realtime.pyindex 410284bf27..e23f8cbd2c 100644--- a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb_realtime.py+++ b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -1,7 +1,8 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recoveryand finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:@@ -55,7 +56,7 @@ import numpy as npimport rcb_servicefrom rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request-PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching upCHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)diff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/instruction.md b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/instruction.mdrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/instruction.mdindex 076291fea8..10fcc2611b 100644--- a/tasks/humanoidbench-loco-hurdle-i00-realtime/instruction.md+++ b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# hurdle — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# hurdle — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid (HumanoidBench / MuJoCo) **as if it were real hardware**: thesimulator runs in a service you cannot open, and you reach it only through a client. Like real hardware, it@@ -76,7 +76,7 @@ service ends the episode the moment the pass mark is reached (`reason = "successrecorded episode, the servo targets the robot actually executed at each control step, must reach it as welland end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/task.toml b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/task.tomlsimilarity index 70%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/task.tomlrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/task.tomlindex 74206360cd..1b6af5861a 100644--- a/tasks/humanoidbench-loco-hurdle-i00-realtime/task.toml+++ b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of humanoidbench-loco-hurdle-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of humanoidbench-loco-hurdle-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,10 +16,10 @@ artifacts = [][task]-name = "robot-coding-bench/humanoidbench-loco-hurdle-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench hurdle on the same frozen instance as humanoidbench-loco-hurdle-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/humanoidbench-loco-hurdle-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench hurdle on the same frozen instance as humanoidbench-loco-hurdle-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1", "locomotion", "realtime-mode", "trajectory", "hurdle"]+keywords = ["robotics", "humanoidbench", "mujoco", "h1", "locomotion", "standard-realtime-mode", "trajectory", "hurdle"][metadata]author_name = "pingyue"@@ -27,7 +27,7 @@ task_type = "trajectory"simulator = "humanoidbench"robot = "unitree-h1"family = "hurdle"-mode = "realtime"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = 0expert_time_estimate_hours = 6.0diff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/tests/grade.py b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/tests/test.sh b/tasks/humanoidbench-loco-hurdle-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/tests/test.shrename to tasks/humanoidbench-loco-hurdle-i00-standard-realtime/tests/test.shdiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/README.md b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/README.mdsimilarity index 82%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/README.mdrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/README.mdindex b4bd6088d5..b3da9a73d6 100644--- a/tasks/humanoidbench-loco-sit-simple-i00-realtime/README.md+++ b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-loco-sit-simple-i00-realtime+# humanoidbench-loco-sit-simple-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-loco-sit-simple-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1` || Family / instance | `sit_simple` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-loco-sit-simple-i00-privileged`: the same frozen instance and the same success rule as it andas `humanoidbench-loco-sit-simple-i00-standard`, but time does not stop. Once an episode's clock starts (itsfirst `submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the agent thinks,@@ -57,8 +57,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no chair pose, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no chair pose, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 19 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 88%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/docker-compose.yamlindex dfb70fcd83..bedea512f8 100644--- a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/rcb_client.pysimilarity index 98%rename from tasks/humanoidbench-loco-hurdle-i00-realtime/environment/rcb_client.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/rcb_client.pyindex 8e057bb6e9..f2b0559bf9 100644--- a/tasks/humanoidbench-loco-hurdle-i00-realtime/environment/rcb_client.py+++ b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/rcb_client.py@@ -1,4 +1,4 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, onPYTHONPATH).from rcb_client import Robot, EpisodeOverdiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb_realtime.pysimilarity index 97%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb_realtime.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb_realtime.pyindex 410284bf27..e23f8cbd2c 100644--- a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb_realtime.py+++ b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -1,7 +1,8 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recoveryand finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:@@ -55,7 +56,7 @@ import numpy as npimport rcb_servicefrom rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request-PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching upCHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)diff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/instruction.md b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/instruction.mdrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/instruction.mdindex fe57979e7d..8d988c9811 100644--- a/tasks/humanoidbench-loco-sit-simple-i00-realtime/instruction.md+++ b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# sit_simple — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# sit_simple — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid (HumanoidBench / MuJoCo) **as if it were real hardware**: thesimulator runs in a service you cannot open, and you reach it only through a client. Like real hardware, it@@ -74,7 +74,7 @@ could have paid, so both *how fast* and *how steadily* the robot sits matter. Thmoment the pass mark is reached (`reason = "success"`), and a fresh replay of the recorded episode, the servotargets the robot actually executed at each control step, must reach it as well and end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/solution/oracle_stream.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/solution/oracle_stream.pysimilarity index 90%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/solution/oracle_stream.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/solution/oracle_stream.pyindex 339ee2c597..92357e2c7d 100644--- a/tasks/humanoidbench-loco-sit-simple-i00-realtime/solution/oracle_stream.py+++ b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/solution/oracle_stream.py@@ -1,4 +1,4 @@-"""Oracle (realtime mode): the reference actions (sit_simple_actions.npy, the privileged oracle's 999 rows of+"""Oracle (standard-realtime mode): the reference actions (sit_simple_actions.npy, the privileged oracle's 999 rows ofnormalised servo targets) streamed through the client into the running episode's plan, one row per control step(dt = control_dt, hold): the first chunk starts the clock at step 0, the second is planned from the step where thefirst ends, long before that step comes, so every row runs at its own control step. The service ends the episode thediff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/solution/sit_simple_actions.npy b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/solution/sit_simple_actions.npysimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/solution/sit_simple_actions.npyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/solution/sit_simple_actions.npydiff --git a/scripts/humanoidbench/oracles/sit_simple/realtime/solve.sh b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/solution/solve.shsimilarity index 68%rename from scripts/humanoidbench/oracles/sit_simple/realtime/solve.shrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/solution/solve.shindex e2239bea1f..898a00c1f6 100755--- a/scripts/humanoidbench/oracles/sit_simple/realtime/solve.sh+++ b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/solution/solve.sh@@ -1,5 +1,5 @@#!/bin/bash-# Oracle (realtime mode): the privileged oracle's recorded actions (solution/sit_simple_actions.npy) streamed+# Oracle (standard-realtime mode): the privileged oracle's recorded actions (solution/sit_simple_actions.npy) streamed# through the client into the episode's plan, one row per control step. No privileged information at run time: only# the recorded actions.set -ediff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/task.toml b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/task.tomlsimilarity index 69%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/task.tomlrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/task.tomlindex 3e7cf5ee73..efdcf57c1e 100644--- a/tasks/humanoidbench-loco-sit-simple-i00-realtime/task.toml+++ b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of humanoidbench-loco-sit-simple-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of humanoidbench-loco-sit-simple-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,10 +16,10 @@ artifacts = [][task]-name = "robot-coding-bench/humanoidbench-loco-sit-simple-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench sit_simple on the same frozen instance as humanoidbench-loco-sit-simple-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/humanoidbench-loco-sit-simple-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench sit_simple on the same frozen instance as humanoidbench-loco-sit-simple-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1", "locomotion", "realtime-mode", "trajectory", "sit_simple"]+keywords = ["robotics", "humanoidbench", "mujoco", "h1", "locomotion", "standard-realtime-mode", "trajectory", "sit_simple"][metadata]author_name = "pingyue"@@ -27,7 +27,7 @@ task_type = "trajectory"simulator = "humanoidbench"robot = "unitree-h1"family = "sit_simple"-mode = "realtime"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = 0expert_time_estimate_hours = 2.0diff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/tests/grade.py b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-loco-sit-simple-i00-realtime/tests/test.sh b/tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-loco-sit-simple-i00-realtime/tests/test.shrename to tasks/humanoidbench-loco-sit-simple-i00-standard-realtime/tests/test.shdiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/README.md b/tasks/humanoidbench-loco-walk-i00-standard-realtime/README.mdsimilarity index 83%rename from tasks/humanoidbench-loco-walk-i00-realtime/README.mdrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/README.mdindex 85136cd4d9..60c974841d 100644--- a/tasks/humanoidbench-loco-walk-i00-realtime/README.md+++ b/tasks/humanoidbench-loco-walk-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-loco-walk-i00-realtime+# humanoidbench-loco-walk-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-loco-walk-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1` || Family / instance | `walk` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-loco-walk-i00-privileged`: the same frozen instance and the same success rule as it and as`humanoidbench-loco-walk-i00-standard`, but time does not stop. Once an episode's clock starts (its first`submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the agent thinks, so@@ -56,8 +56,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no base velocity, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no base velocity, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 19 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 89%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/docker-compose.yamlindex 827101ffc6..7f0e798b4b 100644--- a/tasks/humanoidbench-loco-walk-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/rcb_client.pysimilarity index 98%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/rcb_client.pyrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/rcb_client.pyindex 8e057bb6e9..f2b0559bf9 100644--- a/tasks/humanoidbench-loco-walk-i00-realtime/environment/rcb_client.py+++ b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/rcb_client.py@@ -1,4 +1,4 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, onPYTHONPATH).from rcb_client import Robot, EpisodeOverdiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb_realtime.pysimilarity index 97%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb_realtime.pyrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb_realtime.pyindex 410284bf27..e23f8cbd2c 100644--- a/tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb_realtime.py+++ b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -1,7 +1,8 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recoveryand finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:@@ -55,7 +56,7 @@ import numpy as npimport rcb_servicefrom rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request-PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching upCHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)diff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/instruction.md b/tasks/humanoidbench-loco-walk-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-loco-walk-i00-realtime/instruction.mdrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/instruction.mdindex bf8ec91bdd..576e5fae46 100644--- a/tasks/humanoidbench-loco-walk-i00-realtime/instruction.md+++ b/tasks/humanoidbench-loco-walk-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# walk — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# walk — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid (HumanoidBench / MuJoCo) **as if it were real hardware**: thesimulator runs in a service you cannot open, and you reach it only through a client. Like real hardware, it@@ -73,7 +73,7 @@ on the order of 1 m/s. The service ends the episode the moment the pass mark is(`reason = "success"`), and a fresh replay of the recorded episode, the servo targets the robot actuallyexecuted at each control step, must reach it as well and end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-loco-walk-i00-realtime/task.toml b/tasks/humanoidbench-loco-walk-i00-standard-realtime/task.tomlsimilarity index 73%rename from tasks/humanoidbench-loco-walk-i00-realtime/task.tomlrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/task.tomlindex ff86cc8564..4e6aa9a039 100644--- a/tasks/humanoidbench-loco-walk-i00-realtime/task.toml+++ b/tasks/humanoidbench-loco-walk-i00-standard-realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of humanoidbench-loco-walk-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of humanoidbench-loco-walk-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,10 +16,10 @@ artifacts = [][task]-name = "robot-coding-bench/humanoidbench-loco-walk-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench walk on the same frozen instance as humanoidbench-loco-walk-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/humanoidbench-loco-walk-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench walk on the same frozen instance as humanoidbench-loco-walk-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1", "locomotion", "realtime-mode", "trajectory", "walk"]+keywords = ["robotics", "humanoidbench", "mujoco", "h1", "locomotion", "standard-realtime-mode", "trajectory", "walk"][metadata]author_name = "pingyue"@@ -27,7 +27,7 @@ task_type = "trajectory"simulator = "humanoidbench"robot = "unitree-h1"family = "walk"-mode = "realtime"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = 0expert_time_estimate_hours = 4.0diff --git a/tasks/humanoidbench-loco-walk-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-loco-walk-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-loco-walk-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-loco-walk-i00-realtime/tests/grade.py b/tasks/humanoidbench-loco-walk-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-loco-walk-i00-realtime/tests/test.sh b/tasks/humanoidbench-loco-walk-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-loco-walk-i00-realtime/tests/test.shrename to tasks/humanoidbench-loco-walk-i00-standard-realtime/tests/test.shdiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/rcb_client.pydeleted file mode 100644index 8e057bb6e9..0000000000--- a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/rcb_client.py+++ /dev/null@@ -1,209 +0,0 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on-PYTHONPATH).-- from rcb_client import Robot, EpisodeOver- r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready--TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or-get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x-spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a-plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out-the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.--Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):- r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"- ("joint_position": dim, layout (actuator order), lower/upper, range_rad,- units, semantics), "control_dt" (0.02 s), "max_control_steps" (the- episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:- rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"- (joints, axes, positions), "imu", "links" (masses, centres of mass, the- feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"- r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,- planned_steps, lag_s (and episode, resets_left with a reset budget)- r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")- POST /observe. seconds=0: the robot now, at once. seconds>0 (at most- spec()["max_watch_s"]): waits while that much robot time runs (starting the- clock if it is not running) and returns what the cameras saw; it never- changes the plan- r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,- video="none", include_depth=False, note="")- POST /submit: answers at once with the robot as it is now (the rows- have not run yet). actions (T, dim): rows dt apart (a multiple of- control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's- target is the row (interpolate="hold") or a straight line from the previous- row's ("linear", starting from the target in force just before `at`). They- become the plan from control step `at` (an episode step number; None = the- next step), replacing whatever was planned from `at` on and keeping what is- planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows- for steps already gone when the request arrives are dropped- (meta["late_steps"]). The first submit of an episode starts its clock- r.reset(cameras=None, include_depth=False)- POST /reset, only when spec()["budget"]["reset"] > 0: abandons the- running episode and starts a new one, paused, from the same starting state;- only the last episode is graded, and a success ends the trial--cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the-fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),-"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth-for the cameras that have it.--Result (what get_obs / submit / reset return):- res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),- sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |- "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,- lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]- (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,- resets_left- res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])- res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes- res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)- res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s- of a watch, one at its end- res.done the episode is over- res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy--Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an-unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over-(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this-run (404).-"""-from __future__ import annotations--import json-import os-import pathlib-import time--import numpy as np-import requests--import rcb1--PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)-MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused---class EpisodeOver(RuntimeError):- pass---class Result:- """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""-- def __init__(self, meta: dict, parts: dict):- self.meta, self.raw = meta, parts- self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None- for name, p in parts.items():- kind, data = p["kind"], p["data"]- if name.startswith("video/") and kind == "mp4":- self.video[name[6:]] = data- elif name.startswith("frames/"):- self.frames[name[7:]] = data- elif name.startswith("last/"):- self.last[name[5:]] = data- elif name.startswith("depth/"):- self.depth[name[6:]] = data- elif name == "proprio":- self.proprio = data-- @property- def done(self) -> bool:- return bool(self.meta.get("done"))-- def save(self, out_dir: str) -> str:- from PIL import Image- out = pathlib.Path(out_dir)- out.mkdir(parents=True, exist_ok=True)- (out / "meta.json").write_text(json.dumps(self.meta, indent=1))- for cam, b in self.video.items():- (out / f"{cam}.mp4").write_bytes(b)- for cam, img in self.last.items():- (out / "last").mkdir(exist_ok=True)- Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)- for cam, arr in self.frames.items():- d = out / "frames" / cam- d.mkdir(parents=True, exist_ok=True)- for i, f in enumerate(arr):- Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)- for cam, arr in self.depth.items():- np.save(out / f"depth_{cam}.npy", arr)- if self.proprio is not None:- np.save(out / "proprio.npy", self.proprio)- return str(out)---class Robot:- def __init__(self, base: str | None = None, timeout: float = 600.0):- candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]- t0, last = time.time(), None- while time.time() - t0 < timeout:- for b in candidates:- try:- r = requests.get(b + "/spec", timeout=(5, 60))- if r.status_code == 503:- last = "warming up"- continue- r.raise_for_status()- self.base, self._spec = b, r.json()- if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):- raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")- return- except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:- last = e # not up yet (the scene takes about a minute), or busy- time.sleep(2)- raise RuntimeError(f"sim service unavailable: {last}")-- def spec(self) -> dict:- return self._spec-- def status(self) -> dict:- r = requests.get(self.base + "/status", timeout=(5, 60))- r.raise_for_status()- return r.json()-- def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,- video: str = "frames") -> Result:- meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),- "include_depth": bool(include_depth), "video": video}- return self._post("/observe", meta, [], timeout=900)-- def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",- at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",- include_depth: bool = False, note: str = "") -> Result:- a = np.asarray(actions, dtype=np.float64)- if a.ndim == 1:- a = a[None, :]- if not np.all(np.isfinite(a)):- raise ValueError("actions contain NaN or inf")- meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,- "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),- "video": video,- "include_depth": bool(include_depth), "note": note}- return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)-- def reset(self, cameras=None, include_depth: bool = False) -> Result:- return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)-- def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:- body = rcb1.encode(meta, parts)- if len(body) > MAX_BODY: # the service would close the connection on it unread- raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")- r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},- timeout=(5, timeout))- if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:- return Result(*rcb1.decode(r.content))- try:- err = r.json()- except ValueError:- err = {"error": r.text[:300]}- msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")- if r.status_code == 409:- raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")- + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))- if r.status_code in (400, 413):- raise ValueError(msg)- raise RuntimeError(msg)---def _names(cameras) -> list[str] | None:- return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb_realtime.pydeleted file mode 100644index 410284bf27..0000000000--- a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb_realtime.py+++ /dev/null@@ -1,479 +0,0 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.--It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery-and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:--- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its- first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then- on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,- until the episode ends; the clock stops there.-- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`- or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the- next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`- loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order- hold): a slow agent's robot keeps its last servo targets.-- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)- and returns what the cameras saw. It never changes the plan.-- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.--Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's-rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was-submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):-ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line-per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:-call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.--Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in-between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting-request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than-MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.--A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.-A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a-crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the-trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends-the graded episode (reason "error") sets infra_error in final.json too.-"""-from __future__ import annotations--import collections-import json-import os-import queue-import sys-import threading-import time-import traceback-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer-from urllib.parse import urlparse--import numpy as np--import rcb_service-from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request--PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1-MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up-CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)---class _Job:- """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""-- def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):- self.method, self.path, self.body, self.content_length = method, path, body, content_length- self.done = threading.Event()- self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)-- def finish(self, *reply) -> None:- self.reply = reply- self.done.set()---class _Watch:- """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""-- def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):- self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id- self.start, self.end = start, start + steps---class RealtimeService(Service):- """rcb_service.Service with the realtime clock (module docstring)."""-- def __init__(self, sim, *, slowdown: float | None = None, **kw):- self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)- if not self.slowdown > 0:- raise ValueError("RCB_SLOWDOWN must be > 0")- self.wall_dt = sim.control_dt * self.slowdown- self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be- self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step- self.running = False- self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t- self.watches: list[_Watch] = []- self.q: queue.Queue = queue.Queue()- self.max_lag = self.lag = 0.0- self.stalls = self.held_steps = self.late_steps = 0- self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk- super().__init__(sim, **kw)- self.ticks_path = os.path.join(self.data, "ticks.f64")- self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))-- # ------------------------------------------------------------------ spec, status, evidence- def _spec(self) -> dict:- spec = super()._spec()- spec["protocol"] = PROTOCOL- cdt = self.sim.control_dt- spec["timing"] = {- "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,- "wall_s_per_control_step": cdt * self.slowdown,- "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "- "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "- "you do, until the episode ends",- "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "- "`at` (default: the next one), replacing what was planned from there; rows for steps already "- "gone are dropped (meta.late_steps); after the plan's last row that target is held",- "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "- "the cameras saw; never changes the plan",- "max_ahead_steps": self.max_ahead,- "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}- return spec-- def _realtime_counters(self) -> dict:- return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}-- def status(self) -> dict:- return {**super().status(), **self._realtime_counters()}-- def meta(self) -> dict:- return {**super().meta(), **self._realtime_counters()}-- def _timing_record(self) -> dict:- return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),- "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}-- def _write_final(self) -> None:- self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's- if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)- self.verdict["infra_error"] = 1- else: # (an episode that erred and was reset away is not graded)- self.verdict.pop("infra_error", None)- super()._write_final()-- def _checkpoint(self, ev: str, **kw) -> None:- if self._ticks: # the tick log first: a crash in between leaves it too long- with open(self.ticks_path, "ab") as f:- f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())- self._ticks = []- super()._checkpoint(ev, **kw)-- def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:- keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",- "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")- line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}- with open(os.path.join(self.data, "calls.jsonl"), "a") as f:- f.write(json.dumps(line) + "\n")- if actions is not None:- rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)- with open(os.path.join(self.data, "submitted.f64"), "ab") as f:- f.write(np.asarray(rows, dtype=np.float64).tobytes())-- # ------------------------------------------------------------------ start, clock- def start(self) -> None:- super().start()- rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)- n = self.journal_rows- if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too- rows[: 2 * n].tofile(self.ticks_path)- elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)- np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)- self.running = False # a resumed episode waits, paused, for the agent's next call-- def _start_clock(self) -> None:- if not self.running and not self.done and not self.stopping:- self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)- self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)-- def _stop_clock(self, why: str) -> None:- if self.running:- self.running = False- self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))- self.plan.clear()-- def _end(self, reason: str) -> None:- super()._end(reason)- self._stop_clock(reason) # pending watches are completed by whoever ended it, after this-- def due_in(self, now: float) -> float | None:- """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""- if not self.running or self.done or self.stopping:- return None- return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now-- def tick(self) -> None:- """One control step of the clock: the plan's row for it, or the held target."""- now = time.monotonic()- lag = -(self.due_in(now) or 0.0)- if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt- self.stalls += 1- self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))- self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0- self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)- if self.plan:- typ, row, src = self.plan.popleft()- else:- (typ, row), src = self.sim.hold(), -1- self.held_steps += 1- before = self.journal_rows- self._ticks.append((time.time(), float(src)))- try:- self._step(typ, row)- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- if self.journal_rows == before:- self._ticks.pop()- self.log("error", path="clock", error=repr(exc)[:500])- self._end("error")- self._release_watches()- return- for w in list(self.watches):- w.rec.step()- if self.done or len(self.episode) >= w.end:- self._complete_watch(w)- if not self.done and len(self.episode) % self.ckpt_every == 0:- self._checkpoint("tick")-- # ------------------------------------------------------------------ requests (simulator thread)- def dispatch(self, job: _Job) -> None:- """Serve one queued request; a watch finishes later, from tick()."""- try:- self.first_request()- path = urlparse(job.path).path- if job.method == "GET" and path in ("/spec", "/status"):- return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))- routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}- if self.reset_budget:- routes["/reset"] = self._rt_reset- if job.method != "POST" or path not in routes:- return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))- if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:- return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))- try:- meta, parts = self._decode(job.body)- except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error- return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))- try:- routes[path](meta, parts, job)- except Refused as exc: # 400 (bad_request), 409: nothing changed- job.finish("raw", *self._reply_json(exc.code, exc.body))- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- self.log("error", path=path, error=repr(exc)[:500])- self._end("error")- self._release_watches()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))- except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever- traceback.print_exc()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))-- def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:- """2.1's _view, plus cameras = [] for no images."""- none = isinstance(meta.get("cameras"), list) and not meta["cameras"]- cams, fps, video, depth = self._view(meta, default_video)- return ([], fps, "none", False) if none else (cams, fps, video, depth)-- def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:- """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the- plan's last row (held), else -- nothing planned, or a step already run -- the current command."""- i = step - nxt- if i < 0 or not self.plan:- return np.asarray(self.sim.command(typ), dtype=np.float64)- return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)-- def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:- if self.done or self.stopping:- raise self._conflict()- nxt = len(self.episode)- with bad_request():- typ = meta.get("type")- if not isinstance(typ, str) or typ not in self.types:- raise ValueError(f"type must be one of {sorted(self.types)}")- try:- A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),- dtype=np.float64)- except (TypeError, ValueError, KeyError):- raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None- dim = int(self.types[typ]["dim"])- if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:- raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")- if not np.all(np.isfinite(A)):- raise ValueError("actions contain NaN or inf")- cdt = self.sim.control_dt- try:- dt = cdt if meta.get("dt") is None else float(meta["dt"])- except (TypeError, ValueError):- raise ValueError("dt must be a number of seconds") from None- k = round(dt / cdt) if np.isfinite(dt / cdt) else 0- if k < 1 or abs(k * cdt - dt) > 1e-9:- raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")- if k > 1 and not self.sim.interpolable(typ):- raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")- if len(A) * k * cdt > self.max_chunk_s + 1e-9:- raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "- f"more than max_chunk_s = {self.max_chunk_s:g}")- interp = meta.get("interpolate", "linear")- if interp not in ("linear", "hold"):- raise ValueError("interpolate must be linear or hold")- at = meta.get("at")- if at is not None:- if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:- raise ValueError("at must be a control step of this episode (an integer >= 0), "- "or null for the next one")- at = int(at)- if at - nxt > self.max_ahead:- raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "- f"of the next one ({nxt})")- at = nxt if at is None else at- cams, fps, video, depth = self._view_rt(meta, "none")- rows, clipped = self.sim.clip(typ, A)- prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`- steps = []- for row in rows:- for j in range(1, k + 1):- steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))- prev = row- late = max(0, nxt - at)- keep = at - nxt if at > nxt else 0- self.calls += 1- self.rows += len(A)- head = list(self.plan)[:keep]- if len(head) < keep: # a gap before `at`: the target in force then is held through it- fill = self._planned(at - 1, nxt, typ)- head += [(typ, fill, -1)] * (keep - len(head))- self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])- self.late_steps += min(late, len(steps))- self._start_clock()- rec = Recorder(self.sim, cams, fps, depth, "none")- rec.start()- rec.finish()- out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,- "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,- "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}- self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)- job.finish("rec", out, rec)-- def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:- with bad_request():- try:- seconds = float(meta.get("seconds") or 0.0)- except (TypeError, ValueError):- raise ValueError("seconds must be a number") from None- if not 0.0 <= seconds <= self.max_watch_s:- raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")- cams, fps, video, depth = self._view_rt(meta, "frames")- n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0- rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video- self._frame_budget(rec, n)- self.calls += 1- rec.start()- if not n:- rec.finish()- out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}- self._save_call(out, None)- return job.finish("rec", out, rec)- self._start_clock()- self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))-- def _complete_watch(self, w: _Watch) -> None:- self.watches.remove(w)- w.rec.finish()- ran = max(0, min(len(self.episode), w.end) - w.start)- out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),- "control_steps_watch": ran, "ok": True}- self._save_call(out, None)- w.job.finish("rec", out, w.rec)-- def _release_watches(self) -> None:- for w in list(self.watches):- self._complete_watch(w)-- def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:- self._release_watches()- out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)- self._stop_clock("reset")- job.finish("rcb1", {**out, **self._realtime_counters()}, prts)-- # ------------------------------------------------------------------ the loop (simulator thread) and HTTP- def poll(self) -> None:- """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""- if self.stopping and not self.finalized:- if not self.done:- self._end("timeout")- self._release_watches()- self._finalize()-- def loop(self, until: threading.Event | None = None) -> None:- """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""- while until is None or not until.is_set():- self.poll()- wait = self.due_in(time.monotonic())- if wait is not None and wait <= 0:- self.tick()- continue- try:- job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))- except queue.Empty:- continue- self.dispatch(job)-- def submit_job(self, job: _Job) -> tuple[int, str, bytes]:- """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""- self.q.put(job)- job.done.wait()- kind, *rest = job.reply- if kind == "raw":- return tuple(rest)- meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread- return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)-- def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:- svc = self-- class Handler(BaseHTTPRequestHandler):- timeout = 120-- def log_message(self, *a):- pass-- def _send(self, code: int, ctype: str, body: bytes) -> None:- self.send_response(code)- self.send_header("Content-Type", ctype)- self.send_header("Content-Length", str(len(body)))- self.end_headers()- self.wfile.write(body)-- def do_GET(self):- self._send(*svc.submit_job(_Job("GET", self.path, b"")))-- def do_POST(self):- try:- n = int(self.headers.get("Content-Length") or 0)- except ValueError:- return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))- if n > MAX_BODY: # refused unread: drop the connection with it- self.close_connection = True- return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))- self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))-- class Server(ThreadingHTTPServer):- daemon_threads = True-- return Server(address, Handler)-- def serve(self, port: int = 8000) -> None:- import signal- signal.signal(signal.SIGUSR1, self.on_sigusr1)- srv = self.bind(port)- threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()- print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)- self.loop()---def main(build, port: int = 8000) -> None:- """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of- Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""- rcb_service.Service = RealtimeService- rcb_service.PROTOCOL = PROTOCOL- rcb_service.main(build, port)---if __name__ == "__main__":- sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))- import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import-- main(server.build)diff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/README.md b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/README.mdsimilarity index 82%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/README.mdrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/README.mdindex a6e2264d70..c8b0fb65ed 100644--- a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/README.md+++ b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-manip-bookshelf-simple-i00-realtime+# humanoidbench-manip-bookshelf-simple-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-manip-bookshelf-simple-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1hand` || Family / instance | `bookshelf_simple` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-manip-bookshelf-simple-i00-privileged`: the same frozen instance and the same success rule asit and as `humanoidbench-manip-bookshelf-simple-i00-standard`, but time does not stop. Once an episode's clockstarts (its first `submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the@@ -56,8 +56,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no object positions, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no object positions, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 61 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 90%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/docker-compose.yamlindex 03aa40ec2c..02910ef917 100644--- a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/rcb_client.pynew file mode 100644index 0000000000..f2b0559bf9--- /dev/null+++ b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/rcb_client.py@@ -0,0 +1,209 @@+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+PYTHONPATH).++ from rcb_client import Robot, EpisodeOver+ r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready++TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or+get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x+spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a+plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out+the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.++Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):+ r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"+ ("joint_position": dim, layout (actuator order), lower/upper, range_rad,+ units, semantics), "control_dt" (0.02 s), "max_control_steps" (the+ episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:+ rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"+ (joints, axes, positions), "imu", "links" (masses, centres of mass, the+ feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"+ r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,+ planned_steps, lag_s (and episode, resets_left with a reset budget)+ r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")+ POST /observe. seconds=0: the robot now, at once. seconds>0 (at most+ spec()["max_watch_s"]): waits while that much robot time runs (starting the+ clock if it is not running) and returns what the cameras saw; it never+ changes the plan+ r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,+ video="none", include_depth=False, note="")+ POST /submit: answers at once with the robot as it is now (the rows+ have not run yet). actions (T, dim): rows dt apart (a multiple of+ control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's+ target is the row (interpolate="hold") or a straight line from the previous+ row's ("linear", starting from the target in force just before `at`). They+ become the plan from control step `at` (an episode step number; None = the+ next step), replacing whatever was planned from `at` on and keeping what is+ planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows+ for steps already gone when the request arrives are dropped+ (meta["late_steps"]). The first submit of an episode starts its clock+ r.reset(cameras=None, include_depth=False)+ POST /reset, only when spec()["budget"]["reset"] > 0: abandons the+ running episode and starts a new one, paused, from the same starting state;+ only the last episode is graded, and a success ends the trial++cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the+fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),+"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth+for the cameras that have it.++Result (what get_obs / submit / reset return):+ res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),+ sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |+ "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,+ lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]+ (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,+ resets_left+ res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])+ res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes+ res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)+ res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s+ of a watch, one at its end+ res.done the episode is over+ res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy++Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an+unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over+(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this+run (404).+"""+from __future__ import annotations++import json+import os+import pathlib+import time++import numpy as np+import requests++import rcb1++PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)+MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused+++class EpisodeOver(RuntimeError):+ pass+++class Result:+ """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""++ def __init__(self, meta: dict, parts: dict):+ self.meta, self.raw = meta, parts+ self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None+ for name, p in parts.items():+ kind, data = p["kind"], p["data"]+ if name.startswith("video/") and kind == "mp4":+ self.video[name[6:]] = data+ elif name.startswith("frames/"):+ self.frames[name[7:]] = data+ elif name.startswith("last/"):+ self.last[name[5:]] = data+ elif name.startswith("depth/"):+ self.depth[name[6:]] = data+ elif name == "proprio":+ self.proprio = data++ @property+ def done(self) -> bool:+ return bool(self.meta.get("done"))++ def save(self, out_dir: str) -> str:+ from PIL import Image+ out = pathlib.Path(out_dir)+ out.mkdir(parents=True, exist_ok=True)+ (out / "meta.json").write_text(json.dumps(self.meta, indent=1))+ for cam, b in self.video.items():+ (out / f"{cam}.mp4").write_bytes(b)+ for cam, img in self.last.items():+ (out / "last").mkdir(exist_ok=True)+ Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)+ for cam, arr in self.frames.items():+ d = out / "frames" / cam+ d.mkdir(parents=True, exist_ok=True)+ for i, f in enumerate(arr):+ Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)+ for cam, arr in self.depth.items():+ np.save(out / f"depth_{cam}.npy", arr)+ if self.proprio is not None:+ np.save(out / "proprio.npy", self.proprio)+ return str(out)+++class Robot:+ def __init__(self, base: str | None = None, timeout: float = 600.0):+ candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]+ t0, last = time.time(), None+ while time.time() - t0 < timeout:+ for b in candidates:+ try:+ r = requests.get(b + "/spec", timeout=(5, 60))+ if r.status_code == 503:+ last = "warming up"+ continue+ r.raise_for_status()+ self.base, self._spec = b, r.json()+ if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):+ raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")+ return+ except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:+ last = e # not up yet (the scene takes about a minute), or busy+ time.sleep(2)+ raise RuntimeError(f"sim service unavailable: {last}")++ def spec(self) -> dict:+ return self._spec++ def status(self) -> dict:+ r = requests.get(self.base + "/status", timeout=(5, 60))+ r.raise_for_status()+ return r.json()++ def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,+ video: str = "frames") -> Result:+ meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),+ "include_depth": bool(include_depth), "video": video}+ return self._post("/observe", meta, [], timeout=900)++ def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",+ at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",+ include_depth: bool = False, note: str = "") -> Result:+ a = np.asarray(actions, dtype=np.float64)+ if a.ndim == 1:+ a = a[None, :]+ if not np.all(np.isfinite(a)):+ raise ValueError("actions contain NaN or inf")+ meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,+ "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),+ "video": video,+ "include_depth": bool(include_depth), "note": note}+ return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)++ def reset(self, cameras=None, include_depth: bool = False) -> Result:+ return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)++ def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:+ body = rcb1.encode(meta, parts)+ if len(body) > MAX_BODY: # the service would close the connection on it unread+ raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")+ r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},+ timeout=(5, timeout))+ if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:+ return Result(*rcb1.decode(r.content))+ try:+ err = r.json()+ except ValueError:+ err = {"error": r.text[:300]}+ msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")+ if r.status_code == 409:+ raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")+ + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))+ if r.status_code in (400, 413):+ raise ValueError(msg)+ raise RuntimeError(msg)+++def _names(cameras) -> list[str] | None:+ return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb_realtime.pynew file mode 100644index 0000000000..e23f8cbd2c--- /dev/null+++ b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -0,0 +1,480 @@+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.++It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery+and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:++- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its+ first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then+ on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,+ until the episode ends; the clock stops there.+- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`+ or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the+ next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`+ loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order+ hold): a slow agent's robot keeps its last servo targets.+- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)+ and returns what the cameras saw. It never changes the plan.+- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.++Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's+rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was+submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):+ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line+per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:+call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.++Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in+between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting+request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than+MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.++A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.+A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a+crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the+trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends+the graded episode (reason "error") sets infra_error in final.json too.+"""+from __future__ import annotations++import collections+import json+import os+import queue+import sys+import threading+import time+import traceback+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from urllib.parse import urlparse++import numpy as np++import rcb_service+from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request++PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up+CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)+++class _Job:+ """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""++ def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):+ self.method, self.path, self.body, self.content_length = method, path, body, content_length+ self.done = threading.Event()+ self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)++ def finish(self, *reply) -> None:+ self.reply = reply+ self.done.set()+++class _Watch:+ """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""++ def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):+ self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id+ self.start, self.end = start, start + steps+++class RealtimeService(Service):+ """rcb_service.Service with the realtime clock (module docstring)."""++ def __init__(self, sim, *, slowdown: float | None = None, **kw):+ self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)+ if not self.slowdown > 0:+ raise ValueError("RCB_SLOWDOWN must be > 0")+ self.wall_dt = sim.control_dt * self.slowdown+ self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be+ self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step+ self.running = False+ self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t+ self.watches: list[_Watch] = []+ self.q: queue.Queue = queue.Queue()+ self.max_lag = self.lag = 0.0+ self.stalls = self.held_steps = self.late_steps = 0+ self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk+ super().__init__(sim, **kw)+ self.ticks_path = os.path.join(self.data, "ticks.f64")+ self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))++ # ------------------------------------------------------------------ spec, status, evidence+ def _spec(self) -> dict:+ spec = super()._spec()+ spec["protocol"] = PROTOCOL+ cdt = self.sim.control_dt+ spec["timing"] = {+ "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,+ "wall_s_per_control_step": cdt * self.slowdown,+ "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "+ "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "+ "you do, until the episode ends",+ "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "+ "`at` (default: the next one), replacing what was planned from there; rows for steps already "+ "gone are dropped (meta.late_steps); after the plan's last row that target is held",+ "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "+ "the cameras saw; never changes the plan",+ "max_ahead_steps": self.max_ahead,+ "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}+ return spec++ def _realtime_counters(self) -> dict:+ return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}++ def status(self) -> dict:+ return {**super().status(), **self._realtime_counters()}++ def meta(self) -> dict:+ return {**super().meta(), **self._realtime_counters()}++ def _timing_record(self) -> dict:+ return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),+ "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}++ def _write_final(self) -> None:+ self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's+ if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)+ self.verdict["infra_error"] = 1+ else: # (an episode that erred and was reset away is not graded)+ self.verdict.pop("infra_error", None)+ super()._write_final()++ def _checkpoint(self, ev: str, **kw) -> None:+ if self._ticks: # the tick log first: a crash in between leaves it too long+ with open(self.ticks_path, "ab") as f:+ f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())+ self._ticks = []+ super()._checkpoint(ev, **kw)++ def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:+ keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",+ "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")+ line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}+ with open(os.path.join(self.data, "calls.jsonl"), "a") as f:+ f.write(json.dumps(line) + "\n")+ if actions is not None:+ rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)+ with open(os.path.join(self.data, "submitted.f64"), "ab") as f:+ f.write(np.asarray(rows, dtype=np.float64).tobytes())++ # ------------------------------------------------------------------ start, clock+ def start(self) -> None:+ super().start()+ rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)+ n = self.journal_rows+ if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too+ rows[: 2 * n].tofile(self.ticks_path)+ elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)+ np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)+ self.running = False # a resumed episode waits, paused, for the agent's next call++ def _start_clock(self) -> None:+ if not self.running and not self.done and not self.stopping:+ self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)+ self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)++ def _stop_clock(self, why: str) -> None:+ if self.running:+ self.running = False+ self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))+ self.plan.clear()++ def _end(self, reason: str) -> None:+ super()._end(reason)+ self._stop_clock(reason) # pending watches are completed by whoever ended it, after this++ def due_in(self, now: float) -> float | None:+ """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""+ if not self.running or self.done or self.stopping:+ return None+ return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now++ def tick(self) -> None:+ """One control step of the clock: the plan's row for it, or the held target."""+ now = time.monotonic()+ lag = -(self.due_in(now) or 0.0)+ if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt+ self.stalls += 1+ self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))+ self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0+ self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)+ if self.plan:+ typ, row, src = self.plan.popleft()+ else:+ (typ, row), src = self.sim.hold(), -1+ self.held_steps += 1+ before = self.journal_rows+ self._ticks.append((time.time(), float(src)))+ try:+ self._step(typ, row)+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ if self.journal_rows == before:+ self._ticks.pop()+ self.log("error", path="clock", error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ return+ for w in list(self.watches):+ w.rec.step()+ if self.done or len(self.episode) >= w.end:+ self._complete_watch(w)+ if not self.done and len(self.episode) % self.ckpt_every == 0:+ self._checkpoint("tick")++ # ------------------------------------------------------------------ requests (simulator thread)+ def dispatch(self, job: _Job) -> None:+ """Serve one queued request; a watch finishes later, from tick()."""+ try:+ self.first_request()+ path = urlparse(job.path).path+ if job.method == "GET" and path in ("/spec", "/status"):+ return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))+ routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}+ if self.reset_budget:+ routes["/reset"] = self._rt_reset+ if job.method != "POST" or path not in routes:+ return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))+ if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:+ return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))+ try:+ meta, parts = self._decode(job.body)+ except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error+ return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))+ try:+ routes[path](meta, parts, job)+ except Refused as exc: # 400 (bad_request), 409: nothing changed+ job.finish("raw", *self._reply_json(exc.code, exc.body))+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ self.log("error", path=path, error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))+ except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever+ traceback.print_exc()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))++ def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:+ """2.1's _view, plus cameras = [] for no images."""+ none = isinstance(meta.get("cameras"), list) and not meta["cameras"]+ cams, fps, video, depth = self._view(meta, default_video)+ return ([], fps, "none", False) if none else (cams, fps, video, depth)++ def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:+ """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the+ plan's last row (held), else -- nothing planned, or a step already run -- the current command."""+ i = step - nxt+ if i < 0 or not self.plan:+ return np.asarray(self.sim.command(typ), dtype=np.float64)+ return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)++ def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:+ if self.done or self.stopping:+ raise self._conflict()+ nxt = len(self.episode)+ with bad_request():+ typ = meta.get("type")+ if not isinstance(typ, str) or typ not in self.types:+ raise ValueError(f"type must be one of {sorted(self.types)}")+ try:+ A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),+ dtype=np.float64)+ except (TypeError, ValueError, KeyError):+ raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None+ dim = int(self.types[typ]["dim"])+ if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:+ raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")+ if not np.all(np.isfinite(A)):+ raise ValueError("actions contain NaN or inf")+ cdt = self.sim.control_dt+ try:+ dt = cdt if meta.get("dt") is None else float(meta["dt"])+ except (TypeError, ValueError):+ raise ValueError("dt must be a number of seconds") from None+ k = round(dt / cdt) if np.isfinite(dt / cdt) else 0+ if k < 1 or abs(k * cdt - dt) > 1e-9:+ raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")+ if k > 1 and not self.sim.interpolable(typ):+ raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")+ if len(A) * k * cdt > self.max_chunk_s + 1e-9:+ raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "+ f"more than max_chunk_s = {self.max_chunk_s:g}")+ interp = meta.get("interpolate", "linear")+ if interp not in ("linear", "hold"):+ raise ValueError("interpolate must be linear or hold")+ at = meta.get("at")+ if at is not None:+ if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:+ raise ValueError("at must be a control step of this episode (an integer >= 0), "+ "or null for the next one")+ at = int(at)+ if at - nxt > self.max_ahead:+ raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "+ f"of the next one ({nxt})")+ at = nxt if at is None else at+ cams, fps, video, depth = self._view_rt(meta, "none")+ rows, clipped = self.sim.clip(typ, A)+ prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`+ steps = []+ for row in rows:+ for j in range(1, k + 1):+ steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))+ prev = row+ late = max(0, nxt - at)+ keep = at - nxt if at > nxt else 0+ self.calls += 1+ self.rows += len(A)+ head = list(self.plan)[:keep]+ if len(head) < keep: # a gap before `at`: the target in force then is held through it+ fill = self._planned(at - 1, nxt, typ)+ head += [(typ, fill, -1)] * (keep - len(head))+ self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])+ self.late_steps += min(late, len(steps))+ self._start_clock()+ rec = Recorder(self.sim, cams, fps, depth, "none")+ rec.start()+ rec.finish()+ out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,+ "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,+ "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}+ self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)+ job.finish("rec", out, rec)++ def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:+ with bad_request():+ try:+ seconds = float(meta.get("seconds") or 0.0)+ except (TypeError, ValueError):+ raise ValueError("seconds must be a number") from None+ if not 0.0 <= seconds <= self.max_watch_s:+ raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")+ cams, fps, video, depth = self._view_rt(meta, "frames")+ n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0+ rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video+ self._frame_budget(rec, n)+ self.calls += 1+ rec.start()+ if not n:+ rec.finish()+ out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}+ self._save_call(out, None)+ return job.finish("rec", out, rec)+ self._start_clock()+ self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))++ def _complete_watch(self, w: _Watch) -> None:+ self.watches.remove(w)+ w.rec.finish()+ ran = max(0, min(len(self.episode), w.end) - w.start)+ out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),+ "control_steps_watch": ran, "ok": True}+ self._save_call(out, None)+ w.job.finish("rec", out, w.rec)++ def _release_watches(self) -> None:+ for w in list(self.watches):+ self._complete_watch(w)++ def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:+ self._release_watches()+ out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)+ self._stop_clock("reset")+ job.finish("rcb1", {**out, **self._realtime_counters()}, prts)++ # ------------------------------------------------------------------ the loop (simulator thread) and HTTP+ def poll(self) -> None:+ """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""+ if self.stopping and not self.finalized:+ if not self.done:+ self._end("timeout")+ self._release_watches()+ self._finalize()++ def loop(self, until: threading.Event | None = None) -> None:+ """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""+ while until is None or not until.is_set():+ self.poll()+ wait = self.due_in(time.monotonic())+ if wait is not None and wait <= 0:+ self.tick()+ continue+ try:+ job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))+ except queue.Empty:+ continue+ self.dispatch(job)++ def submit_job(self, job: _Job) -> tuple[int, str, bytes]:+ """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""+ self.q.put(job)+ job.done.wait()+ kind, *rest = job.reply+ if kind == "raw":+ return tuple(rest)+ meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread+ return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)++ def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:+ svc = self++ class Handler(BaseHTTPRequestHandler):+ timeout = 120++ def log_message(self, *a):+ pass++ def _send(self, code: int, ctype: str, body: bytes) -> None:+ self.send_response(code)+ self.send_header("Content-Type", ctype)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_GET(self):+ self._send(*svc.submit_job(_Job("GET", self.path, b"")))++ def do_POST(self):+ try:+ n = int(self.headers.get("Content-Length") or 0)+ except ValueError:+ return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))+ if n > MAX_BODY: # refused unread: drop the connection with it+ self.close_connection = True+ return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))+ self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))++ class Server(ThreadingHTTPServer):+ daemon_threads = True++ return Server(address, Handler)++ def serve(self, port: int = 8000) -> None:+ import signal+ signal.signal(signal.SIGUSR1, self.on_sigusr1)+ srv = self.bind(port)+ threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()+ print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)+ self.loop()+++def main(build, port: int = 8000) -> None:+ """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of+ Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""+ rcb_service.Service = RealtimeService+ rcb_service.PROTOCOL = PROTOCOL+ rcb_service.main(build, port)+++if __name__ == "__main__":+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+ import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import++ main(server.build)diff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/instruction.md b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/instruction.mdrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/instruction.mdindex 341169e9e7..54886746fb 100644--- a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/instruction.md+++ b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# bookshelf_simple — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# bookshelf_simple — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid with two Shadow dexterous hands (HumanoidBench / MuJoCo) **as ifit were real hardware**: the simulator runs in a service you cannot open, and you reach it only through a@@ -76,7 +76,7 @@ cut into five plateaus and stalling on the first caps the episode low. The servithe pass mark is reached (`reason = "success"`), and a fresh replay of the recorded episode, the servo targetsthe robot actually executed at each control step, must reach it as well and end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/task.toml b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/task.tomlnew file mode 100644index 0000000000..063e6a1fa9--- /dev/null+++ b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/task.toml@@ -0,0 +1,68 @@+# Standard-realtime task of humanoidbench-manip-bookshelf-simple-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# subset.toml or the template, not this.+schema_version = "1.3"+artifacts = [+ { source = "/data/final.json", service = "sim" },+ { source = "/data/episode.jsonl", service = "sim" },+ { source = "/data/trajectory.npz", service = "sim" },+ { source = "/data/trajectory_live.f64", service = "sim" },+ { source = "/data/replay.mp4", service = "sim" },+ { source = "/data/replay.last.png", service = "sim" },+ { source = "/data/ticks.f64", service = "sim" },+ { source = "/data/calls.jsonl", service = "sim" },+ { source = "/data/submitted.f64", service = "sim" },+ { source = "/data/startups.jsonl", service = "sim" },+ "/app/output/",+]++[task]+name = "robot-coding-bench/humanoidbench-manip-bookshelf-simple-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench bookshelf_simple on the same frozen instance as humanoidbench-manip-bookshelf-simple-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+authors = [{ name = "pingyue" }]+keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "standard-realtime-mode", "trajectory", "bookshelf_simple"]++[metadata]+author_name = "pingyue"+task_type = "trajectory"+simulator = "humanoidbench"+robot = "unitree-h1hand"+family = "bookshelf_simple"+mode = "standard-realtime"+protocol = "eai-standard/2.3"+instance_seed = 0+expert_time_estimate_hours = 10.0+oracle = "none"+oracle_evidence = "none known: no run has passed it, and published RL reaches 41% of the bar (paper 763, FastTD3 822, bar 2000); human review in PR #83"++[verifier]+timeout_sec = 900.0+environment_mode = "separate"+network_mode = "no-network"++# The agent's time is up: stop whatever it left running in its container. Harbor stops that container only after+# collecting from it (~12 s), and a script still running there would keep driving the robot past the deadline.+[[verifier.collect]]+service = "main"+command = "kill -9 -1 2>/dev/null; exit 0"+timeout_sec = 30++# Then the episode ends: SIGUSR1 to the service (re-sent every 30 s, in case it was restarting), which grades it.+[[verifier.collect]]+service = "sim"+command = "for i in $(seq 1 90); do [ -f /data/server.pid ] && break; sleep 2; done; for i in $(seq 1 400); do grep -q '\"finalized\": true' /data/final.json 2>/dev/null && break; [ $((i % 15)) -eq 1 ] && kill -USR1 $(cat /data/server.pid) 2>/dev/null; sleep 2; done; cat /data/final.json"+timeout_sec = 900++[agent]+timeout_sec = 3600.0+# The agent container holds no simulator and no reward, but upstream is a public commit: like every HumanoidBench+# task, the agent reaches the model APIs and nothing else (scripts/set_agent_egress.py HOSTS).+network_mode = "allowlist"+allowed_hosts = ["chatgpt.com", "*.chatgpt.com", "auth.openai.com", "api.openai.com", "*.openai.com", "api.anthropic.com", "*.anthropic.com", "openrouter.ai", "*.openai.azure.com", "*.cognitiveservices.azure.com"]++[environment]+build_timeout_sec = 1800.0+cpus = 4+memory_mb = 16384++[verifier.environment]+build_timeout_sec = 600.0diff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/tests/grade.py b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/tests/test.sh b/tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-manip-bookshelf-simple-i00-realtime/tests/test.shrename to tasks/humanoidbench-manip-bookshelf-simple-i00-standard-realtime/tests/test.shdiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-cube-i00-realtime/environment/rcb_client.pydeleted file mode 100644index 8e057bb6e9..0000000000--- a/tasks/humanoidbench-manip-cube-i00-realtime/environment/rcb_client.py+++ /dev/null@@ -1,209 +0,0 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on-PYTHONPATH).-- from rcb_client import Robot, EpisodeOver- r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready--TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or-get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x-spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a-plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out-the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.--Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):- r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"- ("joint_position": dim, layout (actuator order), lower/upper, range_rad,- units, semantics), "control_dt" (0.02 s), "max_control_steps" (the- episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:- rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"- (joints, axes, positions), "imu", "links" (masses, centres of mass, the- feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"- r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,- planned_steps, lag_s (and episode, resets_left with a reset budget)- r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")- POST /observe. seconds=0: the robot now, at once. seconds>0 (at most- spec()["max_watch_s"]): waits while that much robot time runs (starting the- clock if it is not running) and returns what the cameras saw; it never- changes the plan- r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,- video="none", include_depth=False, note="")- POST /submit: answers at once with the robot as it is now (the rows- have not run yet). actions (T, dim): rows dt apart (a multiple of- control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's- target is the row (interpolate="hold") or a straight line from the previous- row's ("linear", starting from the target in force just before `at`). They- become the plan from control step `at` (an episode step number; None = the- next step), replacing whatever was planned from `at` on and keeping what is- planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows- for steps already gone when the request arrives are dropped- (meta["late_steps"]). The first submit of an episode starts its clock- r.reset(cameras=None, include_depth=False)- POST /reset, only when spec()["budget"]["reset"] > 0: abandons the- running episode and starts a new one, paused, from the same starting state;- only the last episode is graded, and a success ends the trial--cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the-fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),-"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth-for the cameras that have it.--Result (what get_obs / submit / reset return):- res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),- sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |- "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,- lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]- (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,- resets_left- res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])- res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes- res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)- res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s- of a watch, one at its end- res.done the episode is over- res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy--Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an-unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over-(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this-run (404).-"""-from __future__ import annotations--import json-import os-import pathlib-import time--import numpy as np-import requests--import rcb1--PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)-MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused---class EpisodeOver(RuntimeError):- pass---class Result:- """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""-- def __init__(self, meta: dict, parts: dict):- self.meta, self.raw = meta, parts- self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None- for name, p in parts.items():- kind, data = p["kind"], p["data"]- if name.startswith("video/") and kind == "mp4":- self.video[name[6:]] = data- elif name.startswith("frames/"):- self.frames[name[7:]] = data- elif name.startswith("last/"):- self.last[name[5:]] = data- elif name.startswith("depth/"):- self.depth[name[6:]] = data- elif name == "proprio":- self.proprio = data-- @property- def done(self) -> bool:- return bool(self.meta.get("done"))-- def save(self, out_dir: str) -> str:- from PIL import Image- out = pathlib.Path(out_dir)- out.mkdir(parents=True, exist_ok=True)- (out / "meta.json").write_text(json.dumps(self.meta, indent=1))- for cam, b in self.video.items():- (out / f"{cam}.mp4").write_bytes(b)- for cam, img in self.last.items():- (out / "last").mkdir(exist_ok=True)- Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)- for cam, arr in self.frames.items():- d = out / "frames" / cam- d.mkdir(parents=True, exist_ok=True)- for i, f in enumerate(arr):- Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)- for cam, arr in self.depth.items():- np.save(out / f"depth_{cam}.npy", arr)- if self.proprio is not None:- np.save(out / "proprio.npy", self.proprio)- return str(out)---class Robot:- def __init__(self, base: str | None = None, timeout: float = 600.0):- candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]- t0, last = time.time(), None- while time.time() - t0 < timeout:- for b in candidates:- try:- r = requests.get(b + "/spec", timeout=(5, 60))- if r.status_code == 503:- last = "warming up"- continue- r.raise_for_status()- self.base, self._spec = b, r.json()- if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):- raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")- return- except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:- last = e # not up yet (the scene takes about a minute), or busy- time.sleep(2)- raise RuntimeError(f"sim service unavailable: {last}")-- def spec(self) -> dict:- return self._spec-- def status(self) -> dict:- r = requests.get(self.base + "/status", timeout=(5, 60))- r.raise_for_status()- return r.json()-- def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,- video: str = "frames") -> Result:- meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),- "include_depth": bool(include_depth), "video": video}- return self._post("/observe", meta, [], timeout=900)-- def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",- at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",- include_depth: bool = False, note: str = "") -> Result:- a = np.asarray(actions, dtype=np.float64)- if a.ndim == 1:- a = a[None, :]- if not np.all(np.isfinite(a)):- raise ValueError("actions contain NaN or inf")- meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,- "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),- "video": video,- "include_depth": bool(include_depth), "note": note}- return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)-- def reset(self, cameras=None, include_depth: bool = False) -> Result:- return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)-- def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:- body = rcb1.encode(meta, parts)- if len(body) > MAX_BODY: # the service would close the connection on it unread- raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")- r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},- timeout=(5, timeout))- if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:- return Result(*rcb1.decode(r.content))- try:- err = r.json()- except ValueError:- err = {"error": r.text[:300]}- msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")- if r.status_code == 409:- raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")- + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))- if r.status_code in (400, 413):- raise ValueError(msg)- raise RuntimeError(msg)---def _names(cameras) -> list[str] | None:- return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb_realtime.pydeleted file mode 100644index 410284bf27..0000000000--- a/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb_realtime.py+++ /dev/null@@ -1,479 +0,0 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.--It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery-and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:--- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its- first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then- on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,- until the episode ends; the clock stops there.-- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`- or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the- next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`- loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order- hold): a slow agent's robot keeps its last servo targets.-- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)- and returns what the cameras saw. It never changes the plan.-- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.--Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's-rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was-submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):-ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line-per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:-call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.--Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in-between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting-request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than-MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.--A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.-A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a-crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the-trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends-the graded episode (reason "error") sets infra_error in final.json too.-"""-from __future__ import annotations--import collections-import json-import os-import queue-import sys-import threading-import time-import traceback-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer-from urllib.parse import urlparse--import numpy as np--import rcb_service-from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request--PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1-MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up-CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)---class _Job:- """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""-- def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):- self.method, self.path, self.body, self.content_length = method, path, body, content_length- self.done = threading.Event()- self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)-- def finish(self, *reply) -> None:- self.reply = reply- self.done.set()---class _Watch:- """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""-- def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):- self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id- self.start, self.end = start, start + steps---class RealtimeService(Service):- """rcb_service.Service with the realtime clock (module docstring)."""-- def __init__(self, sim, *, slowdown: float | None = None, **kw):- self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)- if not self.slowdown > 0:- raise ValueError("RCB_SLOWDOWN must be > 0")- self.wall_dt = sim.control_dt * self.slowdown- self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be- self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step- self.running = False- self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t- self.watches: list[_Watch] = []- self.q: queue.Queue = queue.Queue()- self.max_lag = self.lag = 0.0- self.stalls = self.held_steps = self.late_steps = 0- self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk- super().__init__(sim, **kw)- self.ticks_path = os.path.join(self.data, "ticks.f64")- self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))-- # ------------------------------------------------------------------ spec, status, evidence- def _spec(self) -> dict:- spec = super()._spec()- spec["protocol"] = PROTOCOL- cdt = self.sim.control_dt- spec["timing"] = {- "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,- "wall_s_per_control_step": cdt * self.slowdown,- "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "- "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "- "you do, until the episode ends",- "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "- "`at` (default: the next one), replacing what was planned from there; rows for steps already "- "gone are dropped (meta.late_steps); after the plan's last row that target is held",- "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "- "the cameras saw; never changes the plan",- "max_ahead_steps": self.max_ahead,- "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}- return spec-- def _realtime_counters(self) -> dict:- return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}-- def status(self) -> dict:- return {**super().status(), **self._realtime_counters()}-- def meta(self) -> dict:- return {**super().meta(), **self._realtime_counters()}-- def _timing_record(self) -> dict:- return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),- "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}-- def _write_final(self) -> None:- self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's- if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)- self.verdict["infra_error"] = 1- else: # (an episode that erred and was reset away is not graded)- self.verdict.pop("infra_error", None)- super()._write_final()-- def _checkpoint(self, ev: str, **kw) -> None:- if self._ticks: # the tick log first: a crash in between leaves it too long- with open(self.ticks_path, "ab") as f:- f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())- self._ticks = []- super()._checkpoint(ev, **kw)-- def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:- keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",- "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")- line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}- with open(os.path.join(self.data, "calls.jsonl"), "a") as f:- f.write(json.dumps(line) + "\n")- if actions is not None:- rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)- with open(os.path.join(self.data, "submitted.f64"), "ab") as f:- f.write(np.asarray(rows, dtype=np.float64).tobytes())-- # ------------------------------------------------------------------ start, clock- def start(self) -> None:- super().start()- rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)- n = self.journal_rows- if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too- rows[: 2 * n].tofile(self.ticks_path)- elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)- np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)- self.running = False # a resumed episode waits, paused, for the agent's next call-- def _start_clock(self) -> None:- if not self.running and not self.done and not self.stopping:- self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)- self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)-- def _stop_clock(self, why: str) -> None:- if self.running:- self.running = False- self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))- self.plan.clear()-- def _end(self, reason: str) -> None:- super()._end(reason)- self._stop_clock(reason) # pending watches are completed by whoever ended it, after this-- def due_in(self, now: float) -> float | None:- """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""- if not self.running or self.done or self.stopping:- return None- return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now-- def tick(self) -> None:- """One control step of the clock: the plan's row for it, or the held target."""- now = time.monotonic()- lag = -(self.due_in(now) or 0.0)- if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt- self.stalls += 1- self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))- self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0- self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)- if self.plan:- typ, row, src = self.plan.popleft()- else:- (typ, row), src = self.sim.hold(), -1- self.held_steps += 1- before = self.journal_rows- self._ticks.append((time.time(), float(src)))- try:- self._step(typ, row)- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- if self.journal_rows == before:- self._ticks.pop()- self.log("error", path="clock", error=repr(exc)[:500])- self._end("error")- self._release_watches()- return- for w in list(self.watches):- w.rec.step()- if self.done or len(self.episode) >= w.end:- self._complete_watch(w)- if not self.done and len(self.episode) % self.ckpt_every == 0:- self._checkpoint("tick")-- # ------------------------------------------------------------------ requests (simulator thread)- def dispatch(self, job: _Job) -> None:- """Serve one queued request; a watch finishes later, from tick()."""- try:- self.first_request()- path = urlparse(job.path).path- if job.method == "GET" and path in ("/spec", "/status"):- return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))- routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}- if self.reset_budget:- routes["/reset"] = self._rt_reset- if job.method != "POST" or path not in routes:- return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))- if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:- return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))- try:- meta, parts = self._decode(job.body)- except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error- return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))- try:- routes[path](meta, parts, job)- except Refused as exc: # 400 (bad_request), 409: nothing changed- job.finish("raw", *self._reply_json(exc.code, exc.body))- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- self.log("error", path=path, error=repr(exc)[:500])- self._end("error")- self._release_watches()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))- except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever- traceback.print_exc()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))-- def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:- """2.1's _view, plus cameras = [] for no images."""- none = isinstance(meta.get("cameras"), list) and not meta["cameras"]- cams, fps, video, depth = self._view(meta, default_video)- return ([], fps, "none", False) if none else (cams, fps, video, depth)-- def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:- """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the- plan's last row (held), else -- nothing planned, or a step already run -- the current command."""- i = step - nxt- if i < 0 or not self.plan:- return np.asarray(self.sim.command(typ), dtype=np.float64)- return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)-- def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:- if self.done or self.stopping:- raise self._conflict()- nxt = len(self.episode)- with bad_request():- typ = meta.get("type")- if not isinstance(typ, str) or typ not in self.types:- raise ValueError(f"type must be one of {sorted(self.types)}")- try:- A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),- dtype=np.float64)- except (TypeError, ValueError, KeyError):- raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None- dim = int(self.types[typ]["dim"])- if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:- raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")- if not np.all(np.isfinite(A)):- raise ValueError("actions contain NaN or inf")- cdt = self.sim.control_dt- try:- dt = cdt if meta.get("dt") is None else float(meta["dt"])- except (TypeError, ValueError):- raise ValueError("dt must be a number of seconds") from None- k = round(dt / cdt) if np.isfinite(dt / cdt) else 0- if k < 1 or abs(k * cdt - dt) > 1e-9:- raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")- if k > 1 and not self.sim.interpolable(typ):- raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")- if len(A) * k * cdt > self.max_chunk_s + 1e-9:- raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "- f"more than max_chunk_s = {self.max_chunk_s:g}")- interp = meta.get("interpolate", "linear")- if interp not in ("linear", "hold"):- raise ValueError("interpolate must be linear or hold")- at = meta.get("at")- if at is not None:- if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:- raise ValueError("at must be a control step of this episode (an integer >= 0), "- "or null for the next one")- at = int(at)- if at - nxt > self.max_ahead:- raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "- f"of the next one ({nxt})")- at = nxt if at is None else at- cams, fps, video, depth = self._view_rt(meta, "none")- rows, clipped = self.sim.clip(typ, A)- prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`- steps = []- for row in rows:- for j in range(1, k + 1):- steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))- prev = row- late = max(0, nxt - at)- keep = at - nxt if at > nxt else 0- self.calls += 1- self.rows += len(A)- head = list(self.plan)[:keep]- if len(head) < keep: # a gap before `at`: the target in force then is held through it- fill = self._planned(at - 1, nxt, typ)- head += [(typ, fill, -1)] * (keep - len(head))- self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])- self.late_steps += min(late, len(steps))- self._start_clock()- rec = Recorder(self.sim, cams, fps, depth, "none")- rec.start()- rec.finish()- out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,- "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,- "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}- self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)- job.finish("rec", out, rec)-- def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:- with bad_request():- try:- seconds = float(meta.get("seconds") or 0.0)- except (TypeError, ValueError):- raise ValueError("seconds must be a number") from None- if not 0.0 <= seconds <= self.max_watch_s:- raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")- cams, fps, video, depth = self._view_rt(meta, "frames")- n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0- rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video- self._frame_budget(rec, n)- self.calls += 1- rec.start()- if not n:- rec.finish()- out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}- self._save_call(out, None)- return job.finish("rec", out, rec)- self._start_clock()- self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))-- def _complete_watch(self, w: _Watch) -> None:- self.watches.remove(w)- w.rec.finish()- ran = max(0, min(len(self.episode), w.end) - w.start)- out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),- "control_steps_watch": ran, "ok": True}- self._save_call(out, None)- w.job.finish("rec", out, w.rec)-- def _release_watches(self) -> None:- for w in list(self.watches):- self._complete_watch(w)-- def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:- self._release_watches()- out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)- self._stop_clock("reset")- job.finish("rcb1", {**out, **self._realtime_counters()}, prts)-- # ------------------------------------------------------------------ the loop (simulator thread) and HTTP- def poll(self) -> None:- """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""- if self.stopping and not self.finalized:- if not self.done:- self._end("timeout")- self._release_watches()- self._finalize()-- def loop(self, until: threading.Event | None = None) -> None:- """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""- while until is None or not until.is_set():- self.poll()- wait = self.due_in(time.monotonic())- if wait is not None and wait <= 0:- self.tick()- continue- try:- job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))- except queue.Empty:- continue- self.dispatch(job)-- def submit_job(self, job: _Job) -> tuple[int, str, bytes]:- """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""- self.q.put(job)- job.done.wait()- kind, *rest = job.reply- if kind == "raw":- return tuple(rest)- meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread- return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)-- def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:- svc = self-- class Handler(BaseHTTPRequestHandler):- timeout = 120-- def log_message(self, *a):- pass-- def _send(self, code: int, ctype: str, body: bytes) -> None:- self.send_response(code)- self.send_header("Content-Type", ctype)- self.send_header("Content-Length", str(len(body)))- self.end_headers()- self.wfile.write(body)-- def do_GET(self):- self._send(*svc.submit_job(_Job("GET", self.path, b"")))-- def do_POST(self):- try:- n = int(self.headers.get("Content-Length") or 0)- except ValueError:- return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))- if n > MAX_BODY: # refused unread: drop the connection with it- self.close_connection = True- return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))- self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))-- class Server(ThreadingHTTPServer):- daemon_threads = True-- return Server(address, Handler)-- def serve(self, port: int = 8000) -> None:- import signal- signal.signal(signal.SIGUSR1, self.on_sigusr1)- srv = self.bind(port)- threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()- print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)- self.loop()---def main(build, port: int = 8000) -> None:- """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of- Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""- rcb_service.Service = RealtimeService- rcb_service.PROTOCOL = PROTOCOL- rcb_service.main(build, port)---if __name__ == "__main__":- sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))- import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import-- main(server.build)diff --git a/tasks/humanoidbench-manip-cube-i00-realtime/README.md b/tasks/humanoidbench-manip-cube-i00-standard-realtime/README.mdsimilarity index 82%rename from tasks/humanoidbench-manip-cube-i00-realtime/README.mdrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/README.mdindex 04b5b87757..8167d880d5 100644--- a/tasks/humanoidbench-manip-cube-i00-realtime/README.md+++ b/tasks/humanoidbench-manip-cube-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-manip-cube-i00-realtime+# humanoidbench-manip-cube-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-manip-cube-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1hand` || Family / instance | `cube` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-manip-cube-i00-privileged`: the same frozen instance and the same success rule as it and as`humanoidbench-manip-cube-i00-standard`, but time does not stop. Once an episode's clock starts (its first`submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the agent thinks, so@@ -56,8 +56,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no cube poses, no goal orientation, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no cube poses, no goal orientation, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 61 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 90%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/docker-compose.yamlindex abc57b26dd..46cece2bda 100644--- a/tasks/humanoidbench-manip-cube-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/rcb_client.pynew file mode 100644index 0000000000..f2b0559bf9--- /dev/null+++ b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/rcb_client.py@@ -0,0 +1,209 @@+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+PYTHONPATH).++ from rcb_client import Robot, EpisodeOver+ r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready++TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or+get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x+spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a+plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out+the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.++Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):+ r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"+ ("joint_position": dim, layout (actuator order), lower/upper, range_rad,+ units, semantics), "control_dt" (0.02 s), "max_control_steps" (the+ episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:+ rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"+ (joints, axes, positions), "imu", "links" (masses, centres of mass, the+ feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"+ r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,+ planned_steps, lag_s (and episode, resets_left with a reset budget)+ r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")+ POST /observe. seconds=0: the robot now, at once. seconds>0 (at most+ spec()["max_watch_s"]): waits while that much robot time runs (starting the+ clock if it is not running) and returns what the cameras saw; it never+ changes the plan+ r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,+ video="none", include_depth=False, note="")+ POST /submit: answers at once with the robot as it is now (the rows+ have not run yet). actions (T, dim): rows dt apart (a multiple of+ control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's+ target is the row (interpolate="hold") or a straight line from the previous+ row's ("linear", starting from the target in force just before `at`). They+ become the plan from control step `at` (an episode step number; None = the+ next step), replacing whatever was planned from `at` on and keeping what is+ planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows+ for steps already gone when the request arrives are dropped+ (meta["late_steps"]). The first submit of an episode starts its clock+ r.reset(cameras=None, include_depth=False)+ POST /reset, only when spec()["budget"]["reset"] > 0: abandons the+ running episode and starts a new one, paused, from the same starting state;+ only the last episode is graded, and a success ends the trial++cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the+fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),+"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth+for the cameras that have it.++Result (what get_obs / submit / reset return):+ res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),+ sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |+ "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,+ lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]+ (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,+ resets_left+ res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])+ res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes+ res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)+ res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s+ of a watch, one at its end+ res.done the episode is over+ res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy++Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an+unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over+(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this+run (404).+"""+from __future__ import annotations++import json+import os+import pathlib+import time++import numpy as np+import requests++import rcb1++PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)+MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused+++class EpisodeOver(RuntimeError):+ pass+++class Result:+ """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""++ def __init__(self, meta: dict, parts: dict):+ self.meta, self.raw = meta, parts+ self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None+ for name, p in parts.items():+ kind, data = p["kind"], p["data"]+ if name.startswith("video/") and kind == "mp4":+ self.video[name[6:]] = data+ elif name.startswith("frames/"):+ self.frames[name[7:]] = data+ elif name.startswith("last/"):+ self.last[name[5:]] = data+ elif name.startswith("depth/"):+ self.depth[name[6:]] = data+ elif name == "proprio":+ self.proprio = data++ @property+ def done(self) -> bool:+ return bool(self.meta.get("done"))++ def save(self, out_dir: str) -> str:+ from PIL import Image+ out = pathlib.Path(out_dir)+ out.mkdir(parents=True, exist_ok=True)+ (out / "meta.json").write_text(json.dumps(self.meta, indent=1))+ for cam, b in self.video.items():+ (out / f"{cam}.mp4").write_bytes(b)+ for cam, img in self.last.items():+ (out / "last").mkdir(exist_ok=True)+ Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)+ for cam, arr in self.frames.items():+ d = out / "frames" / cam+ d.mkdir(parents=True, exist_ok=True)+ for i, f in enumerate(arr):+ Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)+ for cam, arr in self.depth.items():+ np.save(out / f"depth_{cam}.npy", arr)+ if self.proprio is not None:+ np.save(out / "proprio.npy", self.proprio)+ return str(out)+++class Robot:+ def __init__(self, base: str | None = None, timeout: float = 600.0):+ candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]+ t0, last = time.time(), None+ while time.time() - t0 < timeout:+ for b in candidates:+ try:+ r = requests.get(b + "/spec", timeout=(5, 60))+ if r.status_code == 503:+ last = "warming up"+ continue+ r.raise_for_status()+ self.base, self._spec = b, r.json()+ if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):+ raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")+ return+ except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:+ last = e # not up yet (the scene takes about a minute), or busy+ time.sleep(2)+ raise RuntimeError(f"sim service unavailable: {last}")++ def spec(self) -> dict:+ return self._spec++ def status(self) -> dict:+ r = requests.get(self.base + "/status", timeout=(5, 60))+ r.raise_for_status()+ return r.json()++ def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,+ video: str = "frames") -> Result:+ meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),+ "include_depth": bool(include_depth), "video": video}+ return self._post("/observe", meta, [], timeout=900)++ def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",+ at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",+ include_depth: bool = False, note: str = "") -> Result:+ a = np.asarray(actions, dtype=np.float64)+ if a.ndim == 1:+ a = a[None, :]+ if not np.all(np.isfinite(a)):+ raise ValueError("actions contain NaN or inf")+ meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,+ "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),+ "video": video,+ "include_depth": bool(include_depth), "note": note}+ return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)++ def reset(self, cameras=None, include_depth: bool = False) -> Result:+ return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)++ def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:+ body = rcb1.encode(meta, parts)+ if len(body) > MAX_BODY: # the service would close the connection on it unread+ raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")+ r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},+ timeout=(5, timeout))+ if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:+ return Result(*rcb1.decode(r.content))+ try:+ err = r.json()+ except ValueError:+ err = {"error": r.text[:300]}+ msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")+ if r.status_code == 409:+ raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")+ + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))+ if r.status_code in (400, 413):+ raise ValueError(msg)+ raise RuntimeError(msg)+++def _names(cameras) -> list[str] | None:+ return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb_realtime.pynew file mode 100644index 0000000000..e23f8cbd2c--- /dev/null+++ b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -0,0 +1,480 @@+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.++It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery+and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:++- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its+ first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then+ on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,+ until the episode ends; the clock stops there.+- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`+ or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the+ next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`+ loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order+ hold): a slow agent's robot keeps its last servo targets.+- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)+ and returns what the cameras saw. It never changes the plan.+- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.++Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's+rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was+submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):+ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line+per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:+call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.++Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in+between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting+request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than+MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.++A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.+A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a+crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the+trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends+the graded episode (reason "error") sets infra_error in final.json too.+"""+from __future__ import annotations++import collections+import json+import os+import queue+import sys+import threading+import time+import traceback+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from urllib.parse import urlparse++import numpy as np++import rcb_service+from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request++PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up+CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)+++class _Job:+ """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""++ def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):+ self.method, self.path, self.body, self.content_length = method, path, body, content_length+ self.done = threading.Event()+ self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)++ def finish(self, *reply) -> None:+ self.reply = reply+ self.done.set()+++class _Watch:+ """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""++ def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):+ self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id+ self.start, self.end = start, start + steps+++class RealtimeService(Service):+ """rcb_service.Service with the realtime clock (module docstring)."""++ def __init__(self, sim, *, slowdown: float | None = None, **kw):+ self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)+ if not self.slowdown > 0:+ raise ValueError("RCB_SLOWDOWN must be > 0")+ self.wall_dt = sim.control_dt * self.slowdown+ self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be+ self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step+ self.running = False+ self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t+ self.watches: list[_Watch] = []+ self.q: queue.Queue = queue.Queue()+ self.max_lag = self.lag = 0.0+ self.stalls = self.held_steps = self.late_steps = 0+ self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk+ super().__init__(sim, **kw)+ self.ticks_path = os.path.join(self.data, "ticks.f64")+ self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))++ # ------------------------------------------------------------------ spec, status, evidence+ def _spec(self) -> dict:+ spec = super()._spec()+ spec["protocol"] = PROTOCOL+ cdt = self.sim.control_dt+ spec["timing"] = {+ "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,+ "wall_s_per_control_step": cdt * self.slowdown,+ "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "+ "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "+ "you do, until the episode ends",+ "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "+ "`at` (default: the next one), replacing what was planned from there; rows for steps already "+ "gone are dropped (meta.late_steps); after the plan's last row that target is held",+ "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "+ "the cameras saw; never changes the plan",+ "max_ahead_steps": self.max_ahead,+ "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}+ return spec++ def _realtime_counters(self) -> dict:+ return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}++ def status(self) -> dict:+ return {**super().status(), **self._realtime_counters()}++ def meta(self) -> dict:+ return {**super().meta(), **self._realtime_counters()}++ def _timing_record(self) -> dict:+ return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),+ "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}++ def _write_final(self) -> None:+ self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's+ if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)+ self.verdict["infra_error"] = 1+ else: # (an episode that erred and was reset away is not graded)+ self.verdict.pop("infra_error", None)+ super()._write_final()++ def _checkpoint(self, ev: str, **kw) -> None:+ if self._ticks: # the tick log first: a crash in between leaves it too long+ with open(self.ticks_path, "ab") as f:+ f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())+ self._ticks = []+ super()._checkpoint(ev, **kw)++ def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:+ keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",+ "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")+ line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}+ with open(os.path.join(self.data, "calls.jsonl"), "a") as f:+ f.write(json.dumps(line) + "\n")+ if actions is not None:+ rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)+ with open(os.path.join(self.data, "submitted.f64"), "ab") as f:+ f.write(np.asarray(rows, dtype=np.float64).tobytes())++ # ------------------------------------------------------------------ start, clock+ def start(self) -> None:+ super().start()+ rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)+ n = self.journal_rows+ if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too+ rows[: 2 * n].tofile(self.ticks_path)+ elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)+ np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)+ self.running = False # a resumed episode waits, paused, for the agent's next call++ def _start_clock(self) -> None:+ if not self.running and not self.done and not self.stopping:+ self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)+ self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)++ def _stop_clock(self, why: str) -> None:+ if self.running:+ self.running = False+ self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))+ self.plan.clear()++ def _end(self, reason: str) -> None:+ super()._end(reason)+ self._stop_clock(reason) # pending watches are completed by whoever ended it, after this++ def due_in(self, now: float) -> float | None:+ """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""+ if not self.running or self.done or self.stopping:+ return None+ return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now++ def tick(self) -> None:+ """One control step of the clock: the plan's row for it, or the held target."""+ now = time.monotonic()+ lag = -(self.due_in(now) or 0.0)+ if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt+ self.stalls += 1+ self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))+ self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0+ self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)+ if self.plan:+ typ, row, src = self.plan.popleft()+ else:+ (typ, row), src = self.sim.hold(), -1+ self.held_steps += 1+ before = self.journal_rows+ self._ticks.append((time.time(), float(src)))+ try:+ self._step(typ, row)+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ if self.journal_rows == before:+ self._ticks.pop()+ self.log("error", path="clock", error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ return+ for w in list(self.watches):+ w.rec.step()+ if self.done or len(self.episode) >= w.end:+ self._complete_watch(w)+ if not self.done and len(self.episode) % self.ckpt_every == 0:+ self._checkpoint("tick")++ # ------------------------------------------------------------------ requests (simulator thread)+ def dispatch(self, job: _Job) -> None:+ """Serve one queued request; a watch finishes later, from tick()."""+ try:+ self.first_request()+ path = urlparse(job.path).path+ if job.method == "GET" and path in ("/spec", "/status"):+ return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))+ routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}+ if self.reset_budget:+ routes["/reset"] = self._rt_reset+ if job.method != "POST" or path not in routes:+ return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))+ if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:+ return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))+ try:+ meta, parts = self._decode(job.body)+ except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error+ return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))+ try:+ routes[path](meta, parts, job)+ except Refused as exc: # 400 (bad_request), 409: nothing changed+ job.finish("raw", *self._reply_json(exc.code, exc.body))+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ self.log("error", path=path, error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))+ except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever+ traceback.print_exc()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))++ def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:+ """2.1's _view, plus cameras = [] for no images."""+ none = isinstance(meta.get("cameras"), list) and not meta["cameras"]+ cams, fps, video, depth = self._view(meta, default_video)+ return ([], fps, "none", False) if none else (cams, fps, video, depth)++ def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:+ """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the+ plan's last row (held), else -- nothing planned, or a step already run -- the current command."""+ i = step - nxt+ if i < 0 or not self.plan:+ return np.asarray(self.sim.command(typ), dtype=np.float64)+ return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)++ def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:+ if self.done or self.stopping:+ raise self._conflict()+ nxt = len(self.episode)+ with bad_request():+ typ = meta.get("type")+ if not isinstance(typ, str) or typ not in self.types:+ raise ValueError(f"type must be one of {sorted(self.types)}")+ try:+ A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),+ dtype=np.float64)+ except (TypeError, ValueError, KeyError):+ raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None+ dim = int(self.types[typ]["dim"])+ if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:+ raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")+ if not np.all(np.isfinite(A)):+ raise ValueError("actions contain NaN or inf")+ cdt = self.sim.control_dt+ try:+ dt = cdt if meta.get("dt") is None else float(meta["dt"])+ except (TypeError, ValueError):+ raise ValueError("dt must be a number of seconds") from None+ k = round(dt / cdt) if np.isfinite(dt / cdt) else 0+ if k < 1 or abs(k * cdt - dt) > 1e-9:+ raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")+ if k > 1 and not self.sim.interpolable(typ):+ raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")+ if len(A) * k * cdt > self.max_chunk_s + 1e-9:+ raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "+ f"more than max_chunk_s = {self.max_chunk_s:g}")+ interp = meta.get("interpolate", "linear")+ if interp not in ("linear", "hold"):+ raise ValueError("interpolate must be linear or hold")+ at = meta.get("at")+ if at is not None:+ if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:+ raise ValueError("at must be a control step of this episode (an integer >= 0), "+ "or null for the next one")+ at = int(at)+ if at - nxt > self.max_ahead:+ raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "+ f"of the next one ({nxt})")+ at = nxt if at is None else at+ cams, fps, video, depth = self._view_rt(meta, "none")+ rows, clipped = self.sim.clip(typ, A)+ prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`+ steps = []+ for row in rows:+ for j in range(1, k + 1):+ steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))+ prev = row+ late = max(0, nxt - at)+ keep = at - nxt if at > nxt else 0+ self.calls += 1+ self.rows += len(A)+ head = list(self.plan)[:keep]+ if len(head) < keep: # a gap before `at`: the target in force then is held through it+ fill = self._planned(at - 1, nxt, typ)+ head += [(typ, fill, -1)] * (keep - len(head))+ self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])+ self.late_steps += min(late, len(steps))+ self._start_clock()+ rec = Recorder(self.sim, cams, fps, depth, "none")+ rec.start()+ rec.finish()+ out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,+ "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,+ "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}+ self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)+ job.finish("rec", out, rec)++ def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:+ with bad_request():+ try:+ seconds = float(meta.get("seconds") or 0.0)+ except (TypeError, ValueError):+ raise ValueError("seconds must be a number") from None+ if not 0.0 <= seconds <= self.max_watch_s:+ raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")+ cams, fps, video, depth = self._view_rt(meta, "frames")+ n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0+ rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video+ self._frame_budget(rec, n)+ self.calls += 1+ rec.start()+ if not n:+ rec.finish()+ out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}+ self._save_call(out, None)+ return job.finish("rec", out, rec)+ self._start_clock()+ self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))++ def _complete_watch(self, w: _Watch) -> None:+ self.watches.remove(w)+ w.rec.finish()+ ran = max(0, min(len(self.episode), w.end) - w.start)+ out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),+ "control_steps_watch": ran, "ok": True}+ self._save_call(out, None)+ w.job.finish("rec", out, w.rec)++ def _release_watches(self) -> None:+ for w in list(self.watches):+ self._complete_watch(w)++ def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:+ self._release_watches()+ out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)+ self._stop_clock("reset")+ job.finish("rcb1", {**out, **self._realtime_counters()}, prts)++ # ------------------------------------------------------------------ the loop (simulator thread) and HTTP+ def poll(self) -> None:+ """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""+ if self.stopping and not self.finalized:+ if not self.done:+ self._end("timeout")+ self._release_watches()+ self._finalize()++ def loop(self, until: threading.Event | None = None) -> None:+ """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""+ while until is None or not until.is_set():+ self.poll()+ wait = self.due_in(time.monotonic())+ if wait is not None and wait <= 0:+ self.tick()+ continue+ try:+ job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))+ except queue.Empty:+ continue+ self.dispatch(job)++ def submit_job(self, job: _Job) -> tuple[int, str, bytes]:+ """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""+ self.q.put(job)+ job.done.wait()+ kind, *rest = job.reply+ if kind == "raw":+ return tuple(rest)+ meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread+ return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)++ def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:+ svc = self++ class Handler(BaseHTTPRequestHandler):+ timeout = 120++ def log_message(self, *a):+ pass++ def _send(self, code: int, ctype: str, body: bytes) -> None:+ self.send_response(code)+ self.send_header("Content-Type", ctype)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_GET(self):+ self._send(*svc.submit_job(_Job("GET", self.path, b"")))++ def do_POST(self):+ try:+ n = int(self.headers.get("Content-Length") or 0)+ except ValueError:+ return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))+ if n > MAX_BODY: # refused unread: drop the connection with it+ self.close_connection = True+ return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))+ self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))++ class Server(ThreadingHTTPServer):+ daemon_threads = True++ return Server(address, Handler)++ def serve(self, port: int = 8000) -> None:+ import signal+ signal.signal(signal.SIGUSR1, self.on_sigusr1)+ srv = self.bind(port)+ threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()+ print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)+ self.loop()+++def main(build, port: int = 8000) -> None:+ """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of+ Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""+ rcb_service.Service = RealtimeService+ rcb_service.PROTOCOL = PROTOCOL+ rcb_service.main(build, port)+++if __name__ == "__main__":+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+ import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import++ main(server.build)diff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/instruction.md b/tasks/humanoidbench-manip-cube-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-manip-cube-i00-realtime/instruction.mdrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/instruction.mdindex bef6c4124c..6eb7114ea4 100644--- a/tasks/humanoidbench-manip-cube-i00-realtime/instruction.md+++ b/tasks/humanoidbench-manip-cube-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# cube — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# cube — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid with two Shadow dexterous hands (HumanoidBench / MuJoCo) **as ifit were real hardware**: the simulator runs in a service you cannot open, and you reach it only through a@@ -77,7 +77,7 @@ which counts as far off. The service ends the episode the moment the pass mark i(`reason = "success"`), and a fresh replay of the recorded episode, the servo targets the robot actuallyexecuted at each control step, must reach it as well and end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-manip-cube-i00-realtime/task.toml b/tasks/humanoidbench-manip-cube-i00-standard-realtime/task.tomlsimilarity index 71%rename from tasks/humanoidbench-manip-cube-i00-realtime/task.tomlrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/task.tomlindex 5b4a5c4308..b848422839 100644--- a/tasks/humanoidbench-manip-cube-i00-realtime/task.toml+++ b/tasks/humanoidbench-manip-cube-i00-standard-realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of humanoidbench-manip-cube-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of humanoidbench-manip-cube-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,10 +16,10 @@ artifacts = [][task]-name = "robot-coding-bench/humanoidbench-manip-cube-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench cube on the same frozen instance as humanoidbench-manip-cube-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/humanoidbench-manip-cube-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench cube on the same frozen instance as humanoidbench-manip-cube-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "realtime-mode", "trajectory", "cube"]+keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "standard-realtime-mode", "trajectory", "cube"][metadata]author_name = "pingyue"@@ -27,7 +27,7 @@ task_type = "trajectory"simulator = "humanoidbench"robot = "unitree-h1hand"family = "cube"-mode = "realtime"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = 0expert_time_estimate_hours = 8.0diff --git a/tasks/humanoidbench-manip-cube-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-manip-cube-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-manip-cube-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-manip-cube-i00-realtime/tests/grade.py b/tasks/humanoidbench-manip-cube-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-manip-cube-i00-realtime/tests/test.sh b/tasks/humanoidbench-manip-cube-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-manip-cube-i00-realtime/tests/test.shrename to tasks/humanoidbench-manip-cube-i00-standard-realtime/tests/test.shdiff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-door-i00-realtime/environment/rcb_client.pydeleted file mode 100644index 8e057bb6e9..0000000000--- a/tasks/humanoidbench-manip-door-i00-realtime/environment/rcb_client.py+++ /dev/null@@ -1,209 +0,0 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on-PYTHONPATH).-- from rcb_client import Robot, EpisodeOver- r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready--TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or-get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x-spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a-plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out-the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.--Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):- r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"- ("joint_position": dim, layout (actuator order), lower/upper, range_rad,- units, semantics), "control_dt" (0.02 s), "max_control_steps" (the- episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:- rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"- (joints, axes, positions), "imu", "links" (masses, centres of mass, the- feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"- r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,- planned_steps, lag_s (and episode, resets_left with a reset budget)- r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")- POST /observe. seconds=0: the robot now, at once. seconds>0 (at most- spec()["max_watch_s"]): waits while that much robot time runs (starting the- clock if it is not running) and returns what the cameras saw; it never- changes the plan- r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,- video="none", include_depth=False, note="")- POST /submit: answers at once with the robot as it is now (the rows- have not run yet). actions (T, dim): rows dt apart (a multiple of- control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's- target is the row (interpolate="hold") or a straight line from the previous- row's ("linear", starting from the target in force just before `at`). They- become the plan from control step `at` (an episode step number; None = the- next step), replacing whatever was planned from `at` on and keeping what is- planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows- for steps already gone when the request arrives are dropped- (meta["late_steps"]). The first submit of an episode starts its clock- r.reset(cameras=None, include_depth=False)- POST /reset, only when spec()["budget"]["reset"] > 0: abandons the- running episode and starts a new one, paused, from the same starting state;- only the last episode is graded, and a success ends the trial--cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the-fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),-"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth-for the cameras that have it.--Result (what get_obs / submit / reset return):- res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),- sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |- "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,- lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]- (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,- resets_left- res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])- res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes- res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)- res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s- of a watch, one at its end- res.done the episode is over- res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy--Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an-unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over-(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this-run (404).-"""-from __future__ import annotations--import json-import os-import pathlib-import time--import numpy as np-import requests--import rcb1--PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)-MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused---class EpisodeOver(RuntimeError):- pass---class Result:- """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""-- def __init__(self, meta: dict, parts: dict):- self.meta, self.raw = meta, parts- self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None- for name, p in parts.items():- kind, data = p["kind"], p["data"]- if name.startswith("video/") and kind == "mp4":- self.video[name[6:]] = data- elif name.startswith("frames/"):- self.frames[name[7:]] = data- elif name.startswith("last/"):- self.last[name[5:]] = data- elif name.startswith("depth/"):- self.depth[name[6:]] = data- elif name == "proprio":- self.proprio = data-- @property- def done(self) -> bool:- return bool(self.meta.get("done"))-- def save(self, out_dir: str) -> str:- from PIL import Image- out = pathlib.Path(out_dir)- out.mkdir(parents=True, exist_ok=True)- (out / "meta.json").write_text(json.dumps(self.meta, indent=1))- for cam, b in self.video.items():- (out / f"{cam}.mp4").write_bytes(b)- for cam, img in self.last.items():- (out / "last").mkdir(exist_ok=True)- Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)- for cam, arr in self.frames.items():- d = out / "frames" / cam- d.mkdir(parents=True, exist_ok=True)- for i, f in enumerate(arr):- Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)- for cam, arr in self.depth.items():- np.save(out / f"depth_{cam}.npy", arr)- if self.proprio is not None:- np.save(out / "proprio.npy", self.proprio)- return str(out)---class Robot:- def __init__(self, base: str | None = None, timeout: float = 600.0):- candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]- t0, last = time.time(), None- while time.time() - t0 < timeout:- for b in candidates:- try:- r = requests.get(b + "/spec", timeout=(5, 60))- if r.status_code == 503:- last = "warming up"- continue- r.raise_for_status()- self.base, self._spec = b, r.json()- if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):- raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")- return- except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:- last = e # not up yet (the scene takes about a minute), or busy- time.sleep(2)- raise RuntimeError(f"sim service unavailable: {last}")-- def spec(self) -> dict:- return self._spec-- def status(self) -> dict:- r = requests.get(self.base + "/status", timeout=(5, 60))- r.raise_for_status()- return r.json()-- def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,- video: str = "frames") -> Result:- meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),- "include_depth": bool(include_depth), "video": video}- return self._post("/observe", meta, [], timeout=900)-- def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",- at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",- include_depth: bool = False, note: str = "") -> Result:- a = np.asarray(actions, dtype=np.float64)- if a.ndim == 1:- a = a[None, :]- if not np.all(np.isfinite(a)):- raise ValueError("actions contain NaN or inf")- meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,- "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),- "video": video,- "include_depth": bool(include_depth), "note": note}- return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)-- def reset(self, cameras=None, include_depth: bool = False) -> Result:- return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)-- def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:- body = rcb1.encode(meta, parts)- if len(body) > MAX_BODY: # the service would close the connection on it unread- raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")- r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},- timeout=(5, timeout))- if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:- return Result(*rcb1.decode(r.content))- try:- err = r.json()- except ValueError:- err = {"error": r.text[:300]}- msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")- if r.status_code == 409:- raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")- + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))- if r.status_code in (400, 413):- raise ValueError(msg)- raise RuntimeError(msg)---def _names(cameras) -> list[str] | None:- return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb_realtime.pydeleted file mode 100644index 410284bf27..0000000000--- a/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb_realtime.py+++ /dev/null@@ -1,479 +0,0 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.--It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery-and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:--- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its- first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then- on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,- until the episode ends; the clock stops there.-- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`- or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the- next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`- loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order- hold): a slow agent's robot keeps its last servo targets.-- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)- and returns what the cameras saw. It never changes the plan.-- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.--Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's-rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was-submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):-ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line-per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:-call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.--Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in-between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting-request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than-MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.--A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.-A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a-crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the-trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends-the graded episode (reason "error") sets infra_error in final.json too.-"""-from __future__ import annotations--import collections-import json-import os-import queue-import sys-import threading-import time-import traceback-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer-from urllib.parse import urlparse--import numpy as np--import rcb_service-from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request--PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1-MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up-CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)---class _Job:- """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""-- def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):- self.method, self.path, self.body, self.content_length = method, path, body, content_length- self.done = threading.Event()- self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)-- def finish(self, *reply) -> None:- self.reply = reply- self.done.set()---class _Watch:- """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""-- def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):- self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id- self.start, self.end = start, start + steps---class RealtimeService(Service):- """rcb_service.Service with the realtime clock (module docstring)."""-- def __init__(self, sim, *, slowdown: float | None = None, **kw):- self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)- if not self.slowdown > 0:- raise ValueError("RCB_SLOWDOWN must be > 0")- self.wall_dt = sim.control_dt * self.slowdown- self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be- self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step- self.running = False- self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t- self.watches: list[_Watch] = []- self.q: queue.Queue = queue.Queue()- self.max_lag = self.lag = 0.0- self.stalls = self.held_steps = self.late_steps = 0- self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk- super().__init__(sim, **kw)- self.ticks_path = os.path.join(self.data, "ticks.f64")- self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))-- # ------------------------------------------------------------------ spec, status, evidence- def _spec(self) -> dict:- spec = super()._spec()- spec["protocol"] = PROTOCOL- cdt = self.sim.control_dt- spec["timing"] = {- "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,- "wall_s_per_control_step": cdt * self.slowdown,- "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "- "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "- "you do, until the episode ends",- "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "- "`at` (default: the next one), replacing what was planned from there; rows for steps already "- "gone are dropped (meta.late_steps); after the plan's last row that target is held",- "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "- "the cameras saw; never changes the plan",- "max_ahead_steps": self.max_ahead,- "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}- return spec-- def _realtime_counters(self) -> dict:- return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}-- def status(self) -> dict:- return {**super().status(), **self._realtime_counters()}-- def meta(self) -> dict:- return {**super().meta(), **self._realtime_counters()}-- def _timing_record(self) -> dict:- return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),- "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}-- def _write_final(self) -> None:- self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's- if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)- self.verdict["infra_error"] = 1- else: # (an episode that erred and was reset away is not graded)- self.verdict.pop("infra_error", None)- super()._write_final()-- def _checkpoint(self, ev: str, **kw) -> None:- if self._ticks: # the tick log first: a crash in between leaves it too long- with open(self.ticks_path, "ab") as f:- f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())- self._ticks = []- super()._checkpoint(ev, **kw)-- def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:- keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",- "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")- line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}- with open(os.path.join(self.data, "calls.jsonl"), "a") as f:- f.write(json.dumps(line) + "\n")- if actions is not None:- rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)- with open(os.path.join(self.data, "submitted.f64"), "ab") as f:- f.write(np.asarray(rows, dtype=np.float64).tobytes())-- # ------------------------------------------------------------------ start, clock- def start(self) -> None:- super().start()- rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)- n = self.journal_rows- if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too- rows[: 2 * n].tofile(self.ticks_path)- elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)- np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)- self.running = False # a resumed episode waits, paused, for the agent's next call-- def _start_clock(self) -> None:- if not self.running and not self.done and not self.stopping:- self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)- self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)-- def _stop_clock(self, why: str) -> None:- if self.running:- self.running = False- self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))- self.plan.clear()-- def _end(self, reason: str) -> None:- super()._end(reason)- self._stop_clock(reason) # pending watches are completed by whoever ended it, after this-- def due_in(self, now: float) -> float | None:- """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""- if not self.running or self.done or self.stopping:- return None- return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now-- def tick(self) -> None:- """One control step of the clock: the plan's row for it, or the held target."""- now = time.monotonic()- lag = -(self.due_in(now) or 0.0)- if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt- self.stalls += 1- self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))- self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0- self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)- if self.plan:- typ, row, src = self.plan.popleft()- else:- (typ, row), src = self.sim.hold(), -1- self.held_steps += 1- before = self.journal_rows- self._ticks.append((time.time(), float(src)))- try:- self._step(typ, row)- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- if self.journal_rows == before:- self._ticks.pop()- self.log("error", path="clock", error=repr(exc)[:500])- self._end("error")- self._release_watches()- return- for w in list(self.watches):- w.rec.step()- if self.done or len(self.episode) >= w.end:- self._complete_watch(w)- if not self.done and len(self.episode) % self.ckpt_every == 0:- self._checkpoint("tick")-- # ------------------------------------------------------------------ requests (simulator thread)- def dispatch(self, job: _Job) -> None:- """Serve one queued request; a watch finishes later, from tick()."""- try:- self.first_request()- path = urlparse(job.path).path- if job.method == "GET" and path in ("/spec", "/status"):- return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))- routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}- if self.reset_budget:- routes["/reset"] = self._rt_reset- if job.method != "POST" or path not in routes:- return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))- if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:- return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))- try:- meta, parts = self._decode(job.body)- except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error- return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))- try:- routes[path](meta, parts, job)- except Refused as exc: # 400 (bad_request), 409: nothing changed- job.finish("raw", *self._reply_json(exc.code, exc.body))- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- self.log("error", path=path, error=repr(exc)[:500])- self._end("error")- self._release_watches()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))- except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever- traceback.print_exc()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))-- def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:- """2.1's _view, plus cameras = [] for no images."""- none = isinstance(meta.get("cameras"), list) and not meta["cameras"]- cams, fps, video, depth = self._view(meta, default_video)- return ([], fps, "none", False) if none else (cams, fps, video, depth)-- def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:- """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the- plan's last row (held), else -- nothing planned, or a step already run -- the current command."""- i = step - nxt- if i < 0 or not self.plan:- return np.asarray(self.sim.command(typ), dtype=np.float64)- return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)-- def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:- if self.done or self.stopping:- raise self._conflict()- nxt = len(self.episode)- with bad_request():- typ = meta.get("type")- if not isinstance(typ, str) or typ not in self.types:- raise ValueError(f"type must be one of {sorted(self.types)}")- try:- A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),- dtype=np.float64)- except (TypeError, ValueError, KeyError):- raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None- dim = int(self.types[typ]["dim"])- if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:- raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")- if not np.all(np.isfinite(A)):- raise ValueError("actions contain NaN or inf")- cdt = self.sim.control_dt- try:- dt = cdt if meta.get("dt") is None else float(meta["dt"])- except (TypeError, ValueError):- raise ValueError("dt must be a number of seconds") from None- k = round(dt / cdt) if np.isfinite(dt / cdt) else 0- if k < 1 or abs(k * cdt - dt) > 1e-9:- raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")- if k > 1 and not self.sim.interpolable(typ):- raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")- if len(A) * k * cdt > self.max_chunk_s + 1e-9:- raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "- f"more than max_chunk_s = {self.max_chunk_s:g}")- interp = meta.get("interpolate", "linear")- if interp not in ("linear", "hold"):- raise ValueError("interpolate must be linear or hold")- at = meta.get("at")- if at is not None:- if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:- raise ValueError("at must be a control step of this episode (an integer >= 0), "- "or null for the next one")- at = int(at)- if at - nxt > self.max_ahead:- raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "- f"of the next one ({nxt})")- at = nxt if at is None else at- cams, fps, video, depth = self._view_rt(meta, "none")- rows, clipped = self.sim.clip(typ, A)- prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`- steps = []- for row in rows:- for j in range(1, k + 1):- steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))- prev = row- late = max(0, nxt - at)- keep = at - nxt if at > nxt else 0- self.calls += 1- self.rows += len(A)- head = list(self.plan)[:keep]- if len(head) < keep: # a gap before `at`: the target in force then is held through it- fill = self._planned(at - 1, nxt, typ)- head += [(typ, fill, -1)] * (keep - len(head))- self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])- self.late_steps += min(late, len(steps))- self._start_clock()- rec = Recorder(self.sim, cams, fps, depth, "none")- rec.start()- rec.finish()- out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,- "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,- "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}- self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)- job.finish("rec", out, rec)-- def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:- with bad_request():- try:- seconds = float(meta.get("seconds") or 0.0)- except (TypeError, ValueError):- raise ValueError("seconds must be a number") from None- if not 0.0 <= seconds <= self.max_watch_s:- raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")- cams, fps, video, depth = self._view_rt(meta, "frames")- n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0- rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video- self._frame_budget(rec, n)- self.calls += 1- rec.start()- if not n:- rec.finish()- out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}- self._save_call(out, None)- return job.finish("rec", out, rec)- self._start_clock()- self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))-- def _complete_watch(self, w: _Watch) -> None:- self.watches.remove(w)- w.rec.finish()- ran = max(0, min(len(self.episode), w.end) - w.start)- out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),- "control_steps_watch": ran, "ok": True}- self._save_call(out, None)- w.job.finish("rec", out, w.rec)-- def _release_watches(self) -> None:- for w in list(self.watches):- self._complete_watch(w)-- def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:- self._release_watches()- out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)- self._stop_clock("reset")- job.finish("rcb1", {**out, **self._realtime_counters()}, prts)-- # ------------------------------------------------------------------ the loop (simulator thread) and HTTP- def poll(self) -> None:- """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""- if self.stopping and not self.finalized:- if not self.done:- self._end("timeout")- self._release_watches()- self._finalize()-- def loop(self, until: threading.Event | None = None) -> None:- """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""- while until is None or not until.is_set():- self.poll()- wait = self.due_in(time.monotonic())- if wait is not None and wait <= 0:- self.tick()- continue- try:- job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))- except queue.Empty:- continue- self.dispatch(job)-- def submit_job(self, job: _Job) -> tuple[int, str, bytes]:- """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""- self.q.put(job)- job.done.wait()- kind, *rest = job.reply- if kind == "raw":- return tuple(rest)- meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread- return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)-- def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:- svc = self-- class Handler(BaseHTTPRequestHandler):- timeout = 120-- def log_message(self, *a):- pass-- def _send(self, code: int, ctype: str, body: bytes) -> None:- self.send_response(code)- self.send_header("Content-Type", ctype)- self.send_header("Content-Length", str(len(body)))- self.end_headers()- self.wfile.write(body)-- def do_GET(self):- self._send(*svc.submit_job(_Job("GET", self.path, b"")))-- def do_POST(self):- try:- n = int(self.headers.get("Content-Length") or 0)- except ValueError:- return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))- if n > MAX_BODY: # refused unread: drop the connection with it- self.close_connection = True- return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))- self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))-- class Server(ThreadingHTTPServer):- daemon_threads = True-- return Server(address, Handler)-- def serve(self, port: int = 8000) -> None:- import signal- signal.signal(signal.SIGUSR1, self.on_sigusr1)- srv = self.bind(port)- threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()- print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)- self.loop()---def main(build, port: int = 8000) -> None:- """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of- Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""- rcb_service.Service = RealtimeService- rcb_service.PROTOCOL = PROTOCOL- rcb_service.main(build, port)---if __name__ == "__main__":- sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))- import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import-- main(server.build)diff --git a/tasks/humanoidbench-manip-door-i00-realtime/README.md b/tasks/humanoidbench-manip-door-i00-standard-realtime/README.mdsimilarity index 83%rename from tasks/humanoidbench-manip-door-i00-realtime/README.mdrename to tasks/humanoidbench-manip-door-i00-standard-realtime/README.mdindex c03b55366d..31f23213a4 100644--- a/tasks/humanoidbench-manip-door-i00-realtime/README.md+++ b/tasks/humanoidbench-manip-door-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-manip-door-i00-realtime+# humanoidbench-manip-door-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-manip-door-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1` || Family / instance | `door` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-manip-door-i00-privileged`: the same frozen instance and the same success rule as it and as`humanoidbench-manip-door-i00-standard`, but time does not stop. Once an episode's clock starts (its first`submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the agent thinks, so@@ -56,8 +56,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no door or handle state, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no door or handle state, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 19 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 89%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/docker-compose.yamlindex 180ef8bf61..518539d47d 100644--- a/tasks/humanoidbench-manip-door-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/rcb_client.pynew file mode 100644index 0000000000..f2b0559bf9--- /dev/null+++ b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/rcb_client.py@@ -0,0 +1,209 @@+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+PYTHONPATH).++ from rcb_client import Robot, EpisodeOver+ r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready++TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or+get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x+spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a+plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out+the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.++Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):+ r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"+ ("joint_position": dim, layout (actuator order), lower/upper, range_rad,+ units, semantics), "control_dt" (0.02 s), "max_control_steps" (the+ episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:+ rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"+ (joints, axes, positions), "imu", "links" (masses, centres of mass, the+ feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"+ r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,+ planned_steps, lag_s (and episode, resets_left with a reset budget)+ r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")+ POST /observe. seconds=0: the robot now, at once. seconds>0 (at most+ spec()["max_watch_s"]): waits while that much robot time runs (starting the+ clock if it is not running) and returns what the cameras saw; it never+ changes the plan+ r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,+ video="none", include_depth=False, note="")+ POST /submit: answers at once with the robot as it is now (the rows+ have not run yet). actions (T, dim): rows dt apart (a multiple of+ control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's+ target is the row (interpolate="hold") or a straight line from the previous+ row's ("linear", starting from the target in force just before `at`). They+ become the plan from control step `at` (an episode step number; None = the+ next step), replacing whatever was planned from `at` on and keeping what is+ planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows+ for steps already gone when the request arrives are dropped+ (meta["late_steps"]). The first submit of an episode starts its clock+ r.reset(cameras=None, include_depth=False)+ POST /reset, only when spec()["budget"]["reset"] > 0: abandons the+ running episode and starts a new one, paused, from the same starting state;+ only the last episode is graded, and a success ends the trial++cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the+fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),+"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth+for the cameras that have it.++Result (what get_obs / submit / reset return):+ res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),+ sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |+ "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,+ lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]+ (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,+ resets_left+ res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])+ res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes+ res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)+ res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s+ of a watch, one at its end+ res.done the episode is over+ res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy++Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an+unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over+(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this+run (404).+"""+from __future__ import annotations++import json+import os+import pathlib+import time++import numpy as np+import requests++import rcb1++PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)+MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused+++class EpisodeOver(RuntimeError):+ pass+++class Result:+ """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""++ def __init__(self, meta: dict, parts: dict):+ self.meta, self.raw = meta, parts+ self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None+ for name, p in parts.items():+ kind, data = p["kind"], p["data"]+ if name.startswith("video/") and kind == "mp4":+ self.video[name[6:]] = data+ elif name.startswith("frames/"):+ self.frames[name[7:]] = data+ elif name.startswith("last/"):+ self.last[name[5:]] = data+ elif name.startswith("depth/"):+ self.depth[name[6:]] = data+ elif name == "proprio":+ self.proprio = data++ @property+ def done(self) -> bool:+ return bool(self.meta.get("done"))++ def save(self, out_dir: str) -> str:+ from PIL import Image+ out = pathlib.Path(out_dir)+ out.mkdir(parents=True, exist_ok=True)+ (out / "meta.json").write_text(json.dumps(self.meta, indent=1))+ for cam, b in self.video.items():+ (out / f"{cam}.mp4").write_bytes(b)+ for cam, img in self.last.items():+ (out / "last").mkdir(exist_ok=True)+ Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)+ for cam, arr in self.frames.items():+ d = out / "frames" / cam+ d.mkdir(parents=True, exist_ok=True)+ for i, f in enumerate(arr):+ Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)+ for cam, arr in self.depth.items():+ np.save(out / f"depth_{cam}.npy", arr)+ if self.proprio is not None:+ np.save(out / "proprio.npy", self.proprio)+ return str(out)+++class Robot:+ def __init__(self, base: str | None = None, timeout: float = 600.0):+ candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]+ t0, last = time.time(), None+ while time.time() - t0 < timeout:+ for b in candidates:+ try:+ r = requests.get(b + "/spec", timeout=(5, 60))+ if r.status_code == 503:+ last = "warming up"+ continue+ r.raise_for_status()+ self.base, self._spec = b, r.json()+ if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):+ raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")+ return+ except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:+ last = e # not up yet (the scene takes about a minute), or busy+ time.sleep(2)+ raise RuntimeError(f"sim service unavailable: {last}")++ def spec(self) -> dict:+ return self._spec++ def status(self) -> dict:+ r = requests.get(self.base + "/status", timeout=(5, 60))+ r.raise_for_status()+ return r.json()++ def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,+ video: str = "frames") -> Result:+ meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),+ "include_depth": bool(include_depth), "video": video}+ return self._post("/observe", meta, [], timeout=900)++ def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",+ at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",+ include_depth: bool = False, note: str = "") -> Result:+ a = np.asarray(actions, dtype=np.float64)+ if a.ndim == 1:+ a = a[None, :]+ if not np.all(np.isfinite(a)):+ raise ValueError("actions contain NaN or inf")+ meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,+ "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),+ "video": video,+ "include_depth": bool(include_depth), "note": note}+ return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)++ def reset(self, cameras=None, include_depth: bool = False) -> Result:+ return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)++ def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:+ body = rcb1.encode(meta, parts)+ if len(body) > MAX_BODY: # the service would close the connection on it unread+ raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")+ r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},+ timeout=(5, timeout))+ if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:+ return Result(*rcb1.decode(r.content))+ try:+ err = r.json()+ except ValueError:+ err = {"error": r.text[:300]}+ msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")+ if r.status_code == 409:+ raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")+ + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))+ if r.status_code in (400, 413):+ raise ValueError(msg)+ raise RuntimeError(msg)+++def _names(cameras) -> list[str] | None:+ return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb_realtime.pynew file mode 100644index 0000000000..e23f8cbd2c--- /dev/null+++ b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -0,0 +1,480 @@+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.++It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery+and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:++- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its+ first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then+ on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,+ until the episode ends; the clock stops there.+- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`+ or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the+ next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`+ loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order+ hold): a slow agent's robot keeps its last servo targets.+- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)+ and returns what the cameras saw. It never changes the plan.+- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.++Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's+rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was+submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):+ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line+per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:+call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.++Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in+between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting+request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than+MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.++A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.+A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a+crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the+trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends+the graded episode (reason "error") sets infra_error in final.json too.+"""+from __future__ import annotations++import collections+import json+import os+import queue+import sys+import threading+import time+import traceback+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from urllib.parse import urlparse++import numpy as np++import rcb_service+from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request++PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up+CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)+++class _Job:+ """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""++ def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):+ self.method, self.path, self.body, self.content_length = method, path, body, content_length+ self.done = threading.Event()+ self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)++ def finish(self, *reply) -> None:+ self.reply = reply+ self.done.set()+++class _Watch:+ """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""++ def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):+ self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id+ self.start, self.end = start, start + steps+++class RealtimeService(Service):+ """rcb_service.Service with the realtime clock (module docstring)."""++ def __init__(self, sim, *, slowdown: float | None = None, **kw):+ self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)+ if not self.slowdown > 0:+ raise ValueError("RCB_SLOWDOWN must be > 0")+ self.wall_dt = sim.control_dt * self.slowdown+ self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be+ self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step+ self.running = False+ self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t+ self.watches: list[_Watch] = []+ self.q: queue.Queue = queue.Queue()+ self.max_lag = self.lag = 0.0+ self.stalls = self.held_steps = self.late_steps = 0+ self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk+ super().__init__(sim, **kw)+ self.ticks_path = os.path.join(self.data, "ticks.f64")+ self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))++ # ------------------------------------------------------------------ spec, status, evidence+ def _spec(self) -> dict:+ spec = super()._spec()+ spec["protocol"] = PROTOCOL+ cdt = self.sim.control_dt+ spec["timing"] = {+ "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,+ "wall_s_per_control_step": cdt * self.slowdown,+ "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "+ "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "+ "you do, until the episode ends",+ "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "+ "`at` (default: the next one), replacing what was planned from there; rows for steps already "+ "gone are dropped (meta.late_steps); after the plan's last row that target is held",+ "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "+ "the cameras saw; never changes the plan",+ "max_ahead_steps": self.max_ahead,+ "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}+ return spec++ def _realtime_counters(self) -> dict:+ return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}++ def status(self) -> dict:+ return {**super().status(), **self._realtime_counters()}++ def meta(self) -> dict:+ return {**super().meta(), **self._realtime_counters()}++ def _timing_record(self) -> dict:+ return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),+ "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}++ def _write_final(self) -> None:+ self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's+ if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)+ self.verdict["infra_error"] = 1+ else: # (an episode that erred and was reset away is not graded)+ self.verdict.pop("infra_error", None)+ super()._write_final()++ def _checkpoint(self, ev: str, **kw) -> None:+ if self._ticks: # the tick log first: a crash in between leaves it too long+ with open(self.ticks_path, "ab") as f:+ f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())+ self._ticks = []+ super()._checkpoint(ev, **kw)++ def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:+ keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",+ "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")+ line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}+ with open(os.path.join(self.data, "calls.jsonl"), "a") as f:+ f.write(json.dumps(line) + "\n")+ if actions is not None:+ rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)+ with open(os.path.join(self.data, "submitted.f64"), "ab") as f:+ f.write(np.asarray(rows, dtype=np.float64).tobytes())++ # ------------------------------------------------------------------ start, clock+ def start(self) -> None:+ super().start()+ rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)+ n = self.journal_rows+ if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too+ rows[: 2 * n].tofile(self.ticks_path)+ elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)+ np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)+ self.running = False # a resumed episode waits, paused, for the agent's next call++ def _start_clock(self) -> None:+ if not self.running and not self.done and not self.stopping:+ self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)+ self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)++ def _stop_clock(self, why: str) -> None:+ if self.running:+ self.running = False+ self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))+ self.plan.clear()++ def _end(self, reason: str) -> None:+ super()._end(reason)+ self._stop_clock(reason) # pending watches are completed by whoever ended it, after this++ def due_in(self, now: float) -> float | None:+ """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""+ if not self.running or self.done or self.stopping:+ return None+ return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now++ def tick(self) -> None:+ """One control step of the clock: the plan's row for it, or the held target."""+ now = time.monotonic()+ lag = -(self.due_in(now) or 0.0)+ if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt+ self.stalls += 1+ self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))+ self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0+ self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)+ if self.plan:+ typ, row, src = self.plan.popleft()+ else:+ (typ, row), src = self.sim.hold(), -1+ self.held_steps += 1+ before = self.journal_rows+ self._ticks.append((time.time(), float(src)))+ try:+ self._step(typ, row)+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ if self.journal_rows == before:+ self._ticks.pop()+ self.log("error", path="clock", error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ return+ for w in list(self.watches):+ w.rec.step()+ if self.done or len(self.episode) >= w.end:+ self._complete_watch(w)+ if not self.done and len(self.episode) % self.ckpt_every == 0:+ self._checkpoint("tick")++ # ------------------------------------------------------------------ requests (simulator thread)+ def dispatch(self, job: _Job) -> None:+ """Serve one queued request; a watch finishes later, from tick()."""+ try:+ self.first_request()+ path = urlparse(job.path).path+ if job.method == "GET" and path in ("/spec", "/status"):+ return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))+ routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}+ if self.reset_budget:+ routes["/reset"] = self._rt_reset+ if job.method != "POST" or path not in routes:+ return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))+ if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:+ return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))+ try:+ meta, parts = self._decode(job.body)+ except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error+ return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))+ try:+ routes[path](meta, parts, job)+ except Refused as exc: # 400 (bad_request), 409: nothing changed+ job.finish("raw", *self._reply_json(exc.code, exc.body))+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ self.log("error", path=path, error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))+ except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever+ traceback.print_exc()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))++ def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:+ """2.1's _view, plus cameras = [] for no images."""+ none = isinstance(meta.get("cameras"), list) and not meta["cameras"]+ cams, fps, video, depth = self._view(meta, default_video)+ return ([], fps, "none", False) if none else (cams, fps, video, depth)++ def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:+ """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the+ plan's last row (held), else -- nothing planned, or a step already run -- the current command."""+ i = step - nxt+ if i < 0 or not self.plan:+ return np.asarray(self.sim.command(typ), dtype=np.float64)+ return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)++ def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:+ if self.done or self.stopping:+ raise self._conflict()+ nxt = len(self.episode)+ with bad_request():+ typ = meta.get("type")+ if not isinstance(typ, str) or typ not in self.types:+ raise ValueError(f"type must be one of {sorted(self.types)}")+ try:+ A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),+ dtype=np.float64)+ except (TypeError, ValueError, KeyError):+ raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None+ dim = int(self.types[typ]["dim"])+ if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:+ raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")+ if not np.all(np.isfinite(A)):+ raise ValueError("actions contain NaN or inf")+ cdt = self.sim.control_dt+ try:+ dt = cdt if meta.get("dt") is None else float(meta["dt"])+ except (TypeError, ValueError):+ raise ValueError("dt must be a number of seconds") from None+ k = round(dt / cdt) if np.isfinite(dt / cdt) else 0+ if k < 1 or abs(k * cdt - dt) > 1e-9:+ raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")+ if k > 1 and not self.sim.interpolable(typ):+ raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")+ if len(A) * k * cdt > self.max_chunk_s + 1e-9:+ raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "+ f"more than max_chunk_s = {self.max_chunk_s:g}")+ interp = meta.get("interpolate", "linear")+ if interp not in ("linear", "hold"):+ raise ValueError("interpolate must be linear or hold")+ at = meta.get("at")+ if at is not None:+ if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:+ raise ValueError("at must be a control step of this episode (an integer >= 0), "+ "or null for the next one")+ at = int(at)+ if at - nxt > self.max_ahead:+ raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "+ f"of the next one ({nxt})")+ at = nxt if at is None else at+ cams, fps, video, depth = self._view_rt(meta, "none")+ rows, clipped = self.sim.clip(typ, A)+ prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`+ steps = []+ for row in rows:+ for j in range(1, k + 1):+ steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))+ prev = row+ late = max(0, nxt - at)+ keep = at - nxt if at > nxt else 0+ self.calls += 1+ self.rows += len(A)+ head = list(self.plan)[:keep]+ if len(head) < keep: # a gap before `at`: the target in force then is held through it+ fill = self._planned(at - 1, nxt, typ)+ head += [(typ, fill, -1)] * (keep - len(head))+ self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])+ self.late_steps += min(late, len(steps))+ self._start_clock()+ rec = Recorder(self.sim, cams, fps, depth, "none")+ rec.start()+ rec.finish()+ out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,+ "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,+ "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}+ self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)+ job.finish("rec", out, rec)++ def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:+ with bad_request():+ try:+ seconds = float(meta.get("seconds") or 0.0)+ except (TypeError, ValueError):+ raise ValueError("seconds must be a number") from None+ if not 0.0 <= seconds <= self.max_watch_s:+ raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")+ cams, fps, video, depth = self._view_rt(meta, "frames")+ n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0+ rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video+ self._frame_budget(rec, n)+ self.calls += 1+ rec.start()+ if not n:+ rec.finish()+ out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}+ self._save_call(out, None)+ return job.finish("rec", out, rec)+ self._start_clock()+ self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))++ def _complete_watch(self, w: _Watch) -> None:+ self.watches.remove(w)+ w.rec.finish()+ ran = max(0, min(len(self.episode), w.end) - w.start)+ out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),+ "control_steps_watch": ran, "ok": True}+ self._save_call(out, None)+ w.job.finish("rec", out, w.rec)++ def _release_watches(self) -> None:+ for w in list(self.watches):+ self._complete_watch(w)++ def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:+ self._release_watches()+ out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)+ self._stop_clock("reset")+ job.finish("rcb1", {**out, **self._realtime_counters()}, prts)++ # ------------------------------------------------------------------ the loop (simulator thread) and HTTP+ def poll(self) -> None:+ """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""+ if self.stopping and not self.finalized:+ if not self.done:+ self._end("timeout")+ self._release_watches()+ self._finalize()++ def loop(self, until: threading.Event | None = None) -> None:+ """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""+ while until is None or not until.is_set():+ self.poll()+ wait = self.due_in(time.monotonic())+ if wait is not None and wait <= 0:+ self.tick()+ continue+ try:+ job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))+ except queue.Empty:+ continue+ self.dispatch(job)++ def submit_job(self, job: _Job) -> tuple[int, str, bytes]:+ """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""+ self.q.put(job)+ job.done.wait()+ kind, *rest = job.reply+ if kind == "raw":+ return tuple(rest)+ meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread+ return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)++ def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:+ svc = self++ class Handler(BaseHTTPRequestHandler):+ timeout = 120++ def log_message(self, *a):+ pass++ def _send(self, code: int, ctype: str, body: bytes) -> None:+ self.send_response(code)+ self.send_header("Content-Type", ctype)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_GET(self):+ self._send(*svc.submit_job(_Job("GET", self.path, b"")))++ def do_POST(self):+ try:+ n = int(self.headers.get("Content-Length") or 0)+ except ValueError:+ return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))+ if n > MAX_BODY: # refused unread: drop the connection with it+ self.close_connection = True+ return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))+ self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))++ class Server(ThreadingHTTPServer):+ daemon_threads = True++ return Server(address, Handler)++ def serve(self, port: int = 8000) -> None:+ import signal+ signal.signal(signal.SIGUSR1, self.on_sigusr1)+ srv = self.bind(port)+ threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()+ print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)+ self.loop()+++def main(build, port: int = 8000) -> None:+ """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of+ Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""+ rcb_service.Service = RealtimeService+ rcb_service.PROTOCOL = PROTOCOL+ rcb_service.main(build, port)+++if __name__ == "__main__":+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+ import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import++ main(server.build)diff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-manip-door-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-manip-door-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-manip-door-i00-realtime/instruction.md b/tasks/humanoidbench-manip-door-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-manip-door-i00-realtime/instruction.mdrename to tasks/humanoidbench-manip-door-i00-standard-realtime/instruction.mdindex 008244e0da..326a7a4b06 100644--- a/tasks/humanoidbench-manip-door-i00-realtime/instruction.md+++ b/tasks/humanoidbench-manip-door-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# door — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# door — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid (HumanoidBench / MuJoCo) **as if it were real hardware**: thesimulator runs in a service you cannot open, and you reach it only through a client. Like real hardware, it@@ -72,7 +72,7 @@ The bar cannot be reached without opening the door. The service ends the episodereached (`reason = "success"`), and a fresh replay of the recorded episode, the servo targets the robotactually executed at each control step, must reach it as well and end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-manip-door-i00-realtime/task.toml b/tasks/humanoidbench-manip-door-i00-standard-realtime/task.tomlsimilarity index 73%rename from tasks/humanoidbench-manip-door-i00-realtime/task.tomlrename to tasks/humanoidbench-manip-door-i00-standard-realtime/task.tomlindex 1ecd9196be..cace12e5cb 100644--- a/tasks/humanoidbench-manip-door-i00-realtime/task.toml+++ b/tasks/humanoidbench-manip-door-i00-standard-realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of humanoidbench-manip-door-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of humanoidbench-manip-door-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,10 +16,10 @@ artifacts = [][task]-name = "robot-coding-bench/humanoidbench-manip-door-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench door on the same frozen instance as humanoidbench-manip-door-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/humanoidbench-manip-door-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench door on the same frozen instance as humanoidbench-manip-door-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1", "manipulation", "realtime-mode", "trajectory", "door"]+keywords = ["robotics", "humanoidbench", "mujoco", "h1", "manipulation", "standard-realtime-mode", "trajectory", "door"][metadata]author_name = "pingyue"@@ -27,7 +27,7 @@ task_type = "trajectory"simulator = "humanoidbench"robot = "unitree-h1"family = "door"-mode = "realtime"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = 0expert_time_estimate_hours = 6.0diff --git a/tasks/humanoidbench-manip-door-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-manip-door-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-manip-door-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-manip-door-i00-realtime/tests/grade.py b/tasks/humanoidbench-manip-door-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-manip-door-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-manip-door-i00-realtime/tests/test.sh b/tasks/humanoidbench-manip-door-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-manip-door-i00-realtime/tests/test.shrename to tasks/humanoidbench-manip-door-i00-standard-realtime/tests/test.shdiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/rcb_client.pydeleted file mode 100644index 8e057bb6e9..0000000000--- a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/rcb_client.py+++ /dev/null@@ -1,209 +0,0 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on-PYTHONPATH).-- from rcb_client import Robot, EpisodeOver- r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready--TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or-get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x-spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a-plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out-the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.--Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):- r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"- ("joint_position": dim, layout (actuator order), lower/upper, range_rad,- units, semantics), "control_dt" (0.02 s), "max_control_steps" (the- episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:- rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"- (joints, axes, positions), "imu", "links" (masses, centres of mass, the- feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"- r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,- planned_steps, lag_s (and episode, resets_left with a reset budget)- r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")- POST /observe. seconds=0: the robot now, at once. seconds>0 (at most- spec()["max_watch_s"]): waits while that much robot time runs (starting the- clock if it is not running) and returns what the cameras saw; it never- changes the plan- r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,- video="none", include_depth=False, note="")- POST /submit: answers at once with the robot as it is now (the rows- have not run yet). actions (T, dim): rows dt apart (a multiple of- control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's- target is the row (interpolate="hold") or a straight line from the previous- row's ("linear", starting from the target in force just before `at`). They- become the plan from control step `at` (an episode step number; None = the- next step), replacing whatever was planned from `at` on and keeping what is- planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows- for steps already gone when the request arrives are dropped- (meta["late_steps"]). The first submit of an episode starts its clock- r.reset(cameras=None, include_depth=False)- POST /reset, only when spec()["budget"]["reset"] > 0: abandons the- running episode and starts a new one, paused, from the same starting state;- only the last episode is graded, and a success ends the trial--cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the-fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),-"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth-for the cameras that have it.--Result (what get_obs / submit / reset return):- res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),- sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |- "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,- lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]- (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,- resets_left- res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])- res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes- res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)- res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s- of a watch, one at its end- res.done the episode is over- res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy--Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an-unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over-(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this-run (404).-"""-from __future__ import annotations--import json-import os-import pathlib-import time--import numpy as np-import requests--import rcb1--PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)-MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused---class EpisodeOver(RuntimeError):- pass---class Result:- """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""-- def __init__(self, meta: dict, parts: dict):- self.meta, self.raw = meta, parts- self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None- for name, p in parts.items():- kind, data = p["kind"], p["data"]- if name.startswith("video/") and kind == "mp4":- self.video[name[6:]] = data- elif name.startswith("frames/"):- self.frames[name[7:]] = data- elif name.startswith("last/"):- self.last[name[5:]] = data- elif name.startswith("depth/"):- self.depth[name[6:]] = data- elif name == "proprio":- self.proprio = data-- @property- def done(self) -> bool:- return bool(self.meta.get("done"))-- def save(self, out_dir: str) -> str:- from PIL import Image- out = pathlib.Path(out_dir)- out.mkdir(parents=True, exist_ok=True)- (out / "meta.json").write_text(json.dumps(self.meta, indent=1))- for cam, b in self.video.items():- (out / f"{cam}.mp4").write_bytes(b)- for cam, img in self.last.items():- (out / "last").mkdir(exist_ok=True)- Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)- for cam, arr in self.frames.items():- d = out / "frames" / cam- d.mkdir(parents=True, exist_ok=True)- for i, f in enumerate(arr):- Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)- for cam, arr in self.depth.items():- np.save(out / f"depth_{cam}.npy", arr)- if self.proprio is not None:- np.save(out / "proprio.npy", self.proprio)- return str(out)---class Robot:- def __init__(self, base: str | None = None, timeout: float = 600.0):- candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]- t0, last = time.time(), None- while time.time() - t0 < timeout:- for b in candidates:- try:- r = requests.get(b + "/spec", timeout=(5, 60))- if r.status_code == 503:- last = "warming up"- continue- r.raise_for_status()- self.base, self._spec = b, r.json()- if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):- raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")- return- except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:- last = e # not up yet (the scene takes about a minute), or busy- time.sleep(2)- raise RuntimeError(f"sim service unavailable: {last}")-- def spec(self) -> dict:- return self._spec-- def status(self) -> dict:- r = requests.get(self.base + "/status", timeout=(5, 60))- r.raise_for_status()- return r.json()-- def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,- video: str = "frames") -> Result:- meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),- "include_depth": bool(include_depth), "video": video}- return self._post("/observe", meta, [], timeout=900)-- def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",- at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",- include_depth: bool = False, note: str = "") -> Result:- a = np.asarray(actions, dtype=np.float64)- if a.ndim == 1:- a = a[None, :]- if not np.all(np.isfinite(a)):- raise ValueError("actions contain NaN or inf")- meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,- "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),- "video": video,- "include_depth": bool(include_depth), "note": note}- return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)-- def reset(self, cameras=None, include_depth: bool = False) -> Result:- return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)-- def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:- body = rcb1.encode(meta, parts)- if len(body) > MAX_BODY: # the service would close the connection on it unread- raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")- r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},- timeout=(5, timeout))- if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:- return Result(*rcb1.decode(r.content))- try:- err = r.json()- except ValueError:- err = {"error": r.text[:300]}- msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")- if r.status_code == 409:- raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")- + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))- if r.status_code in (400, 413):- raise ValueError(msg)- raise RuntimeError(msg)---def _names(cameras) -> list[str] | None:- return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb_realtime.pydeleted file mode 100644index 410284bf27..0000000000--- a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb_realtime.py+++ /dev/null@@ -1,479 +0,0 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.--It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery-and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:--- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its- first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then- on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,- until the episode ends; the clock stops there.-- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`- or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the- next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`- loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order- hold): a slow agent's robot keeps its last servo targets.-- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)- and returns what the cameras saw. It never changes the plan.-- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.--Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's-rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was-submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):-ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line-per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:-call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.--Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in-between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting-request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than-MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.--A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.-A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a-crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the-trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends-the graded episode (reason "error") sets infra_error in final.json too.-"""-from __future__ import annotations--import collections-import json-import os-import queue-import sys-import threading-import time-import traceback-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer-from urllib.parse import urlparse--import numpy as np--import rcb_service-from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request--PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1-MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up-CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)---class _Job:- """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""-- def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):- self.method, self.path, self.body, self.content_length = method, path, body, content_length- self.done = threading.Event()- self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)-- def finish(self, *reply) -> None:- self.reply = reply- self.done.set()---class _Watch:- """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""-- def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):- self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id- self.start, self.end = start, start + steps---class RealtimeService(Service):- """rcb_service.Service with the realtime clock (module docstring)."""-- def __init__(self, sim, *, slowdown: float | None = None, **kw):- self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)- if not self.slowdown > 0:- raise ValueError("RCB_SLOWDOWN must be > 0")- self.wall_dt = sim.control_dt * self.slowdown- self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be- self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step- self.running = False- self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t- self.watches: list[_Watch] = []- self.q: queue.Queue = queue.Queue()- self.max_lag = self.lag = 0.0- self.stalls = self.held_steps = self.late_steps = 0- self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk- super().__init__(sim, **kw)- self.ticks_path = os.path.join(self.data, "ticks.f64")- self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))-- # ------------------------------------------------------------------ spec, status, evidence- def _spec(self) -> dict:- spec = super()._spec()- spec["protocol"] = PROTOCOL- cdt = self.sim.control_dt- spec["timing"] = {- "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,- "wall_s_per_control_step": cdt * self.slowdown,- "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "- "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "- "you do, until the episode ends",- "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "- "`at` (default: the next one), replacing what was planned from there; rows for steps already "- "gone are dropped (meta.late_steps); after the plan's last row that target is held",- "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "- "the cameras saw; never changes the plan",- "max_ahead_steps": self.max_ahead,- "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}- return spec-- def _realtime_counters(self) -> dict:- return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}-- def status(self) -> dict:- return {**super().status(), **self._realtime_counters()}-- def meta(self) -> dict:- return {**super().meta(), **self._realtime_counters()}-- def _timing_record(self) -> dict:- return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),- "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}-- def _write_final(self) -> None:- self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's- if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)- self.verdict["infra_error"] = 1- else: # (an episode that erred and was reset away is not graded)- self.verdict.pop("infra_error", None)- super()._write_final()-- def _checkpoint(self, ev: str, **kw) -> None:- if self._ticks: # the tick log first: a crash in between leaves it too long- with open(self.ticks_path, "ab") as f:- f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())- self._ticks = []- super()._checkpoint(ev, **kw)-- def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:- keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",- "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")- line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}- with open(os.path.join(self.data, "calls.jsonl"), "a") as f:- f.write(json.dumps(line) + "\n")- if actions is not None:- rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)- with open(os.path.join(self.data, "submitted.f64"), "ab") as f:- f.write(np.asarray(rows, dtype=np.float64).tobytes())-- # ------------------------------------------------------------------ start, clock- def start(self) -> None:- super().start()- rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)- n = self.journal_rows- if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too- rows[: 2 * n].tofile(self.ticks_path)- elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)- np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)- self.running = False # a resumed episode waits, paused, for the agent's next call-- def _start_clock(self) -> None:- if not self.running and not self.done and not self.stopping:- self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)- self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)-- def _stop_clock(self, why: str) -> None:- if self.running:- self.running = False- self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))- self.plan.clear()-- def _end(self, reason: str) -> None:- super()._end(reason)- self._stop_clock(reason) # pending watches are completed by whoever ended it, after this-- def due_in(self, now: float) -> float | None:- """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""- if not self.running or self.done or self.stopping:- return None- return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now-- def tick(self) -> None:- """One control step of the clock: the plan's row for it, or the held target."""- now = time.monotonic()- lag = -(self.due_in(now) or 0.0)- if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt- self.stalls += 1- self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))- self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0- self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)- if self.plan:- typ, row, src = self.plan.popleft()- else:- (typ, row), src = self.sim.hold(), -1- self.held_steps += 1- before = self.journal_rows- self._ticks.append((time.time(), float(src)))- try:- self._step(typ, row)- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- if self.journal_rows == before:- self._ticks.pop()- self.log("error", path="clock", error=repr(exc)[:500])- self._end("error")- self._release_watches()- return- for w in list(self.watches):- w.rec.step()- if self.done or len(self.episode) >= w.end:- self._complete_watch(w)- if not self.done and len(self.episode) % self.ckpt_every == 0:- self._checkpoint("tick")-- # ------------------------------------------------------------------ requests (simulator thread)- def dispatch(self, job: _Job) -> None:- """Serve one queued request; a watch finishes later, from tick()."""- try:- self.first_request()- path = urlparse(job.path).path- if job.method == "GET" and path in ("/spec", "/status"):- return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))- routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}- if self.reset_budget:- routes["/reset"] = self._rt_reset- if job.method != "POST" or path not in routes:- return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))- if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:- return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))- try:- meta, parts = self._decode(job.body)- except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error- return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))- try:- routes[path](meta, parts, job)- except Refused as exc: # 400 (bad_request), 409: nothing changed- job.finish("raw", *self._reply_json(exc.code, exc.body))- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- self.log("error", path=path, error=repr(exc)[:500])- self._end("error")- self._release_watches()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))- except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever- traceback.print_exc()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))-- def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:- """2.1's _view, plus cameras = [] for no images."""- none = isinstance(meta.get("cameras"), list) and not meta["cameras"]- cams, fps, video, depth = self._view(meta, default_video)- return ([], fps, "none", False) if none else (cams, fps, video, depth)-- def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:- """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the- plan's last row (held), else -- nothing planned, or a step already run -- the current command."""- i = step - nxt- if i < 0 or not self.plan:- return np.asarray(self.sim.command(typ), dtype=np.float64)- return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)-- def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:- if self.done or self.stopping:- raise self._conflict()- nxt = len(self.episode)- with bad_request():- typ = meta.get("type")- if not isinstance(typ, str) or typ not in self.types:- raise ValueError(f"type must be one of {sorted(self.types)}")- try:- A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),- dtype=np.float64)- except (TypeError, ValueError, KeyError):- raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None- dim = int(self.types[typ]["dim"])- if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:- raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")- if not np.all(np.isfinite(A)):- raise ValueError("actions contain NaN or inf")- cdt = self.sim.control_dt- try:- dt = cdt if meta.get("dt") is None else float(meta["dt"])- except (TypeError, ValueError):- raise ValueError("dt must be a number of seconds") from None- k = round(dt / cdt) if np.isfinite(dt / cdt) else 0- if k < 1 or abs(k * cdt - dt) > 1e-9:- raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")- if k > 1 and not self.sim.interpolable(typ):- raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")- if len(A) * k * cdt > self.max_chunk_s + 1e-9:- raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "- f"more than max_chunk_s = {self.max_chunk_s:g}")- interp = meta.get("interpolate", "linear")- if interp not in ("linear", "hold"):- raise ValueError("interpolate must be linear or hold")- at = meta.get("at")- if at is not None:- if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:- raise ValueError("at must be a control step of this episode (an integer >= 0), "- "or null for the next one")- at = int(at)- if at - nxt > self.max_ahead:- raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "- f"of the next one ({nxt})")- at = nxt if at is None else at- cams, fps, video, depth = self._view_rt(meta, "none")- rows, clipped = self.sim.clip(typ, A)- prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`- steps = []- for row in rows:- for j in range(1, k + 1):- steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))- prev = row- late = max(0, nxt - at)- keep = at - nxt if at > nxt else 0- self.calls += 1- self.rows += len(A)- head = list(self.plan)[:keep]- if len(head) < keep: # a gap before `at`: the target in force then is held through it- fill = self._planned(at - 1, nxt, typ)- head += [(typ, fill, -1)] * (keep - len(head))- self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])- self.late_steps += min(late, len(steps))- self._start_clock()- rec = Recorder(self.sim, cams, fps, depth, "none")- rec.start()- rec.finish()- out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,- "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,- "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}- self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)- job.finish("rec", out, rec)-- def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:- with bad_request():- try:- seconds = float(meta.get("seconds") or 0.0)- except (TypeError, ValueError):- raise ValueError("seconds must be a number") from None- if not 0.0 <= seconds <= self.max_watch_s:- raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")- cams, fps, video, depth = self._view_rt(meta, "frames")- n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0- rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video- self._frame_budget(rec, n)- self.calls += 1- rec.start()- if not n:- rec.finish()- out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}- self._save_call(out, None)- return job.finish("rec", out, rec)- self._start_clock()- self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))-- def _complete_watch(self, w: _Watch) -> None:- self.watches.remove(w)- w.rec.finish()- ran = max(0, min(len(self.episode), w.end) - w.start)- out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),- "control_steps_watch": ran, "ok": True}- self._save_call(out, None)- w.job.finish("rec", out, w.rec)-- def _release_watches(self) -> None:- for w in list(self.watches):- self._complete_watch(w)-- def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:- self._release_watches()- out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)- self._stop_clock("reset")- job.finish("rcb1", {**out, **self._realtime_counters()}, prts)-- # ------------------------------------------------------------------ the loop (simulator thread) and HTTP- def poll(self) -> None:- """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""- if self.stopping and not self.finalized:- if not self.done:- self._end("timeout")- self._release_watches()- self._finalize()-- def loop(self, until: threading.Event | None = None) -> None:- """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""- while until is None or not until.is_set():- self.poll()- wait = self.due_in(time.monotonic())- if wait is not None and wait <= 0:- self.tick()- continue- try:- job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))- except queue.Empty:- continue- self.dispatch(job)-- def submit_job(self, job: _Job) -> tuple[int, str, bytes]:- """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""- self.q.put(job)- job.done.wait()- kind, *rest = job.reply- if kind == "raw":- return tuple(rest)- meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread- return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)-- def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:- svc = self-- class Handler(BaseHTTPRequestHandler):- timeout = 120-- def log_message(self, *a):- pass-- def _send(self, code: int, ctype: str, body: bytes) -> None:- self.send_response(code)- self.send_header("Content-Type", ctype)- self.send_header("Content-Length", str(len(body)))- self.end_headers()- self.wfile.write(body)-- def do_GET(self):- self._send(*svc.submit_job(_Job("GET", self.path, b"")))-- def do_POST(self):- try:- n = int(self.headers.get("Content-Length") or 0)- except ValueError:- return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))- if n > MAX_BODY: # refused unread: drop the connection with it- self.close_connection = True- return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))- self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))-- class Server(ThreadingHTTPServer):- daemon_threads = True-- return Server(address, Handler)-- def serve(self, port: int = 8000) -> None:- import signal- signal.signal(signal.SIGUSR1, self.on_sigusr1)- srv = self.bind(port)- threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()- print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)- self.loop()---def main(build, port: int = 8000) -> None:- """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of- Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""- rcb_service.Service = RealtimeService- rcb_service.PROTOCOL = PROTOCOL- rcb_service.main(build, port)---if __name__ == "__main__":- sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))- import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import-- main(server.build)diff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/README.md b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/README.mdsimilarity index 82%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/README.mdrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/README.mdindex a059a966c0..cdedfd4eec 100644--- a/tasks/humanoidbench-manip-powerlift-i00-realtime/README.md+++ b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-manip-powerlift-i00-realtime+# humanoidbench-manip-powerlift-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-manip-powerlift-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1hand` || Family / instance | `powerlift` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-manip-powerlift-i00-privileged`: the same frozen instance and the same success rule as it andas `humanoidbench-manip-powerlift-i00-standard`, but time does not stop. Once an episode's clock starts (itsfirst `submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the agent thinks,@@ -55,8 +55,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no dumbbell pose, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no dumbbell pose, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 61 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 88%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/docker-compose.yamlindex 1b53e88ca0..7e9eb019c9 100644--- a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/rcb_client.pynew file mode 100644index 0000000000..f2b0559bf9--- /dev/null+++ b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/rcb_client.py@@ -0,0 +1,209 @@+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+PYTHONPATH).++ from rcb_client import Robot, EpisodeOver+ r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready++TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or+get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x+spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a+plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out+the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.++Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):+ r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"+ ("joint_position": dim, layout (actuator order), lower/upper, range_rad,+ units, semantics), "control_dt" (0.02 s), "max_control_steps" (the+ episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:+ rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"+ (joints, axes, positions), "imu", "links" (masses, centres of mass, the+ feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"+ r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,+ planned_steps, lag_s (and episode, resets_left with a reset budget)+ r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")+ POST /observe. seconds=0: the robot now, at once. seconds>0 (at most+ spec()["max_watch_s"]): waits while that much robot time runs (starting the+ clock if it is not running) and returns what the cameras saw; it never+ changes the plan+ r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,+ video="none", include_depth=False, note="")+ POST /submit: answers at once with the robot as it is now (the rows+ have not run yet). actions (T, dim): rows dt apart (a multiple of+ control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's+ target is the row (interpolate="hold") or a straight line from the previous+ row's ("linear", starting from the target in force just before `at`). They+ become the plan from control step `at` (an episode step number; None = the+ next step), replacing whatever was planned from `at` on and keeping what is+ planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows+ for steps already gone when the request arrives are dropped+ (meta["late_steps"]). The first submit of an episode starts its clock+ r.reset(cameras=None, include_depth=False)+ POST /reset, only when spec()["budget"]["reset"] > 0: abandons the+ running episode and starts a new one, paused, from the same starting state;+ only the last episode is graded, and a success ends the trial++cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the+fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),+"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth+for the cameras that have it.++Result (what get_obs / submit / reset return):+ res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),+ sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |+ "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,+ lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]+ (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,+ resets_left+ res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])+ res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes+ res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)+ res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s+ of a watch, one at its end+ res.done the episode is over+ res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy++Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an+unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over+(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this+run (404).+"""+from __future__ import annotations++import json+import os+import pathlib+import time++import numpy as np+import requests++import rcb1++PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)+MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused+++class EpisodeOver(RuntimeError):+ pass+++class Result:+ """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""++ def __init__(self, meta: dict, parts: dict):+ self.meta, self.raw = meta, parts+ self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None+ for name, p in parts.items():+ kind, data = p["kind"], p["data"]+ if name.startswith("video/") and kind == "mp4":+ self.video[name[6:]] = data+ elif name.startswith("frames/"):+ self.frames[name[7:]] = data+ elif name.startswith("last/"):+ self.last[name[5:]] = data+ elif name.startswith("depth/"):+ self.depth[name[6:]] = data+ elif name == "proprio":+ self.proprio = data++ @property+ def done(self) -> bool:+ return bool(self.meta.get("done"))++ def save(self, out_dir: str) -> str:+ from PIL import Image+ out = pathlib.Path(out_dir)+ out.mkdir(parents=True, exist_ok=True)+ (out / "meta.json").write_text(json.dumps(self.meta, indent=1))+ for cam, b in self.video.items():+ (out / f"{cam}.mp4").write_bytes(b)+ for cam, img in self.last.items():+ (out / "last").mkdir(exist_ok=True)+ Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)+ for cam, arr in self.frames.items():+ d = out / "frames" / cam+ d.mkdir(parents=True, exist_ok=True)+ for i, f in enumerate(arr):+ Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)+ for cam, arr in self.depth.items():+ np.save(out / f"depth_{cam}.npy", arr)+ if self.proprio is not None:+ np.save(out / "proprio.npy", self.proprio)+ return str(out)+++class Robot:+ def __init__(self, base: str | None = None, timeout: float = 600.0):+ candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]+ t0, last = time.time(), None+ while time.time() - t0 < timeout:+ for b in candidates:+ try:+ r = requests.get(b + "/spec", timeout=(5, 60))+ if r.status_code == 503:+ last = "warming up"+ continue+ r.raise_for_status()+ self.base, self._spec = b, r.json()+ if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):+ raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")+ return+ except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:+ last = e # not up yet (the scene takes about a minute), or busy+ time.sleep(2)+ raise RuntimeError(f"sim service unavailable: {last}")++ def spec(self) -> dict:+ return self._spec++ def status(self) -> dict:+ r = requests.get(self.base + "/status", timeout=(5, 60))+ r.raise_for_status()+ return r.json()++ def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,+ video: str = "frames") -> Result:+ meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),+ "include_depth": bool(include_depth), "video": video}+ return self._post("/observe", meta, [], timeout=900)++ def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",+ at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",+ include_depth: bool = False, note: str = "") -> Result:+ a = np.asarray(actions, dtype=np.float64)+ if a.ndim == 1:+ a = a[None, :]+ if not np.all(np.isfinite(a)):+ raise ValueError("actions contain NaN or inf")+ meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,+ "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),+ "video": video,+ "include_depth": bool(include_depth), "note": note}+ return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)++ def reset(self, cameras=None, include_depth: bool = False) -> Result:+ return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)++ def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:+ body = rcb1.encode(meta, parts)+ if len(body) > MAX_BODY: # the service would close the connection on it unread+ raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")+ r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},+ timeout=(5, timeout))+ if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:+ return Result(*rcb1.decode(r.content))+ try:+ err = r.json()+ except ValueError:+ err = {"error": r.text[:300]}+ msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")+ if r.status_code == 409:+ raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")+ + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))+ if r.status_code in (400, 413):+ raise ValueError(msg)+ raise RuntimeError(msg)+++def _names(cameras) -> list[str] | None:+ return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb_realtime.pynew file mode 100644index 0000000000..e23f8cbd2c--- /dev/null+++ b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -0,0 +1,480 @@+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.++It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery+and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:++- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its+ first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then+ on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,+ until the episode ends; the clock stops there.+- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`+ or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the+ next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`+ loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order+ hold): a slow agent's robot keeps its last servo targets.+- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)+ and returns what the cameras saw. It never changes the plan.+- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.++Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's+rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was+submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):+ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line+per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:+call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.++Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in+between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting+request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than+MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.++A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.+A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a+crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the+trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends+the graded episode (reason "error") sets infra_error in final.json too.+"""+from __future__ import annotations++import collections+import json+import os+import queue+import sys+import threading+import time+import traceback+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from urllib.parse import urlparse++import numpy as np++import rcb_service+from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request++PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up+CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)+++class _Job:+ """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""++ def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):+ self.method, self.path, self.body, self.content_length = method, path, body, content_length+ self.done = threading.Event()+ self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)++ def finish(self, *reply) -> None:+ self.reply = reply+ self.done.set()+++class _Watch:+ """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""++ def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):+ self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id+ self.start, self.end = start, start + steps+++class RealtimeService(Service):+ """rcb_service.Service with the realtime clock (module docstring)."""++ def __init__(self, sim, *, slowdown: float | None = None, **kw):+ self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)+ if not self.slowdown > 0:+ raise ValueError("RCB_SLOWDOWN must be > 0")+ self.wall_dt = sim.control_dt * self.slowdown+ self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be+ self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step+ self.running = False+ self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t+ self.watches: list[_Watch] = []+ self.q: queue.Queue = queue.Queue()+ self.max_lag = self.lag = 0.0+ self.stalls = self.held_steps = self.late_steps = 0+ self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk+ super().__init__(sim, **kw)+ self.ticks_path = os.path.join(self.data, "ticks.f64")+ self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))++ # ------------------------------------------------------------------ spec, status, evidence+ def _spec(self) -> dict:+ spec = super()._spec()+ spec["protocol"] = PROTOCOL+ cdt = self.sim.control_dt+ spec["timing"] = {+ "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,+ "wall_s_per_control_step": cdt * self.slowdown,+ "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "+ "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "+ "you do, until the episode ends",+ "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "+ "`at` (default: the next one), replacing what was planned from there; rows for steps already "+ "gone are dropped (meta.late_steps); after the plan's last row that target is held",+ "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "+ "the cameras saw; never changes the plan",+ "max_ahead_steps": self.max_ahead,+ "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}+ return spec++ def _realtime_counters(self) -> dict:+ return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}++ def status(self) -> dict:+ return {**super().status(), **self._realtime_counters()}++ def meta(self) -> dict:+ return {**super().meta(), **self._realtime_counters()}++ def _timing_record(self) -> dict:+ return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),+ "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}++ def _write_final(self) -> None:+ self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's+ if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)+ self.verdict["infra_error"] = 1+ else: # (an episode that erred and was reset away is not graded)+ self.verdict.pop("infra_error", None)+ super()._write_final()++ def _checkpoint(self, ev: str, **kw) -> None:+ if self._ticks: # the tick log first: a crash in between leaves it too long+ with open(self.ticks_path, "ab") as f:+ f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())+ self._ticks = []+ super()._checkpoint(ev, **kw)++ def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:+ keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",+ "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")+ line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}+ with open(os.path.join(self.data, "calls.jsonl"), "a") as f:+ f.write(json.dumps(line) + "\n")+ if actions is not None:+ rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)+ with open(os.path.join(self.data, "submitted.f64"), "ab") as f:+ f.write(np.asarray(rows, dtype=np.float64).tobytes())++ # ------------------------------------------------------------------ start, clock+ def start(self) -> None:+ super().start()+ rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)+ n = self.journal_rows+ if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too+ rows[: 2 * n].tofile(self.ticks_path)+ elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)+ np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)+ self.running = False # a resumed episode waits, paused, for the agent's next call++ def _start_clock(self) -> None:+ if not self.running and not self.done and not self.stopping:+ self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)+ self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)++ def _stop_clock(self, why: str) -> None:+ if self.running:+ self.running = False+ self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))+ self.plan.clear()++ def _end(self, reason: str) -> None:+ super()._end(reason)+ self._stop_clock(reason) # pending watches are completed by whoever ended it, after this++ def due_in(self, now: float) -> float | None:+ """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""+ if not self.running or self.done or self.stopping:+ return None+ return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now++ def tick(self) -> None:+ """One control step of the clock: the plan's row for it, or the held target."""+ now = time.monotonic()+ lag = -(self.due_in(now) or 0.0)+ if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt+ self.stalls += 1+ self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))+ self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0+ self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)+ if self.plan:+ typ, row, src = self.plan.popleft()+ else:+ (typ, row), src = self.sim.hold(), -1+ self.held_steps += 1+ before = self.journal_rows+ self._ticks.append((time.time(), float(src)))+ try:+ self._step(typ, row)+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ if self.journal_rows == before:+ self._ticks.pop()+ self.log("error", path="clock", error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ return+ for w in list(self.watches):+ w.rec.step()+ if self.done or len(self.episode) >= w.end:+ self._complete_watch(w)+ if not self.done and len(self.episode) % self.ckpt_every == 0:+ self._checkpoint("tick")++ # ------------------------------------------------------------------ requests (simulator thread)+ def dispatch(self, job: _Job) -> None:+ """Serve one queued request; a watch finishes later, from tick()."""+ try:+ self.first_request()+ path = urlparse(job.path).path+ if job.method == "GET" and path in ("/spec", "/status"):+ return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))+ routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}+ if self.reset_budget:+ routes["/reset"] = self._rt_reset+ if job.method != "POST" or path not in routes:+ return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))+ if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:+ return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))+ try:+ meta, parts = self._decode(job.body)+ except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error+ return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))+ try:+ routes[path](meta, parts, job)+ except Refused as exc: # 400 (bad_request), 409: nothing changed+ job.finish("raw", *self._reply_json(exc.code, exc.body))+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ self.log("error", path=path, error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))+ except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever+ traceback.print_exc()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))++ def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:+ """2.1's _view, plus cameras = [] for no images."""+ none = isinstance(meta.get("cameras"), list) and not meta["cameras"]+ cams, fps, video, depth = self._view(meta, default_video)+ return ([], fps, "none", False) if none else (cams, fps, video, depth)++ def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:+ """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the+ plan's last row (held), else -- nothing planned, or a step already run -- the current command."""+ i = step - nxt+ if i < 0 or not self.plan:+ return np.asarray(self.sim.command(typ), dtype=np.float64)+ return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)++ def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:+ if self.done or self.stopping:+ raise self._conflict()+ nxt = len(self.episode)+ with bad_request():+ typ = meta.get("type")+ if not isinstance(typ, str) or typ not in self.types:+ raise ValueError(f"type must be one of {sorted(self.types)}")+ try:+ A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),+ dtype=np.float64)+ except (TypeError, ValueError, KeyError):+ raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None+ dim = int(self.types[typ]["dim"])+ if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:+ raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")+ if not np.all(np.isfinite(A)):+ raise ValueError("actions contain NaN or inf")+ cdt = self.sim.control_dt+ try:+ dt = cdt if meta.get("dt") is None else float(meta["dt"])+ except (TypeError, ValueError):+ raise ValueError("dt must be a number of seconds") from None+ k = round(dt / cdt) if np.isfinite(dt / cdt) else 0+ if k < 1 or abs(k * cdt - dt) > 1e-9:+ raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")+ if k > 1 and not self.sim.interpolable(typ):+ raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")+ if len(A) * k * cdt > self.max_chunk_s + 1e-9:+ raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "+ f"more than max_chunk_s = {self.max_chunk_s:g}")+ interp = meta.get("interpolate", "linear")+ if interp not in ("linear", "hold"):+ raise ValueError("interpolate must be linear or hold")+ at = meta.get("at")+ if at is not None:+ if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:+ raise ValueError("at must be a control step of this episode (an integer >= 0), "+ "or null for the next one")+ at = int(at)+ if at - nxt > self.max_ahead:+ raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "+ f"of the next one ({nxt})")+ at = nxt if at is None else at+ cams, fps, video, depth = self._view_rt(meta, "none")+ rows, clipped = self.sim.clip(typ, A)+ prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`+ steps = []+ for row in rows:+ for j in range(1, k + 1):+ steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))+ prev = row+ late = max(0, nxt - at)+ keep = at - nxt if at > nxt else 0+ self.calls += 1+ self.rows += len(A)+ head = list(self.plan)[:keep]+ if len(head) < keep: # a gap before `at`: the target in force then is held through it+ fill = self._planned(at - 1, nxt, typ)+ head += [(typ, fill, -1)] * (keep - len(head))+ self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])+ self.late_steps += min(late, len(steps))+ self._start_clock()+ rec = Recorder(self.sim, cams, fps, depth, "none")+ rec.start()+ rec.finish()+ out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,+ "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,+ "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}+ self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)+ job.finish("rec", out, rec)++ def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:+ with bad_request():+ try:+ seconds = float(meta.get("seconds") or 0.0)+ except (TypeError, ValueError):+ raise ValueError("seconds must be a number") from None+ if not 0.0 <= seconds <= self.max_watch_s:+ raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")+ cams, fps, video, depth = self._view_rt(meta, "frames")+ n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0+ rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video+ self._frame_budget(rec, n)+ self.calls += 1+ rec.start()+ if not n:+ rec.finish()+ out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}+ self._save_call(out, None)+ return job.finish("rec", out, rec)+ self._start_clock()+ self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))++ def _complete_watch(self, w: _Watch) -> None:+ self.watches.remove(w)+ w.rec.finish()+ ran = max(0, min(len(self.episode), w.end) - w.start)+ out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),+ "control_steps_watch": ran, "ok": True}+ self._save_call(out, None)+ w.job.finish("rec", out, w.rec)++ def _release_watches(self) -> None:+ for w in list(self.watches):+ self._complete_watch(w)++ def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:+ self._release_watches()+ out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)+ self._stop_clock("reset")+ job.finish("rcb1", {**out, **self._realtime_counters()}, prts)++ # ------------------------------------------------------------------ the loop (simulator thread) and HTTP+ def poll(self) -> None:+ """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""+ if self.stopping and not self.finalized:+ if not self.done:+ self._end("timeout")+ self._release_watches()+ self._finalize()++ def loop(self, until: threading.Event | None = None) -> None:+ """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""+ while until is None or not until.is_set():+ self.poll()+ wait = self.due_in(time.monotonic())+ if wait is not None and wait <= 0:+ self.tick()+ continue+ try:+ job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))+ except queue.Empty:+ continue+ self.dispatch(job)++ def submit_job(self, job: _Job) -> tuple[int, str, bytes]:+ """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""+ self.q.put(job)+ job.done.wait()+ kind, *rest = job.reply+ if kind == "raw":+ return tuple(rest)+ meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread+ return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)++ def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:+ svc = self++ class Handler(BaseHTTPRequestHandler):+ timeout = 120++ def log_message(self, *a):+ pass++ def _send(self, code: int, ctype: str, body: bytes) -> None:+ self.send_response(code)+ self.send_header("Content-Type", ctype)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_GET(self):+ self._send(*svc.submit_job(_Job("GET", self.path, b"")))++ def do_POST(self):+ try:+ n = int(self.headers.get("Content-Length") or 0)+ except ValueError:+ return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))+ if n > MAX_BODY: # refused unread: drop the connection with it+ self.close_connection = True+ return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))+ self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))++ class Server(ThreadingHTTPServer):+ daemon_threads = True++ return Server(address, Handler)++ def serve(self, port: int = 8000) -> None:+ import signal+ signal.signal(signal.SIGUSR1, self.on_sigusr1)+ srv = self.bind(port)+ threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()+ print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)+ self.loop()+++def main(build, port: int = 8000) -> None:+ """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of+ Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""+ rcb_service.Service = RealtimeService+ rcb_service.PROTOCOL = PROTOCOL+ rcb_service.main(build, port)+++if __name__ == "__main__":+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+ import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import++ main(server.build)diff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/instruction.md b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/instruction.mdrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/instruction.mdindex 390c7b8fc7..3d3690200a 100644--- a/tasks/humanoidbench-manip-powerlift-i00-realtime/instruction.md+++ b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# powerlift — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# powerlift — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid with two Shadow dexterous hands (HumanoidBench / MuJoCo) **as ifit were real hardware**: the simulator runs in a service you cannot open, and you reach it only through a@@ -74,7 +74,7 @@ the moment the pass mark is reached (`reason = "success"`), and a fresh replay oservo targets the robot actually executed at each control step, must reach it as well and end in the samestate.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/task.toml b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/task.tomlsimilarity index 70%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/task.tomlrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/task.tomlindex 034db0b0e2..9d2de99aaf 100644--- a/tasks/humanoidbench-manip-powerlift-i00-realtime/task.toml+++ b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of humanoidbench-manip-powerlift-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of humanoidbench-manip-powerlift-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,10 +16,10 @@ artifacts = [][task]-name = "robot-coding-bench/humanoidbench-manip-powerlift-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench powerlift on the same frozen instance as humanoidbench-manip-powerlift-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/humanoidbench-manip-powerlift-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench powerlift on the same frozen instance as humanoidbench-manip-powerlift-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "realtime-mode", "trajectory", "powerlift"]+keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "standard-realtime-mode", "trajectory", "powerlift"][metadata]author_name = "pingyue"@@ -27,7 +27,7 @@ task_type = "trajectory"simulator = "humanoidbench"robot = "unitree-h1hand"family = "powerlift"-mode = "realtime"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = 0expert_time_estimate_hours = 8.0diff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/tests/grade.py b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-manip-powerlift-i00-realtime/tests/test.sh b/tasks/humanoidbench-manip-powerlift-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-manip-powerlift-i00-realtime/tests/test.shrename to tasks/humanoidbench-manip-powerlift-i00-standard-realtime/tests/test.shdiff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-room-i00-realtime/environment/rcb_client.pydeleted file mode 100644index 8e057bb6e9..0000000000--- a/tasks/humanoidbench-manip-room-i00-realtime/environment/rcb_client.py+++ /dev/null@@ -1,209 +0,0 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on-PYTHONPATH).-- from rcb_client import Robot, EpisodeOver- r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready--TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or-get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x-spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a-plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out-the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.--Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):- r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"- ("joint_position": dim, layout (actuator order), lower/upper, range_rad,- units, semantics), "control_dt" (0.02 s), "max_control_steps" (the- episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:- rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"- (joints, axes, positions), "imu", "links" (masses, centres of mass, the- feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"- r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,- planned_steps, lag_s (and episode, resets_left with a reset budget)- r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")- POST /observe. seconds=0: the robot now, at once. seconds>0 (at most- spec()["max_watch_s"]): waits while that much robot time runs (starting the- clock if it is not running) and returns what the cameras saw; it never- changes the plan- r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,- video="none", include_depth=False, note="")- POST /submit: answers at once with the robot as it is now (the rows- have not run yet). actions (T, dim): rows dt apart (a multiple of- control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's- target is the row (interpolate="hold") or a straight line from the previous- row's ("linear", starting from the target in force just before `at`). They- become the plan from control step `at` (an episode step number; None = the- next step), replacing whatever was planned from `at` on and keeping what is- planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows- for steps already gone when the request arrives are dropped- (meta["late_steps"]). The first submit of an episode starts its clock- r.reset(cameras=None, include_depth=False)- POST /reset, only when spec()["budget"]["reset"] > 0: abandons the- running episode and starts a new one, paused, from the same starting state;- only the last episode is graded, and a success ends the trial--cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the-fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),-"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth-for the cameras that have it.--Result (what get_obs / submit / reset return):- res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),- sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |- "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,- lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]- (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,- resets_left- res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])- res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes- res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)- res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s- of a watch, one at its end- res.done the episode is over- res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy--Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an-unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over-(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this-run (404).-"""-from __future__ import annotations--import json-import os-import pathlib-import time--import numpy as np-import requests--import rcb1--PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)-MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused---class EpisodeOver(RuntimeError):- pass---class Result:- """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""-- def __init__(self, meta: dict, parts: dict):- self.meta, self.raw = meta, parts- self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None- for name, p in parts.items():- kind, data = p["kind"], p["data"]- if name.startswith("video/") and kind == "mp4":- self.video[name[6:]] = data- elif name.startswith("frames/"):- self.frames[name[7:]] = data- elif name.startswith("last/"):- self.last[name[5:]] = data- elif name.startswith("depth/"):- self.depth[name[6:]] = data- elif name == "proprio":- self.proprio = data-- @property- def done(self) -> bool:- return bool(self.meta.get("done"))-- def save(self, out_dir: str) -> str:- from PIL import Image- out = pathlib.Path(out_dir)- out.mkdir(parents=True, exist_ok=True)- (out / "meta.json").write_text(json.dumps(self.meta, indent=1))- for cam, b in self.video.items():- (out / f"{cam}.mp4").write_bytes(b)- for cam, img in self.last.items():- (out / "last").mkdir(exist_ok=True)- Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)- for cam, arr in self.frames.items():- d = out / "frames" / cam- d.mkdir(parents=True, exist_ok=True)- for i, f in enumerate(arr):- Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)- for cam, arr in self.depth.items():- np.save(out / f"depth_{cam}.npy", arr)- if self.proprio is not None:- np.save(out / "proprio.npy", self.proprio)- return str(out)---class Robot:- def __init__(self, base: str | None = None, timeout: float = 600.0):- candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]- t0, last = time.time(), None- while time.time() - t0 < timeout:- for b in candidates:- try:- r = requests.get(b + "/spec", timeout=(5, 60))- if r.status_code == 503:- last = "warming up"- continue- r.raise_for_status()- self.base, self._spec = b, r.json()- if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):- raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")- return- except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:- last = e # not up yet (the scene takes about a minute), or busy- time.sleep(2)- raise RuntimeError(f"sim service unavailable: {last}")-- def spec(self) -> dict:- return self._spec-- def status(self) -> dict:- r = requests.get(self.base + "/status", timeout=(5, 60))- r.raise_for_status()- return r.json()-- def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,- video: str = "frames") -> Result:- meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),- "include_depth": bool(include_depth), "video": video}- return self._post("/observe", meta, [], timeout=900)-- def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",- at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",- include_depth: bool = False, note: str = "") -> Result:- a = np.asarray(actions, dtype=np.float64)- if a.ndim == 1:- a = a[None, :]- if not np.all(np.isfinite(a)):- raise ValueError("actions contain NaN or inf")- meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,- "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),- "video": video,- "include_depth": bool(include_depth), "note": note}- return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)-- def reset(self, cameras=None, include_depth: bool = False) -> Result:- return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)-- def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:- body = rcb1.encode(meta, parts)- if len(body) > MAX_BODY: # the service would close the connection on it unread- raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")- r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},- timeout=(5, timeout))- if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:- return Result(*rcb1.decode(r.content))- try:- err = r.json()- except ValueError:- err = {"error": r.text[:300]}- msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")- if r.status_code == 409:- raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")- + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))- if r.status_code in (400, 413):- raise ValueError(msg)- raise RuntimeError(msg)---def _names(cameras) -> list[str] | None:- return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb_realtime.pydeleted file mode 100644index 410284bf27..0000000000--- a/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb_realtime.py+++ /dev/null@@ -1,479 +0,0 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.--It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery-and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:--- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its- first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then- on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,- until the episode ends; the clock stops there.-- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`- or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the- next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`- loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order- hold): a slow agent's robot keeps its last servo targets.-- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)- and returns what the cameras saw. It never changes the plan.-- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.--Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's-rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was-submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):-ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line-per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:-call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.--Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in-between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting-request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than-MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.--A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.-A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a-crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the-trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends-the graded episode (reason "error") sets infra_error in final.json too.-"""-from __future__ import annotations--import collections-import json-import os-import queue-import sys-import threading-import time-import traceback-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer-from urllib.parse import urlparse--import numpy as np--import rcb_service-from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request--PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1-MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up-CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)---class _Job:- """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""-- def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):- self.method, self.path, self.body, self.content_length = method, path, body, content_length- self.done = threading.Event()- self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)-- def finish(self, *reply) -> None:- self.reply = reply- self.done.set()---class _Watch:- """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""-- def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):- self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id- self.start, self.end = start, start + steps---class RealtimeService(Service):- """rcb_service.Service with the realtime clock (module docstring)."""-- def __init__(self, sim, *, slowdown: float | None = None, **kw):- self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)- if not self.slowdown > 0:- raise ValueError("RCB_SLOWDOWN must be > 0")- self.wall_dt = sim.control_dt * self.slowdown- self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be- self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step- self.running = False- self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t- self.watches: list[_Watch] = []- self.q: queue.Queue = queue.Queue()- self.max_lag = self.lag = 0.0- self.stalls = self.held_steps = self.late_steps = 0- self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk- super().__init__(sim, **kw)- self.ticks_path = os.path.join(self.data, "ticks.f64")- self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))-- # ------------------------------------------------------------------ spec, status, evidence- def _spec(self) -> dict:- spec = super()._spec()- spec["protocol"] = PROTOCOL- cdt = self.sim.control_dt- spec["timing"] = {- "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,- "wall_s_per_control_step": cdt * self.slowdown,- "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "- "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "- "you do, until the episode ends",- "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "- "`at` (default: the next one), replacing what was planned from there; rows for steps already "- "gone are dropped (meta.late_steps); after the plan's last row that target is held",- "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "- "the cameras saw; never changes the plan",- "max_ahead_steps": self.max_ahead,- "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}- return spec-- def _realtime_counters(self) -> dict:- return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}-- def status(self) -> dict:- return {**super().status(), **self._realtime_counters()}-- def meta(self) -> dict:- return {**super().meta(), **self._realtime_counters()}-- def _timing_record(self) -> dict:- return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),- "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}-- def _write_final(self) -> None:- self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's- if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)- self.verdict["infra_error"] = 1- else: # (an episode that erred and was reset away is not graded)- self.verdict.pop("infra_error", None)- super()._write_final()-- def _checkpoint(self, ev: str, **kw) -> None:- if self._ticks: # the tick log first: a crash in between leaves it too long- with open(self.ticks_path, "ab") as f:- f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())- self._ticks = []- super()._checkpoint(ev, **kw)-- def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:- keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",- "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")- line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}- with open(os.path.join(self.data, "calls.jsonl"), "a") as f:- f.write(json.dumps(line) + "\n")- if actions is not None:- rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)- with open(os.path.join(self.data, "submitted.f64"), "ab") as f:- f.write(np.asarray(rows, dtype=np.float64).tobytes())-- # ------------------------------------------------------------------ start, clock- def start(self) -> None:- super().start()- rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)- n = self.journal_rows- if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too- rows[: 2 * n].tofile(self.ticks_path)- elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)- np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)- self.running = False # a resumed episode waits, paused, for the agent's next call-- def _start_clock(self) -> None:- if not self.running and not self.done and not self.stopping:- self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)- self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)-- def _stop_clock(self, why: str) -> None:- if self.running:- self.running = False- self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))- self.plan.clear()-- def _end(self, reason: str) -> None:- super()._end(reason)- self._stop_clock(reason) # pending watches are completed by whoever ended it, after this-- def due_in(self, now: float) -> float | None:- """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""- if not self.running or self.done or self.stopping:- return None- return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now-- def tick(self) -> None:- """One control step of the clock: the plan's row for it, or the held target."""- now = time.monotonic()- lag = -(self.due_in(now) or 0.0)- if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt- self.stalls += 1- self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))- self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0- self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)- if self.plan:- typ, row, src = self.plan.popleft()- else:- (typ, row), src = self.sim.hold(), -1- self.held_steps += 1- before = self.journal_rows- self._ticks.append((time.time(), float(src)))- try:- self._step(typ, row)- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- if self.journal_rows == before:- self._ticks.pop()- self.log("error", path="clock", error=repr(exc)[:500])- self._end("error")- self._release_watches()- return- for w in list(self.watches):- w.rec.step()- if self.done or len(self.episode) >= w.end:- self._complete_watch(w)- if not self.done and len(self.episode) % self.ckpt_every == 0:- self._checkpoint("tick")-- # ------------------------------------------------------------------ requests (simulator thread)- def dispatch(self, job: _Job) -> None:- """Serve one queued request; a watch finishes later, from tick()."""- try:- self.first_request()- path = urlparse(job.path).path- if job.method == "GET" and path in ("/spec", "/status"):- return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))- routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}- if self.reset_budget:- routes["/reset"] = self._rt_reset- if job.method != "POST" or path not in routes:- return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))- if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:- return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))- try:- meta, parts = self._decode(job.body)- except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error- return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))- try:- routes[path](meta, parts, job)- except Refused as exc: # 400 (bad_request), 409: nothing changed- job.finish("raw", *self._reply_json(exc.code, exc.body))- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- self.log("error", path=path, error=repr(exc)[:500])- self._end("error")- self._release_watches()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))- except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever- traceback.print_exc()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))-- def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:- """2.1's _view, plus cameras = [] for no images."""- none = isinstance(meta.get("cameras"), list) and not meta["cameras"]- cams, fps, video, depth = self._view(meta, default_video)- return ([], fps, "none", False) if none else (cams, fps, video, depth)-- def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:- """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the- plan's last row (held), else -- nothing planned, or a step already run -- the current command."""- i = step - nxt- if i < 0 or not self.plan:- return np.asarray(self.sim.command(typ), dtype=np.float64)- return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)-- def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:- if self.done or self.stopping:- raise self._conflict()- nxt = len(self.episode)- with bad_request():- typ = meta.get("type")- if not isinstance(typ, str) or typ not in self.types:- raise ValueError(f"type must be one of {sorted(self.types)}")- try:- A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),- dtype=np.float64)- except (TypeError, ValueError, KeyError):- raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None- dim = int(self.types[typ]["dim"])- if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:- raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")- if not np.all(np.isfinite(A)):- raise ValueError("actions contain NaN or inf")- cdt = self.sim.control_dt- try:- dt = cdt if meta.get("dt") is None else float(meta["dt"])- except (TypeError, ValueError):- raise ValueError("dt must be a number of seconds") from None- k = round(dt / cdt) if np.isfinite(dt / cdt) else 0- if k < 1 or abs(k * cdt - dt) > 1e-9:- raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")- if k > 1 and not self.sim.interpolable(typ):- raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")- if len(A) * k * cdt > self.max_chunk_s + 1e-9:- raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "- f"more than max_chunk_s = {self.max_chunk_s:g}")- interp = meta.get("interpolate", "linear")- if interp not in ("linear", "hold"):- raise ValueError("interpolate must be linear or hold")- at = meta.get("at")- if at is not None:- if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:- raise ValueError("at must be a control step of this episode (an integer >= 0), "- "or null for the next one")- at = int(at)- if at - nxt > self.max_ahead:- raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "- f"of the next one ({nxt})")- at = nxt if at is None else at- cams, fps, video, depth = self._view_rt(meta, "none")- rows, clipped = self.sim.clip(typ, A)- prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`- steps = []- for row in rows:- for j in range(1, k + 1):- steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))- prev = row- late = max(0, nxt - at)- keep = at - nxt if at > nxt else 0- self.calls += 1- self.rows += len(A)- head = list(self.plan)[:keep]- if len(head) < keep: # a gap before `at`: the target in force then is held through it- fill = self._planned(at - 1, nxt, typ)- head += [(typ, fill, -1)] * (keep - len(head))- self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])- self.late_steps += min(late, len(steps))- self._start_clock()- rec = Recorder(self.sim, cams, fps, depth, "none")- rec.start()- rec.finish()- out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,- "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,- "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}- self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)- job.finish("rec", out, rec)-- def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:- with bad_request():- try:- seconds = float(meta.get("seconds") or 0.0)- except (TypeError, ValueError):- raise ValueError("seconds must be a number") from None- if not 0.0 <= seconds <= self.max_watch_s:- raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")- cams, fps, video, depth = self._view_rt(meta, "frames")- n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0- rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video- self._frame_budget(rec, n)- self.calls += 1- rec.start()- if not n:- rec.finish()- out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}- self._save_call(out, None)- return job.finish("rec", out, rec)- self._start_clock()- self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))-- def _complete_watch(self, w: _Watch) -> None:- self.watches.remove(w)- w.rec.finish()- ran = max(0, min(len(self.episode), w.end) - w.start)- out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),- "control_steps_watch": ran, "ok": True}- self._save_call(out, None)- w.job.finish("rec", out, w.rec)-- def _release_watches(self) -> None:- for w in list(self.watches):- self._complete_watch(w)-- def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:- self._release_watches()- out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)- self._stop_clock("reset")- job.finish("rcb1", {**out, **self._realtime_counters()}, prts)-- # ------------------------------------------------------------------ the loop (simulator thread) and HTTP- def poll(self) -> None:- """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""- if self.stopping and not self.finalized:- if not self.done:- self._end("timeout")- self._release_watches()- self._finalize()-- def loop(self, until: threading.Event | None = None) -> None:- """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""- while until is None or not until.is_set():- self.poll()- wait = self.due_in(time.monotonic())- if wait is not None and wait <= 0:- self.tick()- continue- try:- job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))- except queue.Empty:- continue- self.dispatch(job)-- def submit_job(self, job: _Job) -> tuple[int, str, bytes]:- """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""- self.q.put(job)- job.done.wait()- kind, *rest = job.reply- if kind == "raw":- return tuple(rest)- meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread- return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)-- def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:- svc = self-- class Handler(BaseHTTPRequestHandler):- timeout = 120-- def log_message(self, *a):- pass-- def _send(self, code: int, ctype: str, body: bytes) -> None:- self.send_response(code)- self.send_header("Content-Type", ctype)- self.send_header("Content-Length", str(len(body)))- self.end_headers()- self.wfile.write(body)-- def do_GET(self):- self._send(*svc.submit_job(_Job("GET", self.path, b"")))-- def do_POST(self):- try:- n = int(self.headers.get("Content-Length") or 0)- except ValueError:- return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))- if n > MAX_BODY: # refused unread: drop the connection with it- self.close_connection = True- return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))- self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))-- class Server(ThreadingHTTPServer):- daemon_threads = True-- return Server(address, Handler)-- def serve(self, port: int = 8000) -> None:- import signal- signal.signal(signal.SIGUSR1, self.on_sigusr1)- srv = self.bind(port)- threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()- print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)- self.loop()---def main(build, port: int = 8000) -> None:- """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of- Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""- rcb_service.Service = RealtimeService- rcb_service.PROTOCOL = PROTOCOL- rcb_service.main(build, port)---if __name__ == "__main__":- sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))- import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import-- main(server.build)diff --git a/tasks/humanoidbench-manip-room-i00-realtime/README.md b/tasks/humanoidbench-manip-room-i00-standard-realtime/README.mdsimilarity index 82%rename from tasks/humanoidbench-manip-room-i00-realtime/README.mdrename to tasks/humanoidbench-manip-room-i00-standard-realtime/README.mdindex 3877e6cb7f..238efc4ac2 100644--- a/tasks/humanoidbench-manip-room-i00-realtime/README.md+++ b/tasks/humanoidbench-manip-room-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-manip-room-i00-realtime+# humanoidbench-manip-room-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-manip-room-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1hand` || Family / instance | `room` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-manip-room-i00-privileged`: the same frozen instance and the same success rule as it and as`humanoidbench-manip-room-i00-standard`, but time does not stop. Once an episode's clock starts (its first`submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the agent thinks, so@@ -55,8 +55,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no object positions, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no object positions, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 61 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 90%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/docker-compose.yamlindex d36fe82325..2a67c7a1f6 100644--- a/tasks/humanoidbench-manip-room-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/rcb_client.pynew file mode 100644index 0000000000..f2b0559bf9--- /dev/null+++ b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/rcb_client.py@@ -0,0 +1,209 @@+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+PYTHONPATH).++ from rcb_client import Robot, EpisodeOver+ r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready++TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or+get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x+spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a+plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out+the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.++Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):+ r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"+ ("joint_position": dim, layout (actuator order), lower/upper, range_rad,+ units, semantics), "control_dt" (0.02 s), "max_control_steps" (the+ episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:+ rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"+ (joints, axes, positions), "imu", "links" (masses, centres of mass, the+ feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"+ r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,+ planned_steps, lag_s (and episode, resets_left with a reset budget)+ r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")+ POST /observe. seconds=0: the robot now, at once. seconds>0 (at most+ spec()["max_watch_s"]): waits while that much robot time runs (starting the+ clock if it is not running) and returns what the cameras saw; it never+ changes the plan+ r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,+ video="none", include_depth=False, note="")+ POST /submit: answers at once with the robot as it is now (the rows+ have not run yet). actions (T, dim): rows dt apart (a multiple of+ control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's+ target is the row (interpolate="hold") or a straight line from the previous+ row's ("linear", starting from the target in force just before `at`). They+ become the plan from control step `at` (an episode step number; None = the+ next step), replacing whatever was planned from `at` on and keeping what is+ planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows+ for steps already gone when the request arrives are dropped+ (meta["late_steps"]). The first submit of an episode starts its clock+ r.reset(cameras=None, include_depth=False)+ POST /reset, only when spec()["budget"]["reset"] > 0: abandons the+ running episode and starts a new one, paused, from the same starting state;+ only the last episode is graded, and a success ends the trial++cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the+fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),+"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth+for the cameras that have it.++Result (what get_obs / submit / reset return):+ res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),+ sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |+ "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,+ lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]+ (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,+ resets_left+ res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])+ res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes+ res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)+ res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s+ of a watch, one at its end+ res.done the episode is over+ res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy++Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an+unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over+(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this+run (404).+"""+from __future__ import annotations++import json+import os+import pathlib+import time++import numpy as np+import requests++import rcb1++PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)+MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused+++class EpisodeOver(RuntimeError):+ pass+++class Result:+ """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""++ def __init__(self, meta: dict, parts: dict):+ self.meta, self.raw = meta, parts+ self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None+ for name, p in parts.items():+ kind, data = p["kind"], p["data"]+ if name.startswith("video/") and kind == "mp4":+ self.video[name[6:]] = data+ elif name.startswith("frames/"):+ self.frames[name[7:]] = data+ elif name.startswith("last/"):+ self.last[name[5:]] = data+ elif name.startswith("depth/"):+ self.depth[name[6:]] = data+ elif name == "proprio":+ self.proprio = data++ @property+ def done(self) -> bool:+ return bool(self.meta.get("done"))++ def save(self, out_dir: str) -> str:+ from PIL import Image+ out = pathlib.Path(out_dir)+ out.mkdir(parents=True, exist_ok=True)+ (out / "meta.json").write_text(json.dumps(self.meta, indent=1))+ for cam, b in self.video.items():+ (out / f"{cam}.mp4").write_bytes(b)+ for cam, img in self.last.items():+ (out / "last").mkdir(exist_ok=True)+ Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)+ for cam, arr in self.frames.items():+ d = out / "frames" / cam+ d.mkdir(parents=True, exist_ok=True)+ for i, f in enumerate(arr):+ Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)+ for cam, arr in self.depth.items():+ np.save(out / f"depth_{cam}.npy", arr)+ if self.proprio is not None:+ np.save(out / "proprio.npy", self.proprio)+ return str(out)+++class Robot:+ def __init__(self, base: str | None = None, timeout: float = 600.0):+ candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]+ t0, last = time.time(), None+ while time.time() - t0 < timeout:+ for b in candidates:+ try:+ r = requests.get(b + "/spec", timeout=(5, 60))+ if r.status_code == 503:+ last = "warming up"+ continue+ r.raise_for_status()+ self.base, self._spec = b, r.json()+ if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):+ raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")+ return+ except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:+ last = e # not up yet (the scene takes about a minute), or busy+ time.sleep(2)+ raise RuntimeError(f"sim service unavailable: {last}")++ def spec(self) -> dict:+ return self._spec++ def status(self) -> dict:+ r = requests.get(self.base + "/status", timeout=(5, 60))+ r.raise_for_status()+ return r.json()++ def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,+ video: str = "frames") -> Result:+ meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),+ "include_depth": bool(include_depth), "video": video}+ return self._post("/observe", meta, [], timeout=900)++ def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",+ at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",+ include_depth: bool = False, note: str = "") -> Result:+ a = np.asarray(actions, dtype=np.float64)+ if a.ndim == 1:+ a = a[None, :]+ if not np.all(np.isfinite(a)):+ raise ValueError("actions contain NaN or inf")+ meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,+ "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),+ "video": video,+ "include_depth": bool(include_depth), "note": note}+ return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)++ def reset(self, cameras=None, include_depth: bool = False) -> Result:+ return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)++ def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:+ body = rcb1.encode(meta, parts)+ if len(body) > MAX_BODY: # the service would close the connection on it unread+ raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")+ r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},+ timeout=(5, timeout))+ if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:+ return Result(*rcb1.decode(r.content))+ try:+ err = r.json()+ except ValueError:+ err = {"error": r.text[:300]}+ msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")+ if r.status_code == 409:+ raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")+ + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))+ if r.status_code in (400, 413):+ raise ValueError(msg)+ raise RuntimeError(msg)+++def _names(cameras) -> list[str] | None:+ return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb_realtime.pynew file mode 100644index 0000000000..e23f8cbd2c--- /dev/null+++ b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -0,0 +1,480 @@+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.++It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery+and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:++- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its+ first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then+ on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,+ until the episode ends; the clock stops there.+- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`+ or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the+ next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`+ loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order+ hold): a slow agent's robot keeps its last servo targets.+- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)+ and returns what the cameras saw. It never changes the plan.+- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.++Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's+rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was+submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):+ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line+per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:+call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.++Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in+between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting+request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than+MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.++A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.+A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a+crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the+trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends+the graded episode (reason "error") sets infra_error in final.json too.+"""+from __future__ import annotations++import collections+import json+import os+import queue+import sys+import threading+import time+import traceback+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from urllib.parse import urlparse++import numpy as np++import rcb_service+from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request++PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up+CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)+++class _Job:+ """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""++ def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):+ self.method, self.path, self.body, self.content_length = method, path, body, content_length+ self.done = threading.Event()+ self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)++ def finish(self, *reply) -> None:+ self.reply = reply+ self.done.set()+++class _Watch:+ """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""++ def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):+ self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id+ self.start, self.end = start, start + steps+++class RealtimeService(Service):+ """rcb_service.Service with the realtime clock (module docstring)."""++ def __init__(self, sim, *, slowdown: float | None = None, **kw):+ self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)+ if not self.slowdown > 0:+ raise ValueError("RCB_SLOWDOWN must be > 0")+ self.wall_dt = sim.control_dt * self.slowdown+ self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be+ self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step+ self.running = False+ self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t+ self.watches: list[_Watch] = []+ self.q: queue.Queue = queue.Queue()+ self.max_lag = self.lag = 0.0+ self.stalls = self.held_steps = self.late_steps = 0+ self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk+ super().__init__(sim, **kw)+ self.ticks_path = os.path.join(self.data, "ticks.f64")+ self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))++ # ------------------------------------------------------------------ spec, status, evidence+ def _spec(self) -> dict:+ spec = super()._spec()+ spec["protocol"] = PROTOCOL+ cdt = self.sim.control_dt+ spec["timing"] = {+ "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,+ "wall_s_per_control_step": cdt * self.slowdown,+ "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "+ "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "+ "you do, until the episode ends",+ "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "+ "`at` (default: the next one), replacing what was planned from there; rows for steps already "+ "gone are dropped (meta.late_steps); after the plan's last row that target is held",+ "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "+ "the cameras saw; never changes the plan",+ "max_ahead_steps": self.max_ahead,+ "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}+ return spec++ def _realtime_counters(self) -> dict:+ return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}++ def status(self) -> dict:+ return {**super().status(), **self._realtime_counters()}++ def meta(self) -> dict:+ return {**super().meta(), **self._realtime_counters()}++ def _timing_record(self) -> dict:+ return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),+ "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}++ def _write_final(self) -> None:+ self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's+ if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)+ self.verdict["infra_error"] = 1+ else: # (an episode that erred and was reset away is not graded)+ self.verdict.pop("infra_error", None)+ super()._write_final()++ def _checkpoint(self, ev: str, **kw) -> None:+ if self._ticks: # the tick log first: a crash in between leaves it too long+ with open(self.ticks_path, "ab") as f:+ f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())+ self._ticks = []+ super()._checkpoint(ev, **kw)++ def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:+ keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",+ "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")+ line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}+ with open(os.path.join(self.data, "calls.jsonl"), "a") as f:+ f.write(json.dumps(line) + "\n")+ if actions is not None:+ rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)+ with open(os.path.join(self.data, "submitted.f64"), "ab") as f:+ f.write(np.asarray(rows, dtype=np.float64).tobytes())++ # ------------------------------------------------------------------ start, clock+ def start(self) -> None:+ super().start()+ rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)+ n = self.journal_rows+ if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too+ rows[: 2 * n].tofile(self.ticks_path)+ elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)+ np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)+ self.running = False # a resumed episode waits, paused, for the agent's next call++ def _start_clock(self) -> None:+ if not self.running and not self.done and not self.stopping:+ self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)+ self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)++ def _stop_clock(self, why: str) -> None:+ if self.running:+ self.running = False+ self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))+ self.plan.clear()++ def _end(self, reason: str) -> None:+ super()._end(reason)+ self._stop_clock(reason) # pending watches are completed by whoever ended it, after this++ def due_in(self, now: float) -> float | None:+ """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""+ if not self.running or self.done or self.stopping:+ return None+ return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now++ def tick(self) -> None:+ """One control step of the clock: the plan's row for it, or the held target."""+ now = time.monotonic()+ lag = -(self.due_in(now) or 0.0)+ if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt+ self.stalls += 1+ self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))+ self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0+ self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)+ if self.plan:+ typ, row, src = self.plan.popleft()+ else:+ (typ, row), src = self.sim.hold(), -1+ self.held_steps += 1+ before = self.journal_rows+ self._ticks.append((time.time(), float(src)))+ try:+ self._step(typ, row)+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ if self.journal_rows == before:+ self._ticks.pop()+ self.log("error", path="clock", error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ return+ for w in list(self.watches):+ w.rec.step()+ if self.done or len(self.episode) >= w.end:+ self._complete_watch(w)+ if not self.done and len(self.episode) % self.ckpt_every == 0:+ self._checkpoint("tick")++ # ------------------------------------------------------------------ requests (simulator thread)+ def dispatch(self, job: _Job) -> None:+ """Serve one queued request; a watch finishes later, from tick()."""+ try:+ self.first_request()+ path = urlparse(job.path).path+ if job.method == "GET" and path in ("/spec", "/status"):+ return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))+ routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}+ if self.reset_budget:+ routes["/reset"] = self._rt_reset+ if job.method != "POST" or path not in routes:+ return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))+ if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:+ return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))+ try:+ meta, parts = self._decode(job.body)+ except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error+ return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))+ try:+ routes[path](meta, parts, job)+ except Refused as exc: # 400 (bad_request), 409: nothing changed+ job.finish("raw", *self._reply_json(exc.code, exc.body))+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ self.log("error", path=path, error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))+ except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever+ traceback.print_exc()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))++ def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:+ """2.1's _view, plus cameras = [] for no images."""+ none = isinstance(meta.get("cameras"), list) and not meta["cameras"]+ cams, fps, video, depth = self._view(meta, default_video)+ return ([], fps, "none", False) if none else (cams, fps, video, depth)++ def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:+ """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the+ plan's last row (held), else -- nothing planned, or a step already run -- the current command."""+ i = step - nxt+ if i < 0 or not self.plan:+ return np.asarray(self.sim.command(typ), dtype=np.float64)+ return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)++ def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:+ if self.done or self.stopping:+ raise self._conflict()+ nxt = len(self.episode)+ with bad_request():+ typ = meta.get("type")+ if not isinstance(typ, str) or typ not in self.types:+ raise ValueError(f"type must be one of {sorted(self.types)}")+ try:+ A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),+ dtype=np.float64)+ except (TypeError, ValueError, KeyError):+ raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None+ dim = int(self.types[typ]["dim"])+ if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:+ raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")+ if not np.all(np.isfinite(A)):+ raise ValueError("actions contain NaN or inf")+ cdt = self.sim.control_dt+ try:+ dt = cdt if meta.get("dt") is None else float(meta["dt"])+ except (TypeError, ValueError):+ raise ValueError("dt must be a number of seconds") from None+ k = round(dt / cdt) if np.isfinite(dt / cdt) else 0+ if k < 1 or abs(k * cdt - dt) > 1e-9:+ raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")+ if k > 1 and not self.sim.interpolable(typ):+ raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")+ if len(A) * k * cdt > self.max_chunk_s + 1e-9:+ raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "+ f"more than max_chunk_s = {self.max_chunk_s:g}")+ interp = meta.get("interpolate", "linear")+ if interp not in ("linear", "hold"):+ raise ValueError("interpolate must be linear or hold")+ at = meta.get("at")+ if at is not None:+ if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:+ raise ValueError("at must be a control step of this episode (an integer >= 0), "+ "or null for the next one")+ at = int(at)+ if at - nxt > self.max_ahead:+ raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "+ f"of the next one ({nxt})")+ at = nxt if at is None else at+ cams, fps, video, depth = self._view_rt(meta, "none")+ rows, clipped = self.sim.clip(typ, A)+ prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`+ steps = []+ for row in rows:+ for j in range(1, k + 1):+ steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))+ prev = row+ late = max(0, nxt - at)+ keep = at - nxt if at > nxt else 0+ self.calls += 1+ self.rows += len(A)+ head = list(self.plan)[:keep]+ if len(head) < keep: # a gap before `at`: the target in force then is held through it+ fill = self._planned(at - 1, nxt, typ)+ head += [(typ, fill, -1)] * (keep - len(head))+ self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])+ self.late_steps += min(late, len(steps))+ self._start_clock()+ rec = Recorder(self.sim, cams, fps, depth, "none")+ rec.start()+ rec.finish()+ out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,+ "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,+ "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}+ self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)+ job.finish("rec", out, rec)++ def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:+ with bad_request():+ try:+ seconds = float(meta.get("seconds") or 0.0)+ except (TypeError, ValueError):+ raise ValueError("seconds must be a number") from None+ if not 0.0 <= seconds <= self.max_watch_s:+ raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")+ cams, fps, video, depth = self._view_rt(meta, "frames")+ n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0+ rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video+ self._frame_budget(rec, n)+ self.calls += 1+ rec.start()+ if not n:+ rec.finish()+ out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}+ self._save_call(out, None)+ return job.finish("rec", out, rec)+ self._start_clock()+ self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))++ def _complete_watch(self, w: _Watch) -> None:+ self.watches.remove(w)+ w.rec.finish()+ ran = max(0, min(len(self.episode), w.end) - w.start)+ out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),+ "control_steps_watch": ran, "ok": True}+ self._save_call(out, None)+ w.job.finish("rec", out, w.rec)++ def _release_watches(self) -> None:+ for w in list(self.watches):+ self._complete_watch(w)++ def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:+ self._release_watches()+ out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)+ self._stop_clock("reset")+ job.finish("rcb1", {**out, **self._realtime_counters()}, prts)++ # ------------------------------------------------------------------ the loop (simulator thread) and HTTP+ def poll(self) -> None:+ """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""+ if self.stopping and not self.finalized:+ if not self.done:+ self._end("timeout")+ self._release_watches()+ self._finalize()++ def loop(self, until: threading.Event | None = None) -> None:+ """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""+ while until is None or not until.is_set():+ self.poll()+ wait = self.due_in(time.monotonic())+ if wait is not None and wait <= 0:+ self.tick()+ continue+ try:+ job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))+ except queue.Empty:+ continue+ self.dispatch(job)++ def submit_job(self, job: _Job) -> tuple[int, str, bytes]:+ """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""+ self.q.put(job)+ job.done.wait()+ kind, *rest = job.reply+ if kind == "raw":+ return tuple(rest)+ meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread+ return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)++ def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:+ svc = self++ class Handler(BaseHTTPRequestHandler):+ timeout = 120++ def log_message(self, *a):+ pass++ def _send(self, code: int, ctype: str, body: bytes) -> None:+ self.send_response(code)+ self.send_header("Content-Type", ctype)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_GET(self):+ self._send(*svc.submit_job(_Job("GET", self.path, b"")))++ def do_POST(self):+ try:+ n = int(self.headers.get("Content-Length") or 0)+ except ValueError:+ return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))+ if n > MAX_BODY: # refused unread: drop the connection with it+ self.close_connection = True+ return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))+ self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))++ class Server(ThreadingHTTPServer):+ daemon_threads = True++ return Server(address, Handler)++ def serve(self, port: int = 8000) -> None:+ import signal+ signal.signal(signal.SIGUSR1, self.on_sigusr1)+ srv = self.bind(port)+ threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()+ print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)+ self.loop()+++def main(build, port: int = 8000) -> None:+ """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of+ Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""+ rcb_service.Service = RealtimeService+ rcb_service.PROTOCOL = PROTOCOL+ rcb_service.main(build, port)+++if __name__ == "__main__":+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+ import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import++ main(server.build)diff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-manip-room-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-manip-room-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-manip-room-i00-realtime/instruction.md b/tasks/humanoidbench-manip-room-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-manip-room-i00-realtime/instruction.mdrename to tasks/humanoidbench-manip-room-i00-standard-realtime/instruction.mdindex 76b0fd901b..36d16ca996 100644--- a/tasks/humanoidbench-manip-room-i00-realtime/instruction.md+++ b/tasks/humanoidbench-manip-room-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# room — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# room — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid with two Shadow dexterous hands (HumanoidBench / MuJoCo) **as ifit were real hardware**: the simulator runs in a service you cannot open, and you reach it only through a@@ -75,7 +75,7 @@ service ends the episode the moment the pass mark is reached (`reason = "successrecorded episode, the servo targets the robot actually executed at each control step, must reach it as welland end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-manip-room-i00-realtime/task.toml b/tasks/humanoidbench-manip-room-i00-standard-realtime/task.tomlsimilarity index 71%rename from tasks/humanoidbench-manip-room-i00-realtime/task.tomlrename to tasks/humanoidbench-manip-room-i00-standard-realtime/task.tomlindex ef82fb7a25..fe6d5f04fd 100644--- a/tasks/humanoidbench-manip-room-i00-realtime/task.toml+++ b/tasks/humanoidbench-manip-room-i00-standard-realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of humanoidbench-manip-room-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of humanoidbench-manip-room-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,10 +16,10 @@ artifacts = [][task]-name = "robot-coding-bench/humanoidbench-manip-room-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench room on the same frozen instance as humanoidbench-manip-room-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/humanoidbench-manip-room-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench room on the same frozen instance as humanoidbench-manip-room-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "realtime-mode", "trajectory", "room"]+keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "standard-realtime-mode", "trajectory", "room"][metadata]author_name = "pingyue"@@ -27,7 +27,7 @@ task_type = "trajectory"simulator = "humanoidbench"robot = "unitree-h1hand"family = "room"-mode = "realtime"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = 0expert_time_estimate_hours = 10.0diff --git a/tasks/humanoidbench-manip-room-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-manip-room-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-manip-room-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-manip-room-i00-realtime/tests/grade.py b/tasks/humanoidbench-manip-room-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-manip-room-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-manip-room-i00-realtime/tests/test.sh b/tasks/humanoidbench-manip-room-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-manip-room-i00-realtime/tests/test.shrename to tasks/humanoidbench-manip-room-i00-standard-realtime/tests/test.shdiff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-window-i00-realtime/environment/rcb_client.pydeleted file mode 100644index 8e057bb6e9..0000000000--- a/tasks/humanoidbench-manip-window-i00-realtime/environment/rcb_client.py+++ /dev/null@@ -1,209 +0,0 @@-"""eai-standard/2.3 client (realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on-PYTHONPATH).-- from rcb_client import Robot, EpisodeOver- r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready--TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or-get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x-spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a-plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out-the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.--Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):- r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"- ("joint_position": dim, layout (actuator order), lower/upper, range_rad,- units, semantics), "control_dt" (0.02 s), "max_control_steps" (the- episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:- rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"- (joints, axes, positions), "imu", "links" (masses, centres of mass, the- feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"- r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,- planned_steps, lag_s (and episode, resets_left with a reset budget)- r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")- POST /observe. seconds=0: the robot now, at once. seconds>0 (at most- spec()["max_watch_s"]): waits while that much robot time runs (starting the- clock if it is not running) and returns what the cameras saw; it never- changes the plan- r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,- video="none", include_depth=False, note="")- POST /submit: answers at once with the robot as it is now (the rows- have not run yet). actions (T, dim): rows dt apart (a multiple of- control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's- target is the row (interpolate="hold") or a straight line from the previous- row's ("linear", starting from the target in force just before `at`). They- become the plan from control step `at` (an episode step number; None = the- next step), replacing whatever was planned from `at` on and keeping what is- planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows- for steps already gone when the request arrives are dropped- (meta["late_steps"]). The first submit of an episode starts its clock- r.reset(cameras=None, include_depth=False)- POST /reset, only when spec()["budget"]["reset"] > 0: abandons the- running episode and starts a new one, paused, from the same starting state;- only the last episode is graded, and a success ends the trial--cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the-fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),-"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth-for the cameras that have it.--Result (what get_obs / submit / reset return):- res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),- sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |- "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,- lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]- (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,- resets_left- res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])- res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes- res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)- res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s- of a watch, one at its end- res.done the episode is over- res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy--Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an-unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over-(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this-run (404).-"""-from __future__ import annotations--import json-import os-import pathlib-import time--import numpy as np-import requests--import rcb1--PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)-MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused---class EpisodeOver(RuntimeError):- pass---class Result:- """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""-- def __init__(self, meta: dict, parts: dict):- self.meta, self.raw = meta, parts- self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None- for name, p in parts.items():- kind, data = p["kind"], p["data"]- if name.startswith("video/") and kind == "mp4":- self.video[name[6:]] = data- elif name.startswith("frames/"):- self.frames[name[7:]] = data- elif name.startswith("last/"):- self.last[name[5:]] = data- elif name.startswith("depth/"):- self.depth[name[6:]] = data- elif name == "proprio":- self.proprio = data-- @property- def done(self) -> bool:- return bool(self.meta.get("done"))-- def save(self, out_dir: str) -> str:- from PIL import Image- out = pathlib.Path(out_dir)- out.mkdir(parents=True, exist_ok=True)- (out / "meta.json").write_text(json.dumps(self.meta, indent=1))- for cam, b in self.video.items():- (out / f"{cam}.mp4").write_bytes(b)- for cam, img in self.last.items():- (out / "last").mkdir(exist_ok=True)- Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)- for cam, arr in self.frames.items():- d = out / "frames" / cam- d.mkdir(parents=True, exist_ok=True)- for i, f in enumerate(arr):- Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)- for cam, arr in self.depth.items():- np.save(out / f"depth_{cam}.npy", arr)- if self.proprio is not None:- np.save(out / "proprio.npy", self.proprio)- return str(out)---class Robot:- def __init__(self, base: str | None = None, timeout: float = 600.0):- candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]- t0, last = time.time(), None- while time.time() - t0 < timeout:- for b in candidates:- try:- r = requests.get(b + "/spec", timeout=(5, 60))- if r.status_code == 503:- last = "warming up"- continue- r.raise_for_status()- self.base, self._spec = b, r.json()- if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):- raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")- return- except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:- last = e # not up yet (the scene takes about a minute), or busy- time.sleep(2)- raise RuntimeError(f"sim service unavailable: {last}")-- def spec(self) -> dict:- return self._spec-- def status(self) -> dict:- r = requests.get(self.base + "/status", timeout=(5, 60))- r.raise_for_status()- return r.json()-- def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,- video: str = "frames") -> Result:- meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),- "include_depth": bool(include_depth), "video": video}- return self._post("/observe", meta, [], timeout=900)-- def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",- at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",- include_depth: bool = False, note: str = "") -> Result:- a = np.asarray(actions, dtype=np.float64)- if a.ndim == 1:- a = a[None, :]- if not np.all(np.isfinite(a)):- raise ValueError("actions contain NaN or inf")- meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,- "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),- "video": video,- "include_depth": bool(include_depth), "note": note}- return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)-- def reset(self, cameras=None, include_depth: bool = False) -> Result:- return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)-- def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:- body = rcb1.encode(meta, parts)- if len(body) > MAX_BODY: # the service would close the connection on it unread- raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")- r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},- timeout=(5, timeout))- if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:- return Result(*rcb1.decode(r.content))- try:- err = r.json()- except ValueError:- err = {"error": r.text[:300]}- msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")- if r.status_code == 409:- raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")- + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))- if r.status_code in (400, 413):- raise ValueError(msg)- raise RuntimeError(msg)---def _names(cameras) -> list[str] | None:- return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb_realtime.pydeleted file mode 100644index 410284bf27..0000000000--- a/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb_realtime.py+++ /dev/null@@ -1,479 +0,0 @@-"""eai-standard/2.3, the realtime mode's protocol: the sim service of HumanoidBench's `-realtime` tasks (docs/PROTOCOL.md-§4b, docs/LIMITED_MODE_PROTOCOL.md "Realtime mode"; docs/benchmarks/humanoidbench.md). Entry point of the sidecar:-`python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime capability, every 2.2-addition optional: this service is rcb_service's 2.1 service with the capability, and reports eai-standard/2.3.--It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery-and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:--- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its- first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then- on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,- until the episode ends; the clock stops there.-- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`- or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the- next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`- loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order- hold): a slow agent's robot keeps its last servo targets.-- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)- and returns what the cameras saw. It never changes the plan.-- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.--Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's-rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was-submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):-ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line-per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:-call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.--Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in-between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting-request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than-MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.--A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.-A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a-crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the-trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends-the graded episode (reason "error") sets infra_error in final.json too.-"""-from __future__ import annotations--import collections-import json-import os-import queue-import sys-import threading-import time-import traceback-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer-from urllib.parse import urlparse--import numpy as np--import rcb_service-from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request--PROTOCOL = "eai-standard/2.3" # the realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1-MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up-CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)---class _Job:- """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""-- def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):- self.method, self.path, self.body, self.content_length = method, path, body, content_length- self.done = threading.Event()- self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)-- def finish(self, *reply) -> None:- self.reply = reply- self.done.set()---class _Watch:- """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""-- def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):- self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id- self.start, self.end = start, start + steps---class RealtimeService(Service):- """rcb_service.Service with the realtime clock (module docstring)."""-- def __init__(self, sim, *, slowdown: float | None = None, **kw):- self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)- if not self.slowdown > 0:- raise ValueError("RCB_SLOWDOWN must be > 0")- self.wall_dt = sim.control_dt * self.slowdown- self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be- self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step- self.running = False- self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t- self.watches: list[_Watch] = []- self.q: queue.Queue = queue.Queue()- self.max_lag = self.lag = 0.0- self.stalls = self.held_steps = self.late_steps = 0- self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk- super().__init__(sim, **kw)- self.ticks_path = os.path.join(self.data, "ticks.f64")- self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))-- # ------------------------------------------------------------------ spec, status, evidence- def _spec(self) -> dict:- spec = super()._spec()- spec["protocol"] = PROTOCOL- cdt = self.sim.control_dt- spec["timing"] = {- "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,- "wall_s_per_control_step": cdt * self.slowdown,- "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "- "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "- "you do, until the episode ends",- "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "- "`at` (default: the next one), replacing what was planned from there; rows for steps already "- "gone are dropped (meta.late_steps); after the plan's last row that target is held",- "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "- "the cameras saw; never changes the plan",- "max_ahead_steps": self.max_ahead,- "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}- return spec-- def _realtime_counters(self) -> dict:- return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}-- def status(self) -> dict:- return {**super().status(), **self._realtime_counters()}-- def meta(self) -> dict:- return {**super().meta(), **self._realtime_counters()}-- def _timing_record(self) -> dict:- return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),- "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}-- def _write_final(self) -> None:- self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's- if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)- self.verdict["infra_error"] = 1- else: # (an episode that erred and was reset away is not graded)- self.verdict.pop("infra_error", None)- super()._write_final()-- def _checkpoint(self, ev: str, **kw) -> None:- if self._ticks: # the tick log first: a crash in between leaves it too long- with open(self.ticks_path, "ab") as f:- f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())- self._ticks = []- super()._checkpoint(ev, **kw)-- def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:- keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",- "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")- line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}- with open(os.path.join(self.data, "calls.jsonl"), "a") as f:- f.write(json.dumps(line) + "\n")- if actions is not None:- rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)- with open(os.path.join(self.data, "submitted.f64"), "ab") as f:- f.write(np.asarray(rows, dtype=np.float64).tobytes())-- # ------------------------------------------------------------------ start, clock- def start(self) -> None:- super().start()- rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)- n = self.journal_rows- if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too- rows[: 2 * n].tofile(self.ticks_path)- elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)- np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)- self.running = False # a resumed episode waits, paused, for the agent's next call-- def _start_clock(self) -> None:- if not self.running and not self.done and not self.stopping:- self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)- self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)-- def _stop_clock(self, why: str) -> None:- if self.running:- self.running = False- self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))- self.plan.clear()-- def _end(self, reason: str) -> None:- super()._end(reason)- self._stop_clock(reason) # pending watches are completed by whoever ended it, after this-- def due_in(self, now: float) -> float | None:- """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""- if not self.running or self.done or self.stopping:- return None- return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now-- def tick(self) -> None:- """One control step of the clock: the plan's row for it, or the held target."""- now = time.monotonic()- lag = -(self.due_in(now) or 0.0)- if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt- self.stalls += 1- self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))- self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0- self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)- if self.plan:- typ, row, src = self.plan.popleft()- else:- (typ, row), src = self.sim.hold(), -1- self.held_steps += 1- before = self.journal_rows- self._ticks.append((time.time(), float(src)))- try:- self._step(typ, row)- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- if self.journal_rows == before:- self._ticks.pop()- self.log("error", path="clock", error=repr(exc)[:500])- self._end("error")- self._release_watches()- return- for w in list(self.watches):- w.rec.step()- if self.done or len(self.episode) >= w.end:- self._complete_watch(w)- if not self.done and len(self.episode) % self.ckpt_every == 0:- self._checkpoint("tick")-- # ------------------------------------------------------------------ requests (simulator thread)- def dispatch(self, job: _Job) -> None:- """Serve one queued request; a watch finishes later, from tick()."""- try:- self.first_request()- path = urlparse(job.path).path- if job.method == "GET" and path in ("/spec", "/status"):- return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))- routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}- if self.reset_budget:- routes["/reset"] = self._rt_reset- if job.method != "POST" or path not in routes:- return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))- if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:- return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))- try:- meta, parts = self._decode(job.body)- except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error- return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))- try:- routes[path](meta, parts, job)- except Refused as exc: # 400 (bad_request), 409: nothing changed- job.finish("raw", *self._reply_json(exc.code, exc.body))- except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware- traceback.print_exc()- self.log("error", path=path, error=repr(exc)[:500])- self._end("error")- self._release_watches()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))- except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever- traceback.print_exc()- if not job.done.is_set():- job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))-- def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:- """2.1's _view, plus cameras = [] for no images."""- none = isinstance(meta.get("cameras"), list) and not meta["cameras"]- cams, fps, video, depth = self._view(meta, default_video)- return ([], fps, "none", False) if none else (cams, fps, video, depth)-- def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:- """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the- plan's last row (held), else -- nothing planned, or a step already run -- the current command."""- i = step - nxt- if i < 0 or not self.plan:- return np.asarray(self.sim.command(typ), dtype=np.float64)- return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)-- def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:- if self.done or self.stopping:- raise self._conflict()- nxt = len(self.episode)- with bad_request():- typ = meta.get("type")- if not isinstance(typ, str) or typ not in self.types:- raise ValueError(f"type must be one of {sorted(self.types)}")- try:- A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),- dtype=np.float64)- except (TypeError, ValueError, KeyError):- raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None- dim = int(self.types[typ]["dim"])- if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:- raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")- if not np.all(np.isfinite(A)):- raise ValueError("actions contain NaN or inf")- cdt = self.sim.control_dt- try:- dt = cdt if meta.get("dt") is None else float(meta["dt"])- except (TypeError, ValueError):- raise ValueError("dt must be a number of seconds") from None- k = round(dt / cdt) if np.isfinite(dt / cdt) else 0- if k < 1 or abs(k * cdt - dt) > 1e-9:- raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")- if k > 1 and not self.sim.interpolable(typ):- raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")- if len(A) * k * cdt > self.max_chunk_s + 1e-9:- raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "- f"more than max_chunk_s = {self.max_chunk_s:g}")- interp = meta.get("interpolate", "linear")- if interp not in ("linear", "hold"):- raise ValueError("interpolate must be linear or hold")- at = meta.get("at")- if at is not None:- if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:- raise ValueError("at must be a control step of this episode (an integer >= 0), "- "or null for the next one")- at = int(at)- if at - nxt > self.max_ahead:- raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "- f"of the next one ({nxt})")- at = nxt if at is None else at- cams, fps, video, depth = self._view_rt(meta, "none")- rows, clipped = self.sim.clip(typ, A)- prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`- steps = []- for row in rows:- for j in range(1, k + 1):- steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))- prev = row- late = max(0, nxt - at)- keep = at - nxt if at > nxt else 0- self.calls += 1- self.rows += len(A)- head = list(self.plan)[:keep]- if len(head) < keep: # a gap before `at`: the target in force then is held through it- fill = self._planned(at - 1, nxt, typ)- head += [(typ, fill, -1)] * (keep - len(head))- self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])- self.late_steps += min(late, len(steps))- self._start_clock()- rec = Recorder(self.sim, cams, fps, depth, "none")- rec.start()- rec.finish()- out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,- "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,- "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}- self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)- job.finish("rec", out, rec)-- def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:- with bad_request():- try:- seconds = float(meta.get("seconds") or 0.0)- except (TypeError, ValueError):- raise ValueError("seconds must be a number") from None- if not 0.0 <= seconds <= self.max_watch_s:- raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")- cams, fps, video, depth = self._view_rt(meta, "frames")- n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0- rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video- self._frame_budget(rec, n)- self.calls += 1- rec.start()- if not n:- rec.finish()- out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}- self._save_call(out, None)- return job.finish("rec", out, rec)- self._start_clock()- self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))-- def _complete_watch(self, w: _Watch) -> None:- self.watches.remove(w)- w.rec.finish()- ran = max(0, min(len(self.episode), w.end) - w.start)- out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),- "control_steps_watch": ran, "ok": True}- self._save_call(out, None)- w.job.finish("rec", out, w.rec)-- def _release_watches(self) -> None:- for w in list(self.watches):- self._complete_watch(w)-- def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:- self._release_watches()- out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)- self._stop_clock("reset")- job.finish("rcb1", {**out, **self._realtime_counters()}, prts)-- # ------------------------------------------------------------------ the loop (simulator thread) and HTTP- def poll(self) -> None:- """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""- if self.stopping and not self.finalized:- if not self.done:- self._end("timeout")- self._release_watches()- self._finalize()-- def loop(self, until: threading.Event | None = None) -> None:- """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""- while until is None or not until.is_set():- self.poll()- wait = self.due_in(time.monotonic())- if wait is not None and wait <= 0:- self.tick()- continue- try:- job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))- except queue.Empty:- continue- self.dispatch(job)-- def submit_job(self, job: _Job) -> tuple[int, str, bytes]:- """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""- self.q.put(job)- job.done.wait()- kind, *rest = job.reply- if kind == "raw":- return tuple(rest)- meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread- return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)-- def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:- svc = self-- class Handler(BaseHTTPRequestHandler):- timeout = 120-- def log_message(self, *a):- pass-- def _send(self, code: int, ctype: str, body: bytes) -> None:- self.send_response(code)- self.send_header("Content-Type", ctype)- self.send_header("Content-Length", str(len(body)))- self.end_headers()- self.wfile.write(body)-- def do_GET(self):- self._send(*svc.submit_job(_Job("GET", self.path, b"")))-- def do_POST(self):- try:- n = int(self.headers.get("Content-Length") or 0)- except ValueError:- return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))- if n > MAX_BODY: # refused unread: drop the connection with it- self.close_connection = True- return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))- self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))-- class Server(ThreadingHTTPServer):- daemon_threads = True-- return Server(address, Handler)-- def serve(self, port: int = 8000) -> None:- import signal- signal.signal(signal.SIGUSR1, self.on_sigusr1)- srv = self.bind(port)- threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()- print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)- self.loop()---def main(build, port: int = 8000) -> None:- """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of- Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""- rcb_service.Service = RealtimeService- rcb_service.PROTOCOL = PROTOCOL- rcb_service.main(build, port)---if __name__ == "__main__":- sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))- import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import-- main(server.build)diff --git a/tasks/humanoidbench-manip-window-i00-realtime/README.md b/tasks/humanoidbench-manip-window-i00-standard-realtime/README.mdsimilarity index 82%rename from tasks/humanoidbench-manip-window-i00-realtime/README.mdrename to tasks/humanoidbench-manip-window-i00-standard-realtime/README.mdindex bbce226dcc..266b8da954 100644--- a/tasks/humanoidbench-manip-window-i00-realtime/README.md+++ b/tasks/humanoidbench-manip-window-i00-standard-realtime/README.md@@ -1,9 +1,9 @@-# humanoidbench-manip-window-i00-realtime+# humanoidbench-manip-window-i00-standard-realtime| | ||---|---|| Benchmark | HumanoidBench |-| Mode | realtime (`eai-standard/2.3`) |+| Mode | standard-realtime (`eai-standard/2.3`) || Twin | `humanoidbench-manip-window-i00-privileged` || Simulator / robot | `humanoidbench` / `unitree-h1hand` || Family / instance | `window` / seed 0 |@@ -12,7 +12,7 @@| Base image | `ghcr.io/mll-lab-nu/eai-humanoidbench:0.1.6` |## Task-Realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of+Standard-realtime mode (robot as a service on a clock that does not stop, `eai-standard/2.3`, PROTOCOL §4b) of`humanoidbench-manip-window-i00-privileged`: the same frozen instance and the same success rule as it and as`humanoidbench-manip-window-i00-standard`, but time does not stop. Once an episode's clock starts (its first`submit`), the robot runs at 50 Hz in real time and holds its last servo targets while the agent thinks, so@@ -55,8 +55,8 @@ Generated by `scripts/humanoidbench/gen_tasks.py` from `subset.toml`, `templates## Details| | ||---|---|-| mode / sensors | realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no tool pose, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) |+| mode / sensors | standard-realtime: head-camera RGB + joint angles and velocities (HumanoidBench's own `image` and `proprio` sensors), a pelvis IMU and a camera fixed in the room; no root pose, no tool pose, no reward; 50 resets (a new episode from the frozen start each, paused until its first `submit`; only the last episode is graded) || time | the episode's clock starts at its first `submit` (or `get_obs(seconds > 0)`) and runs one control step every 0.02 s of wall time until the episode ends; `submit` writes a plan of future targets (from step `at`) and answers at once; the last target is held when the plan runs out || action space | HumanoidBench's own: 61 normalised position targets, 50 Hz, streamed one row at a time or in chunks of up to 10 s |-| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under |+| knobs (compose env) | `RCB_SLOWDOWN` (1: real time, the official configuration; S: one control step every S x 0.02 s, results reported apart from S = 1), `RCB_RESET_BUDGET` (50: the standard-realtime mode's default, PROTOCOL §4b), `RCB_OBS_PROPRIO` (1), `RCB_OBS_IMU` (1), `RCB_OBS_ROOM_CAMERA` (1), `RCB_OBS_DEPTH` (0): the defaults are this task's configuration; a run that changes one is reported separately, and `episode.jsonl` records the `/spec` it ran under || agent API | `environment/rcb_client.py` (the realtime client; its docstring is the reference); `instruction.md` lists the endpoints |diff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/Dockerfile b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/Dockerfilerename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/Dockerfilediff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/client-requirements.txt b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/client-requirements.txtsimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/client-requirements.txtrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/client-requirements.txtdiff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/docker-compose.yaml b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/docker-compose.yamlsimilarity index 89%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/docker-compose.yamlrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/docker-compose.yamlindex 53ecca5f07..10772bd308 100644--- a/tasks/humanoidbench-manip-window-i00-realtime/environment/docker-compose.yaml+++ b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/docker-compose.yaml@@ -1,4 +1,4 @@-# Realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an+# Standard-realtime mode (PROTOCOL §4b): the simulator runs in the `sim` sidecar (the agent cannot open it), on its own clock once an# episode starts (sim/rcb_realtime.py); the agent only reaches port 8000.# No `expose`/`ports` on sim: under Harbor's egress control it shares main's network namespace (localhost:8000).# GPU on sim: MuJoCo physics is CPU either way, but every image the agent asks for is rendered here, and EGL on the@@ -21,7 +21,7 @@ services:# Knobs: ${NAME:-default} takes NAME from the shell that runs harbor. The defaults are the standard# configuration; results under other settings are reported separately.RCB_SLOWDOWN: "${RCB_SLOWDOWN:-1}" # 1: real time (official); S: one step every S x control_dt s, reported apart- RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episode+ RCB_RESET_BUDGET: "${RCB_RESET_BUDGET:-50}" # standard-realtime mode's default: 50 resets (PROTOCOL §4b); 0: one episodeRCB_OBS_PROPRIO: "${RCB_OBS_PROPRIO:-1}" # 1: + joint angles/velocities; 0: images onlyRCB_OBS_IMU: "${RCB_OBS_IMU:-1}" # 1: + pelvis IMU (angular velocity, gravity direction)RCB_OBS_ROOM_CAMERA: "${RCB_OBS_ROOM_CAMERA:-1}" # 1: + room_camera, fixed in the roomdiff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/rcb1.py b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/rcb1.pyrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/rcb1.pydiff --git a/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/rcb_client.py b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/rcb_client.pynew file mode 100644index 0000000000..f2b0559bf9--- /dev/null+++ b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/rcb_client.py@@ -0,0 +1,209 @@+"""eai-standard/2.3 client (standard-realtime mode): your whole interface to the robot (installed at /opt/tools/rcb_client.py, on+PYTHONPATH).++ from rcb_client import Robot, EpisodeOver+ r = Robot() # finds the service (SIM_URL, http://sim:8000, http://localhost:8000), waits until it is ready++TIME DOES NOT STOP (spec()["timing"]). An episode starts paused; its clock starts at its first submit() or+get_obs(seconds > 0) (again after every reset()) and from then on runs one control step every spec()["control_dt"] x+spec()["timing"]["slowdown"] seconds of wall time, whether or not you call, until the episode ends. The service keeps a+plan, one servo-target row per coming control step: submit() writes into it and answers at once; when the plan runs out+the last target is held. Whatever you do in between -- think, print, run a model -- the robot keeps executing.++Calls (each is one HTTP request; every answer that carries data is an RCB1 frame, decoded into a Result):+ r.spec() GET /spec, fetched when connecting: the robot's facts. "action_types"+ ("joint_position": dim, layout (actuator order), lower/upper, range_rad,+ units, semantics), "control_dt" (0.02 s), "max_control_steps" (the+ episode), "timing" (realtime, slowdown, max_ahead_steps), "cameras" {name:+ rgb, depth, hw, fovy_deg, pose, mounted}, "proprio_layout" and "proprio"+ (joints, axes, positions), "imu", "links" (masses, centres of mass, the+ feet's boxes), "depth", "budget" {"reset", "wall_clock_s"}, "episode"+ r.status() GET /status: calls, rows, control_steps, done, reason, elapsed_s, running,+ planned_steps, lag_s (and episode, resets_left with a reset budget)+ r.get_obs(cameras=None, seconds=0.0, video_fps=10, include_depth=False, video="frames")+ POST /observe. seconds=0: the robot now, at once. seconds>0 (at most+ spec()["max_watch_s"]): waits while that much robot time runs (starting the+ clock if it is not running) and returns what the cameras saw; it never+ changes the plan+ r.submit("joint_position", actions, dt=None, interpolate="linear", at=None, cameras=None, video_fps=10,+ video="none", include_depth=False, note="")+ POST /submit: answers at once with the robot as it is now (the rows+ have not run yet). actions (T, dim): rows dt apart (a multiple of+ control_dt, the default; T x dt at most spec()["max_chunk_s"]); each step's+ target is the row (interpolate="hold") or a straight line from the previous+ row's ("linear", starting from the target in force just before `at`). They+ become the plan from control step `at` (an episode step number; None = the+ next step), replacing whatever was planned from `at` on and keeping what is+ planned before it; at most spec()["timing"]["max_ahead_steps"] ahead. Rows+ for steps already gone when the request arrives are dropped+ (meta["late_steps"]). The first submit of an episode starts its clock+ r.reset(cameras=None, include_depth=False)+ POST /reset, only when spec()["budget"]["reset"] > 0: abandons the+ running episode and starts a new one, paused, from the same starting state;+ only the last episode is graded, and a success ends the trial++cameras=None means every camera in spec()["cameras"]; cameras=[] means no images at all, only proprioception (the+fastest call: what a controller loop wants). video: "mp4" (one H.264 file per camera), "frames" (the images as arrays),+"both" or "none" (only the last frame); a submit returns no video (no time passes inside it). include_depth adds depth+for the cameras that have it.++Result (what get_obs / submit / reset return):+ res.meta dict: calls, rows, control_steps (steps run in this episode: the next one is this number),+ sim_time_s, time_s, time_left_s, proprio {"layout", "now"}, done, reason ("success" |+ "failed" | "timeout" | "error"; None while the episode runs), running, planned_steps,+ lag_s; per submit: start_step, late_steps, plan_end_step, clipped [{row, index, reason}]+ (values outside [-1, 1] are clipped, not refused); with a reset budget also episode,+ resets_left+ res.last[cam] uint8 H x W x 3: the last frame (unless cameras=[])+ res.frames[cam] uint8 N x H x W x 3 (get_obs, video="frames"/"both") res.video[cam] mp4 bytes+ res.depth[cam] float16 N x H x W, metres, 0 = nothing in range (include_depth)+ res.proprio float32 rows of spec()["proprio_layout"]: one at the call's start, one every 1/video_fps s+ of a watch, one at its end+ res.done the episode is over+ res.save(dir) meta.json, <cam>.mp4, last/<cam>.jpg, frames/<cam>/NNN.jpg, depth_<cam>.npy, proprio.npy++Errors: ValueError -- refused, nothing changed (400: wrong type or shape, NaN, a bad dt or at, too many frames, an+unknown camera; or a request over 64 MB, refused before it is sent); EpisodeOver -- the episode, or the trial, is over+(409); RuntimeError -- the service failed (5xx: see spec()["on_service_error"]) or the endpoint does not exist in this+run (404).+"""+from __future__ import annotations++import json+import os+import pathlib+import time++import numpy as np+import requests++import rcb1++PROTOCOLS = ("eai-standard/2.", "rcb-limited/2.") # any 2.x service, under either name (rcb-limited is the old one)+MAX_BODY = 64 * 1024 * 1024 # the protocol's request limit: a larger request is refused+++class EpisodeOver(RuntimeError):+ pass+++class Result:+ """A decoded RCB1 answer: meta, video, frames, last, depth, proprio; save(dir) writes them all out."""++ def __init__(self, meta: dict, parts: dict):+ self.meta, self.raw = meta, parts+ self.video, self.frames, self.last, self.depth, self.proprio = {}, {}, {}, {}, None+ for name, p in parts.items():+ kind, data = p["kind"], p["data"]+ if name.startswith("video/") and kind == "mp4":+ self.video[name[6:]] = data+ elif name.startswith("frames/"):+ self.frames[name[7:]] = data+ elif name.startswith("last/"):+ self.last[name[5:]] = data+ elif name.startswith("depth/"):+ self.depth[name[6:]] = data+ elif name == "proprio":+ self.proprio = data++ @property+ def done(self) -> bool:+ return bool(self.meta.get("done"))++ def save(self, out_dir: str) -> str:+ from PIL import Image+ out = pathlib.Path(out_dir)+ out.mkdir(parents=True, exist_ok=True)+ (out / "meta.json").write_text(json.dumps(self.meta, indent=1))+ for cam, b in self.video.items():+ (out / f"{cam}.mp4").write_bytes(b)+ for cam, img in self.last.items():+ (out / "last").mkdir(exist_ok=True)+ Image.fromarray(np.ascontiguousarray(img)).save(out / "last" / f"{cam}.jpg", quality=90)+ for cam, arr in self.frames.items():+ d = out / "frames" / cam+ d.mkdir(parents=True, exist_ok=True)+ for i, f in enumerate(arr):+ Image.fromarray(np.ascontiguousarray(f)).save(d / f"{i:03d}.jpg", quality=85)+ for cam, arr in self.depth.items():+ np.save(out / f"depth_{cam}.npy", arr)+ if self.proprio is not None:+ np.save(out / "proprio.npy", self.proprio)+ return str(out)+++class Robot:+ def __init__(self, base: str | None = None, timeout: float = 600.0):+ candidates = [c for c in (base or os.environ.get("SIM_URL"), "http://sim:8000", "http://localhost:8000") if c]+ t0, last = time.time(), None+ while time.time() - t0 < timeout:+ for b in candidates:+ try:+ r = requests.get(b + "/spec", timeout=(5, 60))+ if r.status_code == 503:+ last = "warming up"+ continue+ r.raise_for_status()+ self.base, self._spec = b, r.json()+ if not str(self._spec.get("protocol", "")).startswith(PROTOCOLS):+ raise RuntimeError(f"not an eai-standard/2.x service: {self._spec.get('protocol')}")+ return+ except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:+ last = e # not up yet (the scene takes about a minute), or busy+ time.sleep(2)+ raise RuntimeError(f"sim service unavailable: {last}")++ def spec(self) -> dict:+ return self._spec++ def status(self) -> dict:+ r = requests.get(self.base + "/status", timeout=(5, 60))+ r.raise_for_status()+ return r.json()++ def get_obs(self, cameras=None, seconds: float = 0.0, video_fps: int = 10, include_depth: bool = False,+ video: str = "frames") -> Result:+ meta = {"cameras": _names(cameras), "seconds": float(seconds), "video_fps": int(video_fps),+ "include_depth": bool(include_depth), "video": video}+ return self._post("/observe", meta, [], timeout=900)++ def submit(self, action_type: str, actions, dt: float | None = None, interpolate: str = "linear",+ at: int | None = None, cameras=None, video_fps: int = 10, video: str = "none",+ include_depth: bool = False, note: str = "") -> Result:+ a = np.asarray(actions, dtype=np.float64)+ if a.ndim == 1:+ a = a[None, :]+ if not np.all(np.isfinite(a)):+ raise ValueError("actions contain NaN or inf")+ meta = {"type": action_type, "dt": None if dt is None else float(dt), "interpolate": interpolate,+ "at": None if at is None else int(at), "cameras": _names(cameras), "video_fps": int(video_fps),+ "video": video,+ "include_depth": bool(include_depth), "note": note}+ return self._post("/submit", meta, [rcb1.array_part("actions", a)], timeout=1800)++ def reset(self, cameras=None, include_depth: bool = False) -> Result:+ return self._post("/reset", {"cameras": _names(cameras), "include_depth": bool(include_depth)}, [], timeout=900)++ def _post(self, path: str, meta: dict, parts: list, timeout: float) -> Result:+ body = rcb1.encode(meta, parts)+ if len(body) > MAX_BODY: # the service would close the connection on it unread+ raise ValueError(f"{path}: {len(body)} bytes, over the protocol's {MAX_BODY}: send fewer rows")+ r = requests.post(self.base + path, data=body, headers={"Content-Type": "application/x-rcb1"},+ timeout=(5, timeout))+ if r.headers.get("Content-Type", "").startswith("application/x-rcb1") and r.status_code == 200:+ return Result(*rcb1.decode(r.content))+ try:+ err = r.json()+ except ValueError:+ err = {"error": r.text[:300]}+ msg = f"{r.status_code} {path}: {err.get('error')}" + (f" ({err['detail']})" if err.get("detail") else "")+ if r.status_code == 409:+ raise EpisodeOver(msg + (f"; reason {err['reason']}" if err.get("reason") else "")+ + (f"; resets_left {err['resets_left']}" if "resets_left" in err else ""))+ if r.status_code in (400, 413):+ raise ValueError(msg)+ raise RuntimeError(msg)+++def _names(cameras) -> list[str] | None:+ return None if cameras is None else [cameras] if isinstance(cameras, str) else list(cameras)diff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/instance.npz b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/instance.npzsimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/sim/instance.npzrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/instance.npzdiff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb1.py b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb1.pysimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb1.pyrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb1.pydiff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb_hb.py b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb_hb.pysimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb_hb.pyrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb_hb.pydiff --git a/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb_realtime.py b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb_realtime.pynew file mode 100644index 0000000000..e23f8cbd2c--- /dev/null+++ b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb_realtime.py@@ -0,0 +1,480 @@+"""eai-standard/2.3, the standard-realtime mode's protocol: the sim service of HumanoidBench's `-standard-realtime` tasks+(docs/PROTOCOL.md §4b, docs/LIMITED_MODE_PROTOCOL.md "Standard-realtime mode"; docs/benchmarks/humanoidbench.md). Entry+point of the sidecar: `python /srv/rcb_realtime.py` (the adapter is server.py's, unchanged). 2.3 is 2.2 plus the realtime+capability, every 2.2 addition optional: this service is rcb_service's 2.1 service with the capability, and reports+eai-standard/2.3.++It extends rcb_service.Service: the same endpoints, RCB1 frames, budgets (reset, wall clock), evidence, crash recovery+and finalize. Only the time model differs. In standard mode time moves only inside calls; here the episode has a clock:++- An episode starts paused, at the frozen start with the servos holding their starting targets. Its clock starts at its+ first POST /submit or POST /observe with seconds > 0 (again after every reset, and after a crash recovery). From then+ on one control step runs every control_dt x slowdown seconds of wall time, whether or not the agent sends anything,+ until the episode ends; the clock stops there.+- The service keeps a plan: one target row per coming control step. /submit expands its chunk (rows dt apart, `linear`+ or `hold`, exactly as 2.1 does) into control steps and writes it into the plan from control step `at` (default: the+ next one), replacing whatever was planned from there on, and answers at once. A chunk that arrives after its `at`+ loses the rows whose steps are gone (`late_steps`). When the plan runs out the last target is held (zero-order+ hold): a slow agent's robot keeps its last servo targets.+- /observe with seconds = 0 answers at once; with seconds > 0 it watches the running episode that long (robot time)+ and returns what the cameras saw. It never changes the plan.+- `cameras = []` asks for no images at all (in 2.1 an empty list means every camera): a controller's proprio-only call.++Grading is unchanged: every executed control step's sim row is journaled (trajectory_live.f64) and the graded episode's+rows are replayed in a fresh simulator at the end. What is graded is the rows the simulator applied, not what was+submitted, so the replay equals the live run however the timing went. Evidence added (never visible to the agent):+ticks.f64 (per journal row: wall time, and the call whose chunk gave the row, -1 for a held one), calls.jsonl (one line+per call; no calls/NNNNNN/ directories: a controller calls 50 times a second) and submitted.f64 (every row as sent:+call id, then the row). final.json adds realtime, slowdown, max_lag_s, stalls, held_steps and late_steps.++Threads: the simulator and its GL context belong to the main thread, which runs the clock and serves every request in+between control steps; the HTTP threads only read, queue and encode. A control step that is due runs before any waiting+request. A service that falls behind (a long render) runs the missed steps back to back, skipping none; more than+MAX_LAG_S behind, it drops the debt instead (a stall, counted) rather than fast-forward the robot.++A crash resumes the same episode from its last checkpoint (one per second of robot time while the clock runs), paused.+A simulator crash is an infrastructure failure, never the agent's (PROTOCOL §4b, owner 2026-10-08). No call spans a+crash here (submit answers at once), so a crash is never put down to the agent's call: past RCB_MAX_RESUMES restarts the+trial ends `sim_crashed` (infra_error), a failed resume `resume_failed` (infra_error), and a simulator error that ends+the graded episode (reason "error") sets infra_error in final.json too.+"""+from __future__ import annotations++import collections+import json+import os+import queue+import sys+import threading+import time+import traceback+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from urllib.parse import urlparse++import numpy as np++import rcb_service+from rcb_service import MAX_BODY, Recorder, Refused, Service, bad_request++PROTOCOL = "eai-standard/2.3" # the standard-realtime mode's protocol (PROTOCOL §4b); rcb_service.py itself serves 2.1+MAX_LAG_S = 0.5 # behind the clock by more than this: drop the debt (a stall) instead of catching up+CHECKPOINT_S = 1.0 # robot time between checkpoints while the clock runs (the most a crash loses)+++class _Job:+ """One HTTP request, handed from its HTTP thread to the simulator's thread and back."""++ def __init__(self, method: str, path: str, body: bytes, content_length: int | None = None):+ self.method, self.path, self.body, self.content_length = method, path, body, content_length+ self.done = threading.Event()+ self.reply: tuple = () # ("raw", code, ctype, body) | ("rcb1", meta, parts) | ("rec", meta, Recorder)++ def finish(self, *reply) -> None:+ self.reply = reply+ self.done.set()+++class _Watch:+ """A pending /observe with seconds > 0: completes once its control steps have run (or the episode ended)."""++ def __init__(self, job: _Job, rec: Recorder, start: int, steps: int, cameras: list[str], obs_id: int):+ self.job, self.rec, self.cameras, self.obs_id = job, rec, cameras, obs_id+ self.start, self.end = start, start + steps+++class RealtimeService(Service):+ """rcb_service.Service with the realtime clock (module docstring)."""++ def __init__(self, sim, *, slowdown: float | None = None, **kw):+ self.slowdown = float(os.environ.get("RCB_SLOWDOWN") or 1.0) if slowdown is None else float(slowdown)+ if not self.slowdown > 0:+ raise ValueError("RCB_SLOWDOWN must be > 0")+ self.wall_dt = sim.control_dt * self.slowdown+ self.max_ahead = int(round(float(kw.get("max_chunk_s", 10.0)) / sim.control_dt)) # how far ahead `at` may be+ self.plan: collections.deque = collections.deque() # (type, row, call id) for each coming control step+ self.running = False+ self.anchor_t = self.anchor_n = 0 # the clock: control step anchor_n was due at anchor_t+ self.watches: list[_Watch] = []+ self.q: queue.Queue = queue.Queue()+ self.max_lag = self.lag = 0.0+ self.stalls = self.held_steps = self.late_steps = 0+ self._ticks: list[tuple[float, float]] = [] # (wall time, call id) per journal row not yet on disk+ super().__init__(sim, **kw)+ self.ticks_path = os.path.join(self.data, "ticks.f64")+ self.ckpt_every = max(1, int(round(CHECKPOINT_S / sim.control_dt)))++ # ------------------------------------------------------------------ spec, status, evidence+ def _spec(self) -> dict:+ spec = super()._spec()+ spec["protocol"] = PROTOCOL+ cdt = self.sim.control_dt+ spec["timing"] = {+ "realtime": True, "slowdown": self.slowdown, "control_dt": cdt,+ "wall_s_per_control_step": cdt * self.slowdown,+ "clock": "an episode starts paused; its clock starts at its first submit or observe with seconds > 0 (also "+ "after a reset) and then runs one control step every wall_s_per_control_step seconds, whatever "+ "you do, until the episode ends",+ "submit": "answers at once: the chunk (rows dt apart, linear or hold) becomes the plan from control step "+ "`at` (default: the next one), replacing what was planned from there; rows for steps already "+ "gone are dropped (meta.late_steps); after the plan's last row that target is held",+ "observe": "seconds = 0: now, at once; seconds > 0: watches the running episode that long and returns what "+ "the cameras saw; never changes the plan",+ "max_ahead_steps": self.max_ahead,+ "no_images": "cameras = [] returns no images, only proprioception (the fastest call)"}+ return spec++ def _realtime_counters(self) -> dict:+ return {"running": self.running, "planned_steps": len(self.plan), "lag_s": round(self.lag, 4)}++ def status(self) -> dict:+ return {**super().status(), **self._realtime_counters()}++ def meta(self) -> dict:+ return {**super().meta(), **self._realtime_counters()}++ def _timing_record(self) -> dict:+ return {"protocol": PROTOCOL, "realtime": 1, "slowdown": self.slowdown, "max_lag_s": round(self.max_lag, 4),+ "stalls": self.stalls, "held_steps": self.held_steps, "late_steps": self.late_steps}++ def _write_final(self) -> None:+ self.verdict.update(self._timing_record()) # final.json merges the verdict; none of these keys is a verdict's+ if self.reason == "error": # a simulator error ended the graded episode: infrastructure (§4b)+ self.verdict["infra_error"] = 1+ else: # (an episode that erred and was reset away is not graded)+ self.verdict.pop("infra_error", None)+ super()._write_final()++ def _checkpoint(self, ev: str, **kw) -> None:+ if self._ticks: # the tick log first: a crash in between leaves it too long+ with open(self.ticks_path, "ab") as f:+ f.write(np.asarray(self._ticks, dtype=np.float64).tobytes())+ self._ticks = []+ super()._checkpoint(ev, **kw)++ def _save_call(self, meta: dict, actions: np.ndarray | None) -> None:+ keep = ("type", "n_rows", "dt", "interpolate", "at", "start_step", "late_steps", "planned_steps", "seconds",+ "control_steps", "episode", "done", "reason", "n_clipped", "lag_s", "note")+ line = {"call": self.calls, "ts": time.time(), **{k: meta[k] for k in keep if k in meta}}+ with open(os.path.join(self.data, "calls.jsonl"), "a") as f:+ f.write(json.dumps(line) + "\n")+ if actions is not None:+ rows = np.concatenate([np.full((len(actions), 1), float(self.calls)), actions], axis=1)+ with open(os.path.join(self.data, "submitted.f64"), "ab") as f:+ f.write(np.asarray(rows, dtype=np.float64).tobytes())++ # ------------------------------------------------------------------ start, clock+ def start(self) -> None:+ super().start()+ rows = np.fromfile(self.ticks_path, dtype=np.float64) if os.path.exists(self.ticks_path) else np.zeros(0)+ n = self.journal_rows+ if rows.size > 2 * n: # the steps a crash lost from the journal go from the tick log too+ rows[: 2 * n].tofile(self.ticks_path)+ elif rows.size < 2 * n: # (an older tick log, short of the journal: pad it as unknown)+ np.concatenate([rows, np.full(2 * n - rows.size, np.nan)]).tofile(self.ticks_path)+ self.running = False # a resumed episode waits, paused, for the agent's next call++ def _start_clock(self) -> None:+ if not self.running and not self.done and not self.stopping:+ self.running, self.anchor_t, self.anchor_n = True, time.monotonic(), len(self.episode)+ self.log("clock_start", control_steps=len(self.episode), episode=self.resets + 1)++ def _stop_clock(self, why: str) -> None:+ if self.running:+ self.running = False+ self.log("clock_stop", why=why, control_steps=len(self.episode), max_lag_s=round(self.max_lag, 4))+ self.plan.clear()++ def _end(self, reason: str) -> None:+ super()._end(reason)+ self._stop_clock(reason) # pending watches are completed by whoever ended it, after this++ def due_in(self, now: float) -> float | None:+ """Seconds until the next control step is due (<= 0: due now); None while the clock is stopped."""+ if not self.running or self.done or self.stopping:+ return None+ return self.anchor_t + (len(self.episode) - self.anchor_n) * self.wall_dt - now++ def tick(self) -> None:+ """One control step of the clock: the plan's row for it, or the held target."""+ now = time.monotonic()+ lag = -(self.due_in(now) or 0.0)+ if lag > MAX_LAG_S: # far behind: re-anchor rather than rush the robot through the debt+ self.stalls += 1+ self.log("stall", behind_s=round(lag, 3), control_steps=len(self.episode))+ self.anchor_t, self.anchor_n, lag = now, len(self.episode), 0.0+ self.lag, self.max_lag = max(lag, 0.0), max(self.max_lag, lag)+ if self.plan:+ typ, row, src = self.plan.popleft()+ else:+ (typ, row), src = self.sim.hold(), -1+ self.held_steps += 1+ before = self.journal_rows+ self._ticks.append((time.time(), float(src)))+ try:+ self._step(typ, row)+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ if self.journal_rows == before:+ self._ticks.pop()+ self.log("error", path="clock", error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ return+ for w in list(self.watches):+ w.rec.step()+ if self.done or len(self.episode) >= w.end:+ self._complete_watch(w)+ if not self.done and len(self.episode) % self.ckpt_every == 0:+ self._checkpoint("tick")++ # ------------------------------------------------------------------ requests (simulator thread)+ def dispatch(self, job: _Job) -> None:+ """Serve one queued request; a watch finishes later, from tick()."""+ try:+ self.first_request()+ path = urlparse(job.path).path+ if job.method == "GET" and path in ("/spec", "/status"):+ return job.finish("raw", *self._reply_json(200, self.spec if path == "/spec" else self.status()))+ routes = {"/submit": self._rt_submit, "/observe": self._rt_observe}+ if self.reset_budget:+ routes["/reset"] = self._rt_reset+ if job.method != "POST" or path not in routes:+ return job.finish("raw", *self._reply_json(404, {"error": "no such endpoint"}))+ if (len(job.body) if job.content_length is None else job.content_length) > MAX_BODY:+ return job.finish("raw", *self._reply_json(413, {"error": f"request body over {MAX_BODY} bytes"}))+ try:+ meta, parts = self._decode(job.body)+ except Exception as exc: # noqa: BLE001 -- any malformed body is the caller's error+ return job.finish("raw", *self._reply_json(400, {"error": f"malformed body: {exc}"[:300]}))+ try:+ routes[path](meta, parts, job)+ except Refused as exc: # 400 (bad_request), 409: nothing changed+ job.finish("raw", *self._reply_json(exc.code, exc.body))+ except Exception as exc: # noqa: BLE001 -- it ran and failed: the episode ends, as on hardware+ traceback.print_exc()+ self.log("error", path=path, error=repr(exc)[:500])+ self._end("error")+ self._release_watches()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "simulator error", "detail": repr(exc)[:200]}))+ except Exception as exc: # noqa: BLE001 -- never leave an HTTP thread waiting forever+ traceback.print_exc()+ if not job.done.is_set():+ job.finish("raw", *self._reply_json(500, {"error": "service error", "detail": repr(exc)[:200]}))++ def _view_rt(self, meta: dict, default_video: str) -> tuple[list[str], int, str, bool]:+ """2.1's _view, plus cameras = [] for no images."""+ none = isinstance(meta.get("cameras"), list) and not meta["cameras"]+ cams, fps, video, depth = self._view(meta, default_video)+ return ([], fps, "none", False) if none else (cams, fps, video, depth)++ def _planned(self, step: int, nxt: int, typ: str) -> np.ndarray:+ """The target in force at control step `step` (>= nxt - 1) as things stand: the plan's row for it, else the+ plan's last row (held), else -- nothing planned, or a step already run -- the current command."""+ i = step - nxt+ if i < 0 or not self.plan:+ return np.asarray(self.sim.command(typ), dtype=np.float64)+ return np.asarray(self.plan[min(i, len(self.plan) - 1)][1], dtype=np.float64)++ def _rt_submit(self, meta: dict, parts: dict, job: _Job) -> None:+ if self.done or self.stopping:+ raise self._conflict()+ nxt = len(self.episode)+ with bad_request():+ typ = meta.get("type")+ if not isinstance(typ, str) or typ not in self.types:+ raise ValueError(f"type must be one of {sorted(self.types)}")+ try:+ A = np.asarray(parts["actions"]["data"] if "actions" in parts else meta.get("actions"),+ dtype=np.float64)+ except (TypeError, ValueError, KeyError):+ raise ValueError("send the actions as an RCB1 array part `actions` or a JSON list of rows") from None+ dim = int(self.types[typ]["dim"])+ if A.ndim != 2 or A.shape[0] < 1 or A.shape[1] != dim:+ raise ValueError(f"{typ} actions must be (T, {dim}) with T >= 1, got {list(A.shape)}")+ if not np.all(np.isfinite(A)):+ raise ValueError("actions contain NaN or inf")+ cdt = self.sim.control_dt+ try:+ dt = cdt if meta.get("dt") is None else float(meta["dt"])+ except (TypeError, ValueError):+ raise ValueError("dt must be a number of seconds") from None+ k = round(dt / cdt) if np.isfinite(dt / cdt) else 0+ if k < 1 or abs(k * cdt - dt) > 1e-9:+ raise ValueError(f"dt must be a positive whole multiple of control_dt = {cdt} s (never rounded)")+ if k > 1 and not self.sim.interpolable(typ):+ raise ValueError(f"{typ}: one row is one control step, so dt must be control_dt = {cdt} s")+ if len(A) * k * cdt > self.max_chunk_s + 1e-9:+ raise ValueError(f"the chunk covers {len(A) * k * cdt:g} s: "+ f"more than max_chunk_s = {self.max_chunk_s:g}")+ interp = meta.get("interpolate", "linear")+ if interp not in ("linear", "hold"):+ raise ValueError("interpolate must be linear or hold")+ at = meta.get("at")+ if at is not None:+ if isinstance(at, bool) or not isinstance(at, (int, float)) or at != int(at) or at < 0:+ raise ValueError("at must be a control step of this episode (an integer >= 0), "+ "or null for the next one")+ at = int(at)+ if at - nxt > self.max_ahead:+ raise ValueError(f"at = {at} is more than max_ahead_steps = {self.max_ahead} control steps ahead "+ f"of the next one ({nxt})")+ at = nxt if at is None else at+ cams, fps, video, depth = self._view_rt(meta, "none")+ rows, clipped = self.sim.clip(typ, A)+ prev = self._planned(at - 1, nxt, typ) # linear starts from the target in force just before `at`+ steps = []+ for row in rows:+ for j in range(1, k + 1):+ steps.append(row if interp == "hold" else prev + (row - prev) * (j / k))+ prev = row+ late = max(0, nxt - at)+ keep = at - nxt if at > nxt else 0+ self.calls += 1+ self.rows += len(A)+ head = list(self.plan)[:keep]+ if len(head) < keep: # a gap before `at`: the target in force then is held through it+ fill = self._planned(at - 1, nxt, typ)+ head += [(typ, fill, -1)] * (keep - len(head))+ self.plan = collections.deque(head + [(typ, s, self.calls) for s in steps[late:]])+ self.late_steps += min(late, len(steps))+ self._start_clock()+ rec = Recorder(self.sim, cams, fps, depth, "none")+ rec.start()+ rec.finish()+ out = {**self.meta(), "chunk_id": self.calls, "type": typ, "n_rows": int(len(A)), "dt": dt,+ "interpolate": interp, "at": at, "start_step": max(at, nxt), "late_steps": late,+ "plan_end_step": nxt + len(self.plan), "clipped": clipped[:200], "n_clipped": len(clipped), "ok": True}+ self._save_call({**out, "note": str(meta.get("note") or "")[:200]}, A)+ job.finish("rec", out, rec)++ def _rt_observe(self, meta: dict, parts: dict, job: _Job) -> None:+ with bad_request():+ try:+ seconds = float(meta.get("seconds") or 0.0)+ except (TypeError, ValueError):+ raise ValueError("seconds must be a number") from None+ if not 0.0 <= seconds <= self.max_watch_s:+ raise ValueError(f"seconds must be in [0, {self.max_watch_s:g}]")+ cams, fps, video, depth = self._view_rt(meta, "frames")+ n = int(round(seconds / self.sim.control_dt)) if not (self.done or self.stopping) else 0+ rec = Recorder(self.sim, cams, fps, depth, video if n or video != "mp4" else "frames") # no time: no video+ self._frame_budget(rec, n)+ self.calls += 1+ rec.start()+ if not n:+ rec.finish()+ out = {**self.meta(), "obs_id": self.calls, "seconds": 0.0, "control_steps_watch": 0, "ok": True}+ self._save_call(out, None)+ return job.finish("rec", out, rec)+ self._start_clock()+ self.watches.append(_Watch(job, rec, len(self.episode), n, cams, self.calls))++ def _complete_watch(self, w: _Watch) -> None:+ self.watches.remove(w)+ w.rec.finish()+ ran = max(0, min(len(self.episode), w.end) - w.start)+ out = {**self.meta(), "obs_id": w.obs_id, "seconds": round(ran * self.sim.control_dt, 4),+ "control_steps_watch": ran, "ok": True}+ self._save_call(out, None)+ w.job.finish("rec", out, w.rec)++ def _release_watches(self) -> None:+ for w in list(self.watches):+ self._complete_watch(w)++ def _rt_reset(self, meta: dict, parts: dict, job: _Job) -> None:+ self._release_watches()+ out, prts = self._reset(meta, parts) # 2.1's reset: checks, abandons the episode, a new one (paused)+ self._stop_clock("reset")+ job.finish("rcb1", {**out, **self._realtime_counters()}, prts)++ # ------------------------------------------------------------------ the loop (simulator thread) and HTTP+ def poll(self) -> None:+ """After SIGUSR1: end the episode, answer the pending watches, finalize (Service.poll, plus the watches)."""+ if self.stopping and not self.finalized:+ if not self.done:+ self._end("timeout")+ self._release_watches()+ self._finalize()++ def loop(self, until: threading.Event | None = None) -> None:+ """The simulator thread: control steps when due, requests in between, the wrap-up after SIGUSR1."""+ while until is None or not until.is_set():+ self.poll()+ wait = self.due_in(time.monotonic())+ if wait is not None and wait <= 0:+ self.tick()+ continue+ try:+ job = self.q.get(timeout=0.25 if wait is None else min(wait, 0.25))+ except queue.Empty:+ continue+ self.dispatch(job)++ def submit_job(self, job: _Job) -> tuple[int, str, bytes]:+ """From an HTTP thread: queue the request, wait for the simulator thread, encode the answer here."""+ self.q.put(job)+ job.done.wait()+ kind, *rest = job.reply+ if kind == "raw":+ return tuple(rest)+ meta, payload = rest # a Recorder's parts are encoded here, off the simulator's thread+ return self._reply_rcb1(meta, payload.parts() if kind == "rec" else payload)++ def http_server(self, address: tuple[str, int]) -> ThreadingHTTPServer:+ svc = self++ class Handler(BaseHTTPRequestHandler):+ timeout = 120++ def log_message(self, *a):+ pass++ def _send(self, code: int, ctype: str, body: bytes) -> None:+ self.send_response(code)+ self.send_header("Content-Type", ctype)+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_GET(self):+ self._send(*svc.submit_job(_Job("GET", self.path, b"")))++ def do_POST(self):+ try:+ n = int(self.headers.get("Content-Length") or 0)+ except ValueError:+ return self._send(*svc._reply_json(400, {"error": "bad Content-Length"}))+ if n > MAX_BODY: # refused unread: drop the connection with it+ self.close_connection = True+ return self._send(*svc.submit_job(_Job("POST", self.path, b"", content_length=n)))+ self._send(*svc.submit_job(_Job("POST", self.path, self.rfile.read(n))))++ class Server(ThreadingHTTPServer):+ daemon_threads = True++ return Server(address, Handler)++ def serve(self, port: int = 8000) -> None:+ import signal+ signal.signal(signal.SIGUSR1, self.on_sigusr1)+ srv = self.bind(port)+ threading.Thread(target=srv.serve_forever, name="http", daemon=True).start()+ print(f"{rcb_service.PROTOCOL} sim service (realtime, slowdown {self.slowdown:g}) ready on :{port}", flush=True)+ self.loop()+++def main(build, port: int = 8000) -> None:+ """rcb_service.main -- the start count, back-off, startup watchdog, resume -- with this service in place of+ Service (RCB_SLOWDOWN: 1 = real time), and eai-standard/2.3 in everything it logs and writes."""+ rcb_service.Service = RealtimeService+ rcb_service.PROTOCOL = PROTOCOL+ rcb_service.main(build, port)+++if __name__ == "__main__":+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+ import server # noqa: E402 -- the HumanoidBench adapter; picks the GL backend and restores the reward on import++ main(server.build)diff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb_rewards.py b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb_rewards.pysimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb_rewards.pyrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb_rewards.pydiff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb_service.py b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb_service.pysimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rcb_service.pyrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rcb_service.pydiff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rewards.json b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rewards.jsonsimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/sim/rewards.jsonrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/rewards.jsondiff --git a/tasks/humanoidbench-manip-window-i00-realtime/environment/sim/server.py b/tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/server.pysimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/environment/sim/server.pyrename to tasks/humanoidbench-manip-window-i00-standard-realtime/environment/sim/server.pydiff --git a/tasks/humanoidbench-manip-window-i00-realtime/instruction.md b/tasks/humanoidbench-manip-window-i00-standard-realtime/instruction.mdsimilarity index 98%rename from tasks/humanoidbench-manip-window-i00-realtime/instruction.mdrename to tasks/humanoidbench-manip-window-i00-standard-realtime/instruction.mdindex ab9a1495e0..3c726053ea 100644--- a/tasks/humanoidbench-manip-window-i00-realtime/instruction.md+++ b/tasks/humanoidbench-manip-window-i00-standard-realtime/instruction.md@@ -1,4 +1,4 @@-# window — instance 00 — REALTIME MODE (robot as a service; time does not stop)+# window — instance 00 — STANDARD-REALTIME MODE (robot as a service; time does not stop)You control a simulated Unitree H1 humanoid with two Shadow dexterous hands (HumanoidBench / MuJoCo) **as ifit were real hardware**: the simulator runs in a service you cannot open, and you reach it only through a@@ -74,7 +74,7 @@ order of half a metre per second, for keeping both hands near the tool, and for(`reason = "success"`), and a fresh replay of the recorded episode, the servo targets the robot actuallyexecuted at each control step, must reach it as well and end in the same state.-## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, realtime mode)+## Client (`/opt/tools/rcb_client.py`, protocol `eai-standard/2.3`, standard-realtime mode)Your whole interface to the robot. Its docstring is the full reference(`python -c "import rcb_client; help(rcb_client)"`); the calls and the endpoints they use:diff --git a/tasks/humanoidbench-manip-window-i00-realtime/task.toml b/tasks/humanoidbench-manip-window-i00-standard-realtime/task.tomlsimilarity index 70%rename from tasks/humanoidbench-manip-window-i00-realtime/task.tomlrename to tasks/humanoidbench-manip-window-i00-standard-realtime/task.tomlindex dddfb59684..10b27e5392 100644--- a/tasks/humanoidbench-manip-window-i00-realtime/task.toml+++ b/tasks/humanoidbench-manip-window-i00-standard-realtime/task.toml@@ -1,4 +1,4 @@-# Realtime-mode task of humanoidbench-manip-window-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit+# Standard-realtime task of humanoidbench-manip-window-i00-privileged (docs/PROTOCOL.md §4b). Generated from scripts/humanoidbench/templates/realtime/task.toml -- edit# subset.toml or the template, not this.schema_version = "1.3"artifacts = [@@ -16,10 +16,10 @@ artifacts = [][task]-name = "robot-coding-bench/humanoidbench-manip-window-i00-realtime"-description = "REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench window on the same frozen instance as humanoidbench-manip-window-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."+name = "robot-coding-bench/humanoidbench-manip-window-i00-standard-realtime"+description = "STANDARD-REALTIME MODE (robot as a service on a clock that does not stop, eai-standard/2.3): HumanoidBench window on the same frozen instance as humanoidbench-manip-window-i00-privileged. Time does not stop: once an episode starts, the robot runs at 50 Hz in real time and holds its last targets while the agent thinks, so the agent streams joint-position targets from a controller program of its own through the client (/observe, /submit with a plan from step `at`, /reset; head-camera RGB, joint proprioception, a pelvis IMU, a fixed room camera). Up to 50 resets, no privileged state, no reward; the sim service records every executed control step, replays the last episode in a fresh simulator and grades it."authors = [{ name = "pingyue" }]-keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "realtime-mode", "trajectory", "window"]+keywords = ["robotics", "humanoidbench", "mujoco", "h1hand", "manipulation", "standard-realtime-mode", "trajectory", "window"][metadata]author_name = "pingyue"@@ -27,7 +27,7 @@ task_type = "trajectory"simulator = "humanoidbench"robot = "unitree-h1hand"family = "window"-mode = "realtime"+mode = "standard-realtime"protocol = "eai-standard/2.3"instance_seed = 0expert_time_estimate_hours = 8.0diff --git a/tasks/humanoidbench-manip-window-i00-realtime/tests/Dockerfile b/tasks/humanoidbench-manip-window-i00-standard-realtime/tests/Dockerfilesimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/tests/Dockerfilerename to tasks/humanoidbench-manip-window-i00-standard-realtime/tests/Dockerfilediff --git a/tasks/humanoidbench-manip-window-i00-realtime/tests/grade.py b/tasks/humanoidbench-manip-window-i00-standard-realtime/tests/grade.pysimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/tests/grade.pyrename to tasks/humanoidbench-manip-window-i00-standard-realtime/tests/grade.pydiff --git a/tasks/humanoidbench-manip-window-i00-realtime/tests/test.sh b/tasks/humanoidbench-manip-window-i00-standard-realtime/tests/test.shsimilarity index 100%rename from tasks/humanoidbench-manip-window-i00-realtime/tests/test.shrename to tasks/humanoidbench-manip-window-i00-standard-realtime/tests/test.sh