The first exam for code agents in the physical world: frozen robot tasks from many benchmark suites, each answered with a program and verified in a separate sandbox. Runs on Harbor.
Standard-mode wire protocol eai-standard/2.x (robot as a service)
Part of the task protocol (README): the wire format of the standard modes.
Version: eai-standard/2.4 is the current version. This document is the interoperability
specification of every 2.x version a task declares today: what a service answers and a client may send. Minor
versions only add, so each part holds for the versions after it unless a later part says otherwise, and the shipped
clients accept every 2.x service. What a task of the standard modes is, and which rules bind it, is in
the README (§3 Modes); the binding rules on the trust boundary (no simulator, state or success
predicate in the agent's reach; the verdict from a trusted record) are stated once, in
its Appendix H (H.3, H.4), and are not repeated here. How a runner handles a trial that failed
for infrastructure reasons is in running.md. What a benchmark's own service adds (its robot, action
types, cameras, proprioception columns, resume class, measured start-up times, known issues) is in its
tasks/<benchmark>/README.md.
| Version | Content | Part | Implementation |
|---|---|---|---|
eai-standard/2.1 |
the base: /submit and /observe, RCB1 responses, measured proprioception, optional depth noise; the optional reset capability |
Part 1 | each benchmark's own service |
eai-standard/2.2 |
2.1 + optional capabilities: depth last / frames, per-frame steps, times and extrinsics, camera intrinsics and mounting, robot_spec, world, POST /kinematics, a per-call wall-clock bound; one shared service layer |
Part 2 (§1–§18) | embodied_first_exam/standard/ |
eai-standard/2.3 |
2.2 + the realtime capability: a clock that does not stop | Part 3 | embodied_first_exam/standard/realtime/rcb_realtime.py |
eai-standard/2.4 |
2.3 + the operations capability: operation ids, request deadlines, GET /operations/<id>, a call watchdog, a physics-tick ledger |
Part 4 (§19–§25) | embodied_first_exam/standard/v2_4/ |
Which version a task serves is its [metadata] protocol (= /spec.protocol); which benchmarks serve which version is
in their docs, not listed here.
Spelling. Services and tasks report eai-standard/2.x; the shipped clients also accept rcb-limited/2.x. No task
declares a version before 2.1.
Citing this document. Numbered sections are unique in the document: §1–§18 are Part 2 (so "§13" is the shared layer's crash recovery), §19–§25 are Part 4. The sections of Parts 1 and 3 are cited by their titles. A section whose content is per benchmark points to the benchmarks' docs.
Part 1 — eai-standard/2.1: the base
The agent talks to a sim service it cannot open, over HTTP. One trial is one episode unless the task declares a
reset budget (the optional reset capability, below). No step limit (except a technical episode cap where a task
states one in spec.max_control_steps); the agent's wall-clock budget is the real limit; actions and simulated time
are reported as cost. What the service never reveals (object poses, scene files, seeds, the success predicate) is
PROTOCOL Appendix H.3; contact or safety telemetry is published only where a benchmark's doc declares it field by
field. Videos, frames, depth and proprioception come back as binary in the response body (RCB1 frames, below),
never base64; the agent's code decides what to keep.
Base URL: http://sim:8000 (compose network) or http://localhost:8000 (Harbor egress control puts the service in the
agent's network namespace); the client tries both. The socket opens only when the scene is ready; until then
connections are refused and the client retries.
Endpoints
| method | path | body | response |
|---|---|---|---|
| GET | /spec |
— | JSON: action types (dims, layouts, limits, semantics), cameras, proprio layout, budgets |
| GET | /status |
— | JSON: {step, calls, physics_steps, done, reason, elapsed_s} |
| POST | /observe |
RCB1 or JSON {cameras, seconds, video_fps, include_depth, video} |
RCB1 |
| POST | /submit |
RCB1: header {type, dt, interpolate, cameras, video_fps, video, include_depth, note} + part actions float32 (T, dim) |
RCB1 |
Anything else (/state, /success, …) returns 404; so does /reset unless the task declares a reset budget
(below).
RCB1 frame
"RCB1" (4 B) | header_len uint32 LE (4 B) | header JSON (UTF-8) | blob_0 | blob_1 | …
header = {"v": 1, "meta": {...}, "parts": [{"name","kind","len", ...}, ...]} # blobs in this order, no gaps
kinds: "mp4" (fps, n) · "array" (dtype, shape; C-order raw bytes) · "jpeg" · "json" · "bytes"
Response parts: video/<cam> (mp4), frames/<cam> (uint8 [N,H,W,3]), last/<cam> (uint8 [H,W,3], always),
depth/<cam> (float16 [N,H,W], if requested), cam2world_gl/<cam> (float64 [N,4,4], for a camera that moves: its pose
at each returned frame), proprio (float32 [N, len(proprio layout)], one row per video frame; the columns are the
benchmark's). Requests over 64 MB → 413 (the client refuses to send one); a response over 256 MB drops
frames//depth/ and says so in meta.degraded. rcb1.py (encode/decode, python -m rcb1 unpack f dir) is shipped
with the client.
meta (in every response)
step (rows executed so far), calls, physics_steps, sim_time_s, time_s, time_left_s, proprio =
{"layout": [names], "now": [floats]}, cameras (per camera intrinsic_cv, extrinsic_cv, cam2world_gl; the whole
key omitted when the task sets RCB_OBS_CALIBRATION=0), done, reason. cameras is the calibration of the state the
response ends in (the last frame, the final sample of frames / depth): static cameras never move, a camera on the
robot moves with it, and for each requested moving camera the response carries the part cam2world_gl/<cam> (one
pose per returned frame; spec.camera_calibration). time_s / time_left_s are the agent's clock (below; /spec
states it in budget.clock or budget.wall_clock_note). /submit adds chunk_id, type, executed_rows, physics_steps_chunk, clipped [{row,index,reason}], n_clipped, plan [...], ok; /observe adds obs_id, seconds, physics_steps_watch.
/submit semantics
joint_positionrows aredtseconds apart,dta multiple of the physics or control step and at most 0.5 s; anything else is rejected (400), never rounded. The service streams drive targets withinterpolate = "linear"(default) or"hold"between rows, exactly as a real controller consumes a chunk. Out-of-limit values are clipped and reported (continuous joints havenulllimits and are never clipped). A chunk may cover at mostmax_chunk_s.- Planner-executed types such as
ee_posestate their frame, quaternion order and end-effector reference in/spec; a failed plan is reported inmeta.plan, not as an error. - Types a service does not offer yet are listed under
spec.action_types.not_yet. The action layout, the action types and their dtype are the benchmark's (its doc). - Video is captured every
physics_hz / video_fpsphysics steps from the requested cameras, plus the first and last frame. - After the chunk the task predicate is evaluated; the first success ends the episode (
done,reason = "success"). The chunk is appended to the recorded trajectory and toepisode.jsonl.
/observe semantics
seconds = 0 (default) returns the current frames, free (no physics). seconds > 0 (≤ max_watch_s) holds the last
command, advances physics and returns the video; the predicate is checked afterwards. Watching costs no budget beyond
the wall clock; the simulated time is recorded as cost.
Errors (no physics executed)
| code | when |
|---|---|
| 400 | wrong type/dim, a chunk with no rows, NaN/inf, a field of the wrong JSON type (dt, seconds or video_fps not a finite number, cameras not a list), dt or seconds out of range or dt not a multiple of the physics step, chunk too long, bad interpolate/video value, video_fps outside [1, 30], more images than max_frames_per_call in one response (PROTOCOL §3), malformed body |
| 409 | episode already finished (done); the trial is over |
| 413 | body over 64 MB (rcb_client.py raises before sending such a body) |
| 500 | simulator error; the episode ends with reason = "error" (the client raises EpisodeOver; the verifier reports sim_error = 1) |
Episode semantics
The service starts the episode when it is ready (seed hidden inside). The first success ends it; otherwise it ends
when the agent phase ends (timeout) or on an error. The predicate is evaluated once more at episode end.
The agent's clock (every 2.x service). meta.time_s is the wall time since the agent's first
request to the service, whatever the endpoint (GET /spec, which the shipped clients send first, counts), and
meta.time_left_s = spec.budget.wall_clock_s − time_s, never below 0. The clock does not start with the service,
when the scene is ready, or with a probe that only connects to the port; the first request is logged in
episode.jsonl (first_request) and a restarted service reads it back, so a restart never sets the clock back to
full. GET /status elapsed_s and the elapsed_s of final.json are the episode's own time, from the service's
start of the episode. Why: Harbor counts the agent's cutoff around the agent's run only, and the sidecar starts with
the environment, before the harness is installed, so a clock started with the service would run ahead of the agent's.
With this rule time_left_s never runs out before the agent is
stopped, and the agent's real time left is time_left_s minus what it spent before its first request. A service that
ends an episode on its own clock says so in its benchmark's doc. A readiness probe added to a compose file or a runner
must not send a request to the service before the agent runs: connecting to the port is enough.
Service crashes (reference implementation). The reference services journal every executed step and log a
checkpoint (step count + state hash) per completed call; when the service dies mid-episode, compose restarts it
(restart: on-failure:N) and it resumes the same episode from the journal, the call in flight lost as a whole
(spec.on_service_error tells the agent to check GET /status before continuing). This is how the shipped services
behave by default (details: §13), not a rule: what the protocol says about a trial that failed
for infrastructure reasons is the runner's rerun (PROTOCOL Appendix H.5, running.md). A
verdict already written is never overwritten. A scene build that hangs is killed by a startup watchdog
(RCB_STARTUP_TIMEOUT_S; each service has its own default, and each task sets the value its build needs, §13.3). A host
on which the frozen instance does not rebuild to its stored initial state reports reason = instance_mismatch with
infra_error = 1.
Evidence and grading (agent never sees this)
/data/episode.jsonl (one record per call: type, rows, dt, cameras, note, clipped, plan, physics steps, success),
/data/calls/NNNNNN/ (meta + last frames per call), and on end: trajectory.npz (the graded episode, the deliverable),
final.json, then replay.mp4. The collect hook signals the service by PID after a collect hook has stopped
the agent's processes; from then on the service refuses every request, and it judges the episode it ran (the task's success rule, with any settle or hold window run on the live simulator)
and writes the verdict into final.json (success, the verdict; live_success, how the episode itself ended, before
any settle or hold check; graded_by = live, replays = 0); nothing is
replayed to grade it (PROTOCOL H.4). The video is a post-hoc replay of trajectory.npz after the verdict by default, and a live
recording where the simulator is not verified bit-wise replayable (video_source, PROTOCOL §4); it never decides
anything. With resets, the shared 2.1 sidecar service (rcb_service.py, one file copied unchanged into each benchmark
that runs it, also under the 2.3 realtime layer) keeps the first episode as evidence too: trajectory.first.npz (its
rows from the journal) and replay.first.mp4, rendered after the graded video; final.json says how it ended
(first_success, first_reason; a success ends the trial, so after a reset first_success is 0) and decides nothing
with it. A separate verifier transcribes final.json into reward.json (graded_by_live = 1); the shared grader
(embodied_first_exam/standard/v2_1/grade.py) also writes first_success, the first-attempt, no-reset result reported
beside the graded one, and copies the first episode's files. A service of a benchmark's own keeps no first episode and
reports resets only; first_success is then success when no reset was made, else 0 (running.md "Results").
infra_error = 1 means there is no verdict on the agent: final.json is
missing, unreadable or not finalized, or the service reported sim_crashed, resume_failed, instance_mismatch or
(2.4) call_watchdog.
Live recording (sidecar only; the agent sees none of it)
live.mp4, live_timeline.jsonl, live_segments.jsonl, trajectory_live.f64, startups.jsonl: captured during the
episode, or rendered from the recorded trajectory where a post-hoc replay is allowed (PROTOCOL §4), frames stamped with
the call, never affecting the verdict. The reference recording is RoboTwin's
(tasks/robotwin-2/README.md).
Proprioception (2.1)
A benchmark's 2.0 columns stay where they were; 2.1 appends what a real robot reports (measured joint positions and
velocities, the gripper opening, the tool-centre pose, a drive-effort estimate), grouped in spec.proprio_groups (first
column, count, meaning). The columns are the benchmark's (its doc). Not exposed: object poses, segmentation, contact
forces on objects, the success predicate (PROTOCOL Appendix H.3).
Depth (include_depth) is exact by default. RCB_DEPTH_NOISE=realsense (task env) adds stereo-camera-like noise,
σ = 2 mm + 2.5 mm·z² (z in m), and 1 % dropouts (0 = no reading), seeded by RCB_DEPTH_NOISE_SEED; it never affects
physics.
Per-benchmark 2.1 services
The MuJoCo service (images/mujoco/limited/server_mj.py: DexToolBench, MuJoCo Playground manipulation),
the shared KinDER / HumanoidBench service (rcb_service.py), the MetaWorld+ and RoboCasa services serve the same
endpoints and frames with their own action layouts, control rates and cameras: tasks/dextoolbench/README.md,
mujoco-playground.md, kinder.md, humanoidbench.md, metaworldplus.md, robocasa.md.
Optional reset capability
Each task declares its own budget N (PROTOCOL §2): the RCB_RESET_BUDGET default of its sim service, served as
/spec.budget.reset and stated in its instruction (templates: 0 in both standard and realtime mode). Any
2.x service may implement the capability; which ones do is in their benchmarks' docs. With N > 0, /spec.budget.reset = N, /spec.reset describes it, and POST /reset (RCB1 or JSON {cameras, video}, no physics) abandons the running
episode and starts a new one from the same frozen state, N times in all; /status and meta add episode,
resets_left and control_steps_total. After a success or once the agent's time is up, /reset returns 409, as it does
when the budget is spent. Only the last episode is graded. With N = 0 (or no declaration) /reset is 404. The reset
budget is part of a result's configuration: results with different budgets are not ranked together (running.md).
Optional agent-ended episodes
PROTOCOL §3 "Agent-ended episodes" (an optional task feature). A task opts in in its sim service's environment:
RCB_AGENT_END: "1", and RCB_STEP_LIMIT: "<N>" for a step limit (what one step is: the service's /spec and the
task's instruction). The wire:
/spec.success_signalsays the success is hidden; with a step limit/spec.budget.steps = /spec.max_control_steps = N, and everymetaand/statusaddstep_limitandsteps_left.- No answer depends on the task's success: no
done/reason = "success", no 409, no early end of a chunk and no per-call work caused by it,resets_leftand/resetas before a success, no success or progress field of the service's own (stages reached, a settled-check counter) inmetaor/status. POST /finish(JSON{}) ends the episode (reason = "finished") and answers{"ok": true, ...}with/status's fields; after it every call butGET /specandGET /statusis a 409. Without the feature it is a 404.- At the step limit the episode ends by itself (
reason = "step_limit"; the chunk stops at that step); then/submitand/finishare 409,/observestill answers. - The collect hook's signal at the end of the agent's session ends the episode with
reason = "timeout". - The verdict: the task's own rule on the state the episode ended in, with its settle or hold window run from there on the live simulator (a hold window counts the steps already held). A success reached and then undone is 0. A success the task latches as an event (a referee's call, a completed stage sequence) is part of the state: the task's failure rules keep running after it and still end the episode, but do not take it away. A simulator error is judged as without the feature.
final.jsonaddssuccess_signal = "hidden",ended_by(finish,step_limit,session_end,failed: the task's own failure rule,error),hidden_success_seen,hidden_success_first_step,trial_hidden_success_seen,finish_time_sand, with a step limit,step_limitandsteps_used;reward.jsonaddsended_finish,ended_step_limit,ended_session_end(evidence only). Which limit ended a session (session_end) is the runner's record: Harbor'sAgentTimeoutErrorfor the time limit, the gateway'sbudget_stopfor the token budget.- Implementation for the 2.2 layer:
embodied_first_exam/standard/agent_end/rcb_agent_end.py(Service,serve,hidden_success,final_fields,steps_left; files-only imageagent-end-files); an adapter withoutfinish()refuses to start with the feature on; testsagent_end_selftest.py.
Part 2 — eai-standard/2.2: capabilities added to 2.1
2.2 is a minor version of
eai-standard/2.1(Part 1): everything there holds unless this part says otherwise, every 2.2 addition is an optional capability, and a request written for 2.0 or 2.1 gets the same answer as before (plus parts and keys an old client ignores).
One implementation. The source is embodied_first_exam/standard/: the service layer
rcb_limited.py (HTTP, RCB1 responses, the protocol helpers, the /spec self-check, crash recovery), the client
rcb_client.py + rcb1.py, the sidecar entrypoint run.sh, and the client image: client.env (the one client base,
pinned by digest) and client-requirements.txt (exact pins). Every 2.2 task carries the code unchanged (environment/rcb_client.py, environment/rcb1.py, environment/sim/{rcb_limited.py,rcb1.py,run.sh}). Each
benchmark writes only its adapter, environment/sim/server.py: the robot, the cameras, the clock (toy_server.py is
the smallest complete one).
Design principles.
- A robot, not a simulator. No state query, no success query, no step limit; resets only within the task's declared budget; the wall clock is the budget; actions and simulated time are reported as cost; the first success ends the episode.
- One information line. The agent may know the robot and its sensors (spec sheet, kinematics, camera calibration, proprioception), never the world's ground truth (§10).
- Everything new is optional. Depth, for example, costs a render and is off by default.
- Binary data, kept by the agent's code. RCB1 frames, never base64; nothing is pushed into the model's context.
- Evidence the agent cannot touch. The service records every call and judges the episode it ran, in a container the agent never saw; the recorded trajectory stays replayable (the video, audits).
1. Endpoints
| method | path | body | response |
|---|---|---|---|
| GET | /spec |
— | JSON: 2.1's keys + cameras' intrinsics and mounting, proprio groups, robot_spec, world, kinematics, depth, on_service_error, max_call_wall_s |
| GET | /status |
— | JSON: {step, calls, control_steps, done, reason, elapsed_s, resumes} |
| POST | /observe |
RCB1 or JSON {cameras, seconds, video_fps, video, depth} |
RCB1 |
| POST | /submit |
RCB1: header {type, dt, interpolate, cameras, video_fps, video, depth, note} + part actions float32 (T, dim) |
RCB1 |
| POST | /kinematics (2.2) |
JSON {op: "fk" | "ik", ...} (§8) |
JSON |
| POST | /reset |
as in Part 1 | only where the task declares a reset budget N > 0 (Part 1, "Optional reset capability") |
Anything else (/state, /success, /reset without a budget) returns 404. include_depth (2.0/2.1) is still
accepted (§7). The shared 2.2 client has no reset(); an instruction whose task has a budget calls POST /reset
through Robot._post.
2. Response parts
Every recorded frame of a call (the first, one every control_hz / video_fps control steps, the last) has one entry on
one shared frame axis N: the rows of frames/<cam>, the frames of video/<cam>, proprio, frame_step,
frame_time_s, extrinsic_cv/<cam> and depth_frames/<cam>.
| part | kind | content |
|---|---|---|
video/<cam> |
mp4 | H.264 of the N frames (video = mp4 | both) |
frames/<cam> |
array uint8 (N,H,W,3) | the raw frames (video = frames | both) |
last/<cam> |
array uint8 (H,W,3) | the last frame, always |
proprio |
array float32 (N,D) | one proprio row per frame, columns spec.proprio_layout |
frame_step (2.2) |
array int64 (N,) | control steps executed since the episode start when the frame was rendered |
frame_time_s (2.2) |
array float64 (N,) | its simulated time, frame_step / control_hz |
extrinsic_cv/<cam> (2.2) |
array float64 (N,3,4) | world → camera (OpenCV axes) at that frame (constant for a static camera) |
last_depth/<cam> (2.2) |
array float32 (H,W) | z-depth of the last frame in metres (depth = last | frames) |
depth_frames/<cam> (2.2) |
bytes, codec: "zlib", dtype: "uint16", shape: [N,H,W], unit: "mm" |
z-depth of every frame (depth = frames); the client decodes it to float32 metres |
depth/<cam> (2.1) |
array float16 (N,H,W) | z-depth of every frame in metres, only for a 2.1 request (include_depth: true without depth) |
A response over 256 MB drops depth_frames/ and depth/ first, then frames/ (mp4, last frames, last depth, proprio and
the per-frame arrays always stay) and says so in meta.degraded. Requests over 64 MB → 413.
3. Request fields (/submit header, /observe body)
| field | default | meaning |
|---|---|---|
cameras |
every camera | names from spec.cameras; the client sends spec.default_cameras when given none |
video_fps |
the service's | frames per simulated second |
video |
mp4 (submit), frames (observe) |
mp4 | frames | both | none |
depth (2.2) |
none |
none | last | frames; JSON true = last |
include_depth (2.0/2.1) |
false |
with no depth field: true answers as 2.1 did (depth/<cam> float16 per frame) |
seconds (observe) |
0 | 0 = the current frames, no physics; > 0 (≤ max_watch_s) = hold the last command and watch |
type, dt, interpolate, note (submit) |
as 2.1 |
4. meta
2.1's keys (step, calls, control_steps, physics_steps, sim_time_s, time_s, time_left_s, proprio, done,
reason), with cameras = per requested camera intrinsic_cv, extrinsic_cv, cam2world_gl, image_size at the last
frame (omitted when the task sets RCB_OBS_CALIBRATION=0). /submit adds chunk_id, type, executed_rows, control_steps_chunk, physics_steps_chunk, clipped [{row,index,reason}], n_clipped, plan [...], ok; /observe adds
obs_id, seconds, control_steps_watch, physics_steps_watch. Both add stopped (2.2): null, or why the service
stopped the call early (§12).
5. /spec
2.0/2.1 keys, unchanged: protocol ("eai-standard/2.2"), simulator, robot, task_text, control_hz,
physics_substeps, max_chunk_s, max_watch_s, cameras, action_types, proprio_layout, proprio_groups, static
(scene facts: names, sizes, colours; never a position), budget ({"reset": N, "steps": null, "wall_clock_s", "clock"}; clock says what time_s / time_left_s count from: the agent's first request, Part 1 "The agent's clock";
the shared layer fills it in and keeps the clock, rcb_limited.agent_time). 2.2 adds:
| key | content |
|---|---|
protocol_compat |
["rcb-limited/2.0", "rcb-limited/2.1", "eai-standard/2.1"]: requests written for these are served unchanged |
default_cameras |
the cameras the client asks for when a call names none |
cameras.<cam> |
+ depth: true, intrinsic_cv (3×3), mount, depth_range_m |
robot_spec |
the robot's spec sheet (below) |
world |
frame (origin and axes in words), up, units, quaternion_order (wxyz | xyzw, used by every pose of this service), camera_axes |
kinematics |
what /kinematics answers (§8) |
depth |
{"modes": ["none", "last", "frames"], "default": "none", "kind": "z-depth", "unit": "m", "invalid": 0, "max_m": 65.535, "frames_encoding": "uint16 mm, zlib", "noise": "off" | "realsense"} |
on_service_error |
what to do after a connection error or a 5xx (§13) |
max_call_wall_s |
the per-call wall-clock bound (§12) |
Camera geometry. intrinsic_cv K maps a point (X, Y, Z) in the camera's OpenCV frame (x right, y down, z forward) to
u = fx·X/Z + cx, v = fy·Y/Z + cy; the pixel that contains it is (column ⌊u⌋, row ⌊v⌋). extrinsic_cv (3×4) maps world to
that frame; cam2world_gl (4×4) is the camera pose with OpenGL axes = inverse(extrinsic_cv) · diag(1, −1, −1, 1).
mount is {"type": "static", "extrinsic_cv", "cam2world_gl"} or {"type": "link", "parent_link": L, "link_T_cam_cv"}
(world_T_cam = FK(L) · link_T_cam_cv for any joint configuration; L is in kinematics.links).
robot_spec (every value is the robot's own; nothing about the scene):
| key | content |
|---|---|
name, description |
the robot |
base |
{"type": "fixed", "pose"} or {"type": "mobile", ...} (root link, where its pose is in proprio, how it is commanded, its limits) |
joints |
every driven joint: {name, type, unit, position_limits, velocity_limit, effort_limit, group} = the limits the SIMULATOR enforces (null where it enforces none), plus, where known, rated_velocity / rated_effort and rated_source |
grippers |
per gripper: joints, opening_m, command (entries, range, open / close rule), tcp_frame |
controller |
type, control_hz, physics_hz, gains ({joint: {"stiffness", "damping"}}, the drives' own), note |
tcp |
per tool frame: {"parent_link", "link_T_frame", "definition"}: the fingertip centre and the ee_pose frame, with the offset between them |
6. Actions
As 2.1 (joint_position rows dt apart, a multiple of the control period, at most 0.5 s, linear or hold, clipped and
reported; a chunk covers at most max_chunk_s). ee_pose rows are end-effector targets solved by the benchmark's IK or
planner (its doc); a failed row is reported in plan. Not yet: joint_delta, joint_velocity, ee_delta,
joint_torque (spec.action_types.not_yet). After a chunk the task predicate is evaluated; the first success ends the
episode.
7. Depth
z-depth = the distance from the camera plane along the optical axis, in metres, for the same pixels as the RGB frame.
0 means no reading. depth: "last" returns last_depth/<cam> (float32, one depth render per camera per call);
depth: "frames" also returns depth_frames/<cam> (uint16 millimetres, zlib; decoded by the client to float32 metres,
1 mm steps). Exact by default; RCB_DEPTH_NOISE=realsense adds stereo-like noise (σ = 2 mm + 2.5 mm·z², 1 % dropouts,
seeded by RCB_DEPTH_NOISE_SEED), as 2.1. Depth never affects physics.
8. Kinematics
POST /kinematics is stateless: the answer depends only on the request, it never steps physics, never advances time,
never ends the episode and costs only wall clock. It uses the robot's kinematic model (the same as the ee_pose action and
the camera mounts), with no collision checking. spec.kinematics: endpoint, joint_names (the q layout; each a
proprio_layout column), frame (world, or base for a mobile robot unless the request gives base_pose),
base_pose_proprio, quaternion_order, links, tcp_frames, ee_frames, ik_targets, ik (solver, defaults).
| request | response |
|---|---|
{"op": "fk", "q", "links"?, "base_pose"?} |
{"ok": true, "op": "fk", "frame", "poses": {link: [x,y,z, quaternion]}} |
{"op": "ik", "targets": {frame: pose}, "seed"?, "base_pose"?, "tol_pos_m"?, "tol_rot_rad"?, "max_iters"?} |
{"ok", "op": "ik", "frame", "q", "error": {frame: {"pos_m", "rot_rad"}}} |
A malformed request → 400; no solution is ok: false, never an error.
9. Proprioception
2.1's rule, kept: a benchmark's 2.0 columns stay where they were; later groups are appended; every column has a name.
Every 2.2 service has at least the groups joint_pos, joint_vel, gripper_opening (metres between the fingertips),
tcp_pose (7 columns per kinematics.tcp_frames entry) and joint_effort (the PD drive-effort estimate, or the
simulator's measured effort).
10. Never provided
What no endpoint, part or /spec field gives is PROTOCOL Appendix H.3 (object poses, scene files, seeds, the success
predicate, segmentation; physical parameters of the scene other than the robot's own drive gains are not given
either). Provided: the robot and its tool, camera calibration, world conventions, the budget, and what the task's
texts give. Whether a task's texts state world coordinates of workspace items is its benchmark's decision (its doc,
"World-frame information"). rcb_limited.check_spec refuses to start a service whose /spec names a forbidden field or
puts a position into static; ci_checks/check-standard-protocol.py checks every response of a live service by hand.
Accepted consequence: a calibrated camera aimed at the workspace tells roughly where the workspace is.
11. Errors (no physics executed)
As 2.1: 400 malformed request (including a bad kinematics request), 409 episode over (/observe and /kinematics still
answer), 413 body over 64 MB, 500 simulator error: on /submit and /observe the episode ends with reason = "error".
A negative Content-Length is a 400, answered without reading the body. The service listens on port 8000 of a network
namespace that, under Harbor's egress control, the agent's container shares: if a process of the agent holds the port
when the scene is built, the service waits for it (the agent's own time) instead of exiting.
12. Per-call bounds (2.2)
No step cap, and the wall clock stays the episode's only budget. Two bounds keep one call from holding the single-threaded service:
max_call_wall_s(/spec, default 900 s, below the client's read timeout, which is at leastmax_call_wall_s+ 120 s and at least 1800 s; task envRCB_MAX_CALL_WALL_S): a/submitor/observethat runs longer stops at the next control step and returns normally, withmeta.stopped = "max_call_wall_s"andexecuted_rowsbelow the rows sent.- Client hang-up: when the client closes the connection (its read timeout, a killed process), the call stops at the
next control step (
stopped: "client_disconnected"inepisode.jsonl); the response is not delivered. In both cases the executed prefix stays: it is journaled and checkpointed like any call, and/statusand the nextmeta.proprio.nowreport the state after it, like a real controller that loses its link. An old client sees a normal response with fewer rows executed;rcb_clientexposesResult.stoppedand prints a one-line warning.
A third bound keeps one call from exhausting the service's memory (every frame is held until the response is sent):
max_frames_per_call (/spec): the images a call may return; the rule is PROTOCOL §3 "Images per call" (shared
helpers: rcb_limited.frames_per_call, check_frames, Recorder(max_images=...)).
13. Crash recovery (the shared layer's default behaviour)
The shared layer (rcb_limited.Episode; adapters call it as toy_server.py does) resumes a crashed service in the same
episode. This is an implementation of the reference services, not a rule (PROTOCOL Appendix H.5 is the
runner's rerun of an infrastructure-failed trial; running.md).
- Journal and checkpoints. Every executed control step (the exact row given to the simulator's step) is appended to
journal.f64(float64 rows). After every completed call the journal is flushed and a checkpoint is logged inepisode.jsonl: call count, rows, control steps, the state hash, the robot's proprio row, and the adapter'spersistkeys (state that shapes later actions).call_begin/call_endbracket every call. - Restart.
run.shis the sidecar's PID 1 and exits non-zero when the service crashes; compose restarts the container (restart: on-failure:N, same container and record). The new process finds the episode'sstartevent and resumes: it replays the journal up to the last checkpoint (no rendering), truncates the journal there (the call in flight is lost as a whole: its caller got a connection error, not a result), checks the state (§13.1), restores the counters, the episode end if one was logged, and thepersistkeys, and continues the same episode. The agent's clock keeps running from its first request (the loggedfirst_request), the episode'selapsed_sfrom the first start.spec.on_service_errortells the agent to wait forGET /status(client:Robot.wait_ready()) and checkcalls/control_steps. - Limits. At most
RCB_MAX_RESUMES(2) resumes; a third crash ends the episode:sim_crashed_on_callif every crash happened while a call was being served, elsesim_crashed(infra_error = 1). A journal shorter than the log ends it asresume_failed(infra_error = 1). A verdict already on disk is never overwritten; the service then keeps its PID file for the collect hook and answers every request,GET /specand/statusincluded, with 409{"gave_up": true, "reason", "infra_error"}: the client'sRobot()andwait_ready()raiseEpisodeOverat once instead of waiting out their timeout. The same holds for a service that refuses the episode at its start. - Start-up. A crash before the scene exists is a plain restart. The startup watchdog (faulthandler's C thread: it fires
even while a native scene build holds the GIL) dumps every thread's traceback to
server.logand exits afterRCB_STARTUP_TIMEOUT_S(the shared 2.2 layer's default is 900 s; the 2.1 MuJoCo service's is 300 s; a task sets the value its build needs, §13.3); a start that follows failed starts waits min(120, 20·(n − 1)) s first. The watchdog stands down once the scene exists, so a long resume replay is not taken for a hang.
13.1 Resume classes
The resume check depends on the simulator's determinism class, which the adapter declares (Episode(..., resume=...);
each benchmark's class is in its doc):
| class | rule |
|---|---|
exact |
the replayed state hash must equal the checkpoint's; otherwise the episode ends resume_failed (infra_error = 1) |
approx |
the hash is compared and the result recorded; a mismatch does not end the episode: resume_state_mismatch = 1, and the robot's deviation from its checkpointed proprio is recorded (max |Δ| over joint_pos; the largest TCP position error from tcp_pose). Only a gross divergence, a TCP more than RCB_RESUME_MAX_DEV_M (0.05 m) from where it was, ends the episode as resume_failed (infra_error = 1) |
The standard modes are graded by the live episode the trusted service ran (PROTOCOL H.4): the verdict after a resume
is judged on the live state the resumed service continued from, so under approx a mismatch can perturb it as well as
what the agent saw; reward.json reports resumes and resume_state_mismatch for it (a trial with
resume_state_mismatch = 1 can be audited by replaying its trajectory).
13.2 Evidence fields
final.json and episode.json carry sim_startups, resumes, resume_state_mismatch (0/1), resume_max_tcp_dev_m,
resume_max_joint_dev (Episode.final_fields()), and a give-up adds gave_up, reason and infra_error. Graders copy the
numeric ones into reward.json, with infra_error.
13.3 What a 2.2 task declares
environment/docker-compose.yaml, servicesim:command: sh /srv/run.shandrestart: on-failure:N(N ≥ 3 recommended);RCB_DATA= the record dir (/dataunless the benchmark's image uses it otherwise).RCB_STARTUP_TIMEOUT_Sper task, from the measured build time of its family on the evaluation hosts (at least twice the slowest measured build; the benchmarks' docs record the measurements). A family whose build exceeds the watchdog loops through every restart to a give-up.- Optional:
RCB_MAX_RESUMES(2),RCB_MAX_CALL_WALL_S(900),RCB_RESUME_MAX_DEV_M(0.05). artifacts: alsojournal.f64,startups.jsonl,server.logfromservice = "sim".- The agent container (
environment/Dockerfile) and the verifier (tests/Dockerfile)FROMtheCLIENT_IMAGEofembodied_first_exam/standard/client.env, and anypip installinstallsclient-requirements.txt: the full pin list inline, or-r client-requirements.txtwith a byte-identical copy in the build context. A new package or version is a change to the pins. - The collect hook starts with
python3 /srv/rcb_limited.py end "$DATA": it waits through a resume (no PID file until the service is ready), signals only the process that wrote the neweststartups.jsonlline, resends SIGUSR1 every 30 s, and exits 2 whenfinal.jsonalready holds the verdict (a give-up, or an in-process grade), 0 when the service dumped the episode and exited (the hook then merges the live verdict intofinal.jsonand renders the video; it replays nothing to grade). - Adapters:
EP = L.Episode(DATA, dim, resume, persist);EP.startup()before the scene is built andEP.scene_built()after;EP.journal(a)in the control step (its True return stops the call: loops that testE["done"]testEP.stoppingtoo);S.record_start()thenEP.begin(...)instead of loggingstartand installing a SIGUSR1 handler;episode=EP, state_sha=..., proprio=...inL.Service;**EP.final_fields()infinal.json/episode.json.
13.4 Measured scene builds
In the benchmarks' docs: each records its own build times next to its RCB_STARTUP_TIMEOUT_S.
13.5 Cost and tests
One state hash per call (not per step). A resume replays the episode so far without rendering, on the agent's wall
clock (the benchmarks' docs give their replay rates). python3 embodied_first_exam/standard/selftest.py (CPU only, ~2 min) runs
toy_server.py under run.sh with a stand-in for compose's restart policy and checks: resume between and inside calls,
both classes, the third crash, the watchdog and back-off, client hang-up, the wall bound, both collect-hook outcomes,
SIGUSR1 during a resume, a simulator error, the client's protocol names, malformed requests, a port that is taken, a
negative Content-Length (CI job unit-tests).
14. Evidence and grading (the agent never sees this)
episode.jsonl (calls, checkpoints, resumes), journal.f64, startups.jsonl, calls/NNNNNN/ (meta + last frames per call),
server.log, and at the end trajectory.npz, final.json, replay.mp4. The collect hook signals the service by PID
after a collect hook has stopped the agent's processes; the service judges the live episode in-process (or dumps it with its live verdict
and exits), and the hook writes final.json, then the video (a post-hoc replay, or a live recording where PROTOCOL §4
requires one). A separate, network-less verifier transcribes final.json (PROTOCOL H.4). A GPU lost to a host
systemd reload before the episode was judged is reported as gpu_lost (infra_error).
15. Privileged-mode parity
Recommended: a benchmark's privileged Session offers the same robot information from the same code: s.camera_info(),
s.render_depth(cam), s.robot_spec(), s.kinematics(req) (s.kinematics_spec() = /spec kinematics). Privileged
mode stays privileged (object state through the tooling).
16. Per-simulator notes
In the benchmarks' docs: each describes its control rate, joint_position and ee_pose layouts,
cameras, depth source, /kinematics model, proprio columns, live success rule and video, and resume class.
17. Known issues
In the benchmarks' docs.
18. Versioning
protocol in /spec = metadata.protocol = eai-standard/2.2. The 2.x client accepts every 2.x service under both
spellings (eai-standard/2., rcb-limited/2.); 2.1 clients work against a 2.2 service (they get float16 per-frame depth
for include_depth and ignore the new parts and keys).
Part 3 — Realtime mode: eai-standard/2.3
Time does not stop. The wire protocol of the realtime mode, served by
realtime tasks only (directories ending in -realtime, protocol = "eai-standard/2.3"); the shared implementation
is embodied_first_exam/standard/realtime/rcb_realtime.py, which other benchmarks' realtime services use unchanged. 2.3
is 2.2 plus the realtime capability below; every 2.2 addition stays optional, so a 2.1 service with this capability is a
2.3 service. Same endpoints, RCB1 frames and grading as 2.1. /spec.timing declares it: realtime: true, slowdown S
(RCB_SLOWDOWN; each task declares its default in its sim service), control_dt, wall_s_per_control_step,
max_ahead_steps. Each task declares its reset budget (Part 1, "Optional reset capability"). S and the reset budget
are part of a result's configuration (running.md).
- Clock. An episode starts paused at its starting state. Its clock starts at its first
/submitor/observewithseconds > 0(after a reset, and after a crash recovery, the episode is paused again) and runs one control step everycontrol_dt× S seconds of wall time, whatever the agent does, until the episode ends. /submitanswers at once. The chunk is expanded into control steps exactly as in 2.1 (dt,linear/hold) and becomes the plan from control stepat(an integer of the episode; absent: the next step; at mostmax_ahead_stepsahead), replacing what was planned fromaton and keeping what is planned before it;linearstarts from the target in force just beforeat. Rows for steps that have run when the request arrives are dropped. When the plan runs out, the last target is held (zero-order hold).metaaddsstart_step,late_steps,plan_end_step; everymetaand/statusaddrunning,planned_steps,lag_s. The answer carries the frames and proprioception of now, no video. A service with several action types never combines rows of different types: a chunk interpolates from the target of its own type (the simulator's current one when the plan holds another type just beforeat), a type that is not interpolable (a command: one row, one control step) is never interpolated, and a gap left beforeatis held steps (the simulator's hold row, as after the plan's end) unless the chunk's type is an interpolable target in force there, which is then held through the gap. A request the plan cannot take is a 400 and changes nothing./observewithseconds = 0answers at once; withseconds > 0it waits while that much robot time runs and returns what the cameras saw (2.1's video, frames, proprio rows); it never changes the plan.cameras: []asks for no images, only proprioception.- Grading is 2.1's: the live episode, judged by the service (PROTOCOL H.4). The journal holds the sim row the
simulator applied at every control step, so the graded episode's trajectory reproduces the live run whatever the
timing (the video is rendered from it). Evidence
added:
ticks.f64(per journal row: wall time, the call whose chunk gave the row, -1 when held),calls.jsonlandsubmitted.f64(one line per call and every row as sent, instead ofcalls/NNNNNN/), and infinal.jsonrealtime,slowdown,held_steps,late_steps,max_queue_wait_s, the clock's record --clock_s(wall seconds the clock ran),max_lag_s(the furthest a control step ran behind its time; the true maximum),stalls,stalled_s(the wall time the stalls dropped: the robot stood still that long while its clock should have run),stalled_share=stalled_s/clock_s-- and what the requests cost:requests_refused(answered 4xx),request_mb(bodies read),request_busy_s(simulator-thread time spent on requests). All are checkpointed with the episode. The graders repeat the clock's record,requests_refusedandrequest_busy_sinreward.json(numbers; a missing one is 0). None of it decides a verdict. - Service. The simulator's thread runs a due control step before any waiting request; behind the clock it runs the
missed steps back to back, and more than 0.5 s behind it drops the debt (a stall, counted, its seconds added to
stalled_s) instead of fast-forwarding the robot. Checkpoints every second of robot time while the clock runs; the reference implementation resumes a crashed service in the same episode, paused. - What a request can cost the clock. A request's body is read, decoded and checked on its HTTP thread:
a request that is refused for what it says (404, 413, a malformed body, a wrong type / shape /
dt/ camera, a chunk longer thanmax_chunk_s) never reaches the simulator's thread, and neither doesGET /spec. JSON text is parsed in the service's process only up to 1 MiB; a body with more of it is parsed by a child process, one at a time, and answered as any other -- the 64 MB limit stands, and such a body costs the clock nothing whether it is accepted or refused. A negativeContent-Lengthis a 400, unread. Behind the clock, a waiting request is served once the clock has caught up (or, where the simulator cannot keep up by itself, after four times the time the last request of its kind took): requests queue behind each other, not into the clock, so/observemay take longer to answer under parallel load. A request that renders and alone takes more than 0.5 s makes the next rendering request wait four times that long, clock on time or not; requests without images (cameras: [],/status) are not held back by it. Requests can so take at most a fifth of the clock's time. The cameras are rendered (and the proprioception read) at most once per control step: parallel observes and watches share that step's frames. A watch (seconds > 0) renders inside the control steps it watches, so where one render of its cameras takes longer than its frame interval the clock runs late and stalls while it lasts -- counted installsandstalled_s. Refusals the episode's state decides (409; anattoo far ahead; too many frames) are answered by the simulator's thread, and a 409 still comes before a 400 for a request's fields. - Crashes and simulator errors. No call spans the clock (
/submitanswers at once), so what the agent's commands caused is told apart by time and by kind (PROTOCOL H.5: a normal task failure is no infrastructure failure). "After the first command" means after the first control step that ran a row the agent sent (held steps before it do not count).- A simulator error raised by a control step after the first command (
reason = "error") ends the episode as the agent's result:sim_error = 1,infra_error = 0. Before it: infrastructure. - A crash (the process dies) is resumed as in 2.1. It counts as the agent's when it happened inside a control step
after the first command and the process died on SIGSEGV, SIGBUS, SIGFPE, SIGILL or SIGABRT (the service records
the signal and the crashing stack with
faulthandlerincrash.log, a collected artifact, before it dies); a crash while the process builds the scene, replays its journal, renders or finalizes is not the agent's; pastRCB_MAX_RESUMESrestarts the trial then endssim_crashed_on_call(the agent's 0) if every crash was the agent's, elsesim_crashed(infra_error = 1). A process killed from outside (SIGKILL, SIGTERM, the out-of-memory killer: nothing recorded) and a crash before the first command are not the agent's; a failed resume isresume_failed(infra_error = 1). A reset whose new episode the simulator fails to build is infrastructure in both standard modes.
- A simulator error raised by a control step after the first command (
- Agent code runs where it does in standard mode, in the agent's own container: a controller that streams targets through the client, not code uploaded next to the simulator.
Part 4 — eai-standard/2.4: the operations capability
Reference implementation:
embodied_first_exam/standard/v2_4/, with rcb_limited.py and rcb_client.py. rcb1.py,
run.sh, the client image and the client pins are the 2.2 ones in embodied_first_exam/standard/. In the tables below, MUST
means "required of a service that reports 2.4" (what a 2.4 client may rely on).
19. Version
eai-standard/2.4 = 2.3 + the operations capability below.
- Every earlier addition stays an optional capability. That covers 2.2's depth,
frame_time_s, extrinsics,POST /kinematicsand the per-call wall bound, and 2.3's realtime capability (Part 3). So a 2.2 service that adds the operations capability is a 2.4 service. - Standard-mode tasks may declare
protocol = "eai-standard/2.4". Realtime tasks stay on 2.3. - Everything in Parts 1 and 2 holds unchanged: four endpoints (plus
/kinematics,/resetwithin a declared budget and the one below), RCB1, and grading by the live episode. /spec.protocoliseai-standard/2.4.
Why: a native simulator call can run for minutes (IPC and PBD deformables). Under 2.2, a client whose HTTP read timed out cannot tell whether the call ran, and a retry can execute the same actions twice. The capability makes every call identifiable, bounded, and queryable after the fact.
20. Operation ids and request deadlines
| Item | Level |
|---|---|
/submit and /observe accept meta.operation_id (string, [A-Za-z0-9_-]{1,80}) and meta.request_deadline_unix_s (finite number, Unix seconds); a malformed value is 400 |
MUST |
| A request whose deadline has passed before execution starts gets 408; no physics runs | MUST |
An operation_id that the service has already admitted gets 409, with operation_id, the operation record and a pointer to /operations/<id>; it is never executed again |
MUST |
The response meta echoes operation_id |
MUST |
| A request without these fields is served as in 2.2 | MUST |
The client sends a fresh id and a deadline (now + max_call_wall_s) on every call to a 2.4 service. It keeps the last id (Robot.last_operation_id), and its read timeout is at least the service budget + 120 s |
SHOULD (reference client: always) |
| After a timeout the client queries the operation instead of re-sending the actions. A new id for the same actions is a new operation | SHOULD |
21. GET /operations/<id>
Returns the operation record as JSON: 200, or 404 for an unknown id, or 400 for a malformed id. Fields:
operation_id;status:admitted|completed|error|interrupted;rows_before: control rows executed before the call;committed_rows: rows executed when the call ended;stopped: the call's stop reason, as in 2.2'smeta.stopped.
After a crash recovery, an operation that was admitted becomes interrupted, with execution_uncertain: true and
restored_committed_rows (the rows the resumed episode replayed). completed means the call ended. Rows it executed are
in committed_rows. The service keeps no copy of the response video. The endpoint is read-only, never steps physics, and
is the only GET this capability adds.
22. Call watchdog
| Item | Level |
|---|---|
A call's budget is min(max_call_wall_s, request_deadline_unix_s − now). The service checks it, and a client disconnect, at control-step boundaries |
MUST |
A native step cannot be pre-empted safely, so the service also owns a hard watchdog. A call that has not finished 60 s past max_call_wall_s ends the process; the watchdog counts from the service's own cap, never from the request deadline, so a short deadline cannot get a slow step killed. The restart then ends the episode as an infra error, call_watchdog (infra_error = 1), and does not replay into the call that hung |
MUST |
The restart ends the episode as call_watchdog only on evidence that the watchdog fired: the watchdog prints into its own file (call-watchdog.trace, emptied at every call), and its banner there is that evidence. call-watchdog.json alone, or its deadline having passed by the time the container is back, is not: any other crash during a call (a segfault, a CUDA error, an OOM kill) is handled like every crash (§13), and a resume removes both files |
MUST |
23. Physics-tick ledger and durability
| Item | Level |
|---|---|
The journal can record, for each executed row, the physics ticks it actually took (journal.ticks.i64, int64, one per row). A step that ends early inside its control period records fewer ticks, and sim_time_s and frame_time_s follow the actual physics clock |
MAY (MUST for a service whose steps can end mid-period) |
A resume replays each row with its recorded tick count. A missing or incomplete ledger ends the episode as an infra error, resume_failed. The ledger is cut to the last checkpoint together with the journal, also when that checkpoint has no rows |
MUST if the ledger is kept |
| Journal rows, the tick ledger, checkpoints and operation records are fsynced, including their directory, before the call that counts them is reported complete | MUST |
dt must be an integer multiple of the control period within float tolerance; otherwise 400 (2.1 rule, restated) |
MUST |
24. Evidence
As in 2.2 (/data/episode.jsonl, trajectory.npz, final.json, calls/), plus:
/data/operations/<id>.json;journal.ticks.i64, when kept;call-watchdog.json, while a call is running, andcall-watchdog.trace(what the watchdog printed, if it fired).
The verifier still transcribes final.json only.
python3 embodied_first_exam/standard/v2_4/selftest.py tests the watchdog's evidence, the tick ledger and the 400 for malformed
bodies on the CPU (numpy only, about 15 s).
25. Compatibility
- 2.x clients connect to a 2.4 service unchanged. They send no operation ids and get 2.2 behaviour.
- The 2.4 client sends ids only when
/spec.protocolis 2.4, so it also works against 2.1–2.3 services. - A benchmark adopts 2.4 by copying
embodied_first_exam/standard/v2_4/unchanged;ci_checks/check-tooling-identity.shchecks the copies.