Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Keep reconnecting the live screen after it has recovered (#651)

kevind5fc5ff411cf
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c8bc6066..b795ef5a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -144,6 +144,14 @@ expandable group instead of filling the conversation with screenshots.
## 0.0.15
+### The live screen keeps reconnecting after it has recovered
+
+A dropped live screen retries five times, waiting half a second, then one, two, four and eight, and
+then asks for Retry. The count of retries never went back to zero after a retry worked, so a screen
+left open through five short drops over an afternoon gave up on the sixth, although each had
+recovered within a second. The count now starts over once a reconnected screen shows a frame again,
+so only five failures in a row end in Retry.
+
### A model provider's own sign-in can stand in for an API key
`OPENBOT_MODEL_OAUTH_FILE` names a credential file holding a Google or xAI OAuth grant. Set it and
diff --git a/app/src/lib/computers/screen-connection.ts b/app/src/lib/computers/screen-connection.ts
index c0edef06..f540144f 100644
--- a/app/src/lib/computers/screen-connection.ts
+++ b/app/src/lib/computers/screen-connection.ts
@@ -83,6 +83,9 @@ export function connectScreen(options: Options) {
return;
}
}
+ // A frame means this socket recovered, so the next drop starts the backoff over. Without
+ // this, drops a viewer had already recovered from used up the retries for good.
+ if (message?.type === "frame") attempts = 0;
options.onMessage(event);
};
current.onerror = () => {
diff --git a/app/tests/screen-connection.test.ts b/app/tests/screen-connection.test.ts
index 8181f8a7..42a0a79d 100644
--- a/app/tests/screen-connection.test.ts
+++ b/app/tests/screen-connection.test.ts
@@ -66,6 +66,23 @@ test("connection loss disables input, retries boundedly, then asks for Retry", a
r.control.stop();
});
+test("a reconnect that shows the screen again restores the whole retry budget", async () => {
+ const r = rig();
+ const frame = { data: JSON.stringify({ type: "frame", data: "AA==" }) };
+ await r.sockets[0]?.onopen?.();
+ for (let i = 0; i < 6; i++) {
+ r.sockets[i]?.onmessage?.(frame);
+ r.sockets[i]?.onclose?.();
+ r.timers[i]?.run();
+ await r.sockets[i + 1]?.onopen?.();
+ }
+ expect(r.timers.map((t) => t.delay)).toEqual([500, 500, 500, 500, 500, 500]);
+ expect(r.statuses.some((status) => status.includes("Retry"))).toBe(false);
+ // A reconnect still asks who holds the computer before input is ready.
+ expect(r.statuses.at(-1)).toBe("blocked");
+ r.control.stop();
+});
+
test("reconnect reacquires ownership before input becomes ready", async () => {
const r = rig();
r.sockets[0]?.onclose?.();