Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Ask for the wheel by calling for it, not by asking in words (#80)

Rahul Sethurama2cf567313af
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5b14a050..7b6c4fc7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -224,6 +224,14 @@ Sessions survive and nobody signs in again.
laptop `http://localhost` counts as one, so this never showed up in development; on a real
address it does not, and the surface did nothing at all when you pressed send. No message, no
error. Ids now come from an API with no such restriction.
+- **A Bot asked to be signed in, in words, and nothing happened.** Handing over the browser is a tool
+ call, and a sentence in the transcript is not one: "please sign in and let me know" leaves the
+ person with no wheel to take and the page where it was. Bots wrote that sentence anyway, and one
+ went further and asked for a username and password to be typed into a sign-in page nobody could
+ reach. The guidance now says that calling `computer_request_help` is what asking means, names the
+ sentences that are not it, and says the person cannot see the page at all until control is handed
+ over. Asked to file an issue on a site it was not signed in to, a Bot now offers the wheel on the
+ first attempt instead of the third.
- **A package Bot did not know it had a computer.** The instructions that make the computer usable —
snapshot before acting, and ask a person to take the wheel at a sign-in rather than reporting the
task as impossible — were imported by the two shipped Bots and by nothing else, so a built-in agent
diff --git a/app/src/lib/copilot/computer-tools.tsx b/app/src/lib/copilot/computer-tools.tsx
index ceb134c8..c4d93726 100644
--- a/app/src/lib/copilot/computer-tools.tsx
+++ b/app/src/lib/copilot/computer-tools.tsx
@@ -556,7 +556,10 @@ export function ComputerTools() {
"Ask the person to take control of your computer and do something you cannot: sign in, enter a " +
"password or a one-time code, or clear a CAPTCHA. Say specifically what you need done. They " +
"will drive the browser themselves and hand it back, and you carry on in the same session. " +
- "Use this INSTEAD of giving up, and instead of ever asking them to type a password to you.",
+ "Use this INSTEAD of giving up, and instead of ever asking them to type a password to you. " +
+ "This call is the only thing that reaches them: until you make it they are not looking at the " +
+ "page and have no way to help, so saying you need them to sign in, or asking whether they would " +
+ "like to proceed, hands over nothing and leaves the page where it is.",
parameters: z.object({
reason: z
.string()
diff --git a/shared/bot-prompt.ts b/shared/bot-prompt.ts
index ae292a58..63897d0d 100644
--- a/shared/bot-prompt.ts
+++ b/shared/bot-prompt.ts
@@ -45,7 +45,13 @@ const COMPUTER_GUIDANCE_LINES = [
"Some pages need a person: a sign-in, a password, a code sent to their phone, a CAPTCHA.",
"When you hit one, call computer_request_help and say exactly what you need done. The person takes",
"control of your browser, does that part, and hands it back, and you continue in the same session.",
- "NEVER ask the person to tell you a password or a code. You do not need it and must not have it.",
+ "Calling it IS how you ask. Words in your answer are not: nobody is offered the wheel by a sentence,",
+ "and the page stays exactly where it is. So NEVER write 'please sign in and let me know', 'would you",
+ "like to proceed', or 'once you have signed in, tell me' instead of calling it. If you are about to",
+ "say the task needs somebody signed in, that sentence is the tool call: make it.",
+ "The person is not looking at this page and cannot type into it until you hand it over. NEVER ask",
+ "them to enter a username, a password or a code, neither into this conversation nor 'on the sign-in",
+ "page': you do not need it, must not have it, and they cannot reach the page anyway.",
"When you need ONE value only, a password, a code, a card number, do not hand over the whole",
"browser. Click the field first, then call computer_request_secret with that field's ref and a short",
"label. They type it into a masked box that goes straight to the page and you never see it.",