Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Refuse a shell the database, show a screen that ended, and unblock a taken-over sign-in (#289)

David McKayfb0c797ed2e6
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 95f9be68..c851fcd3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,37 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.
## Unreleased
+### A Bot's shell can no longer reach the embedded database without a password
+
+In the all-in-one image the cluster was `trust`-auth on loopback, and the Bot's shell runs in the
+same container: it could `psql -h 127.0.0.1 -U openbot` with no password and read the audit trail,
+the policy store, and the credential vault as the instance owner. The cluster now uses
+`scram-sha-256` with a password generated on first init and kept beside the data, handed to the API
+over the container environment. The shell has no way to learn it, so the connection is refused. An
+external `DATABASE_URL` deployment is unaffected.
+
+### A live screen that ends says so, instead of freezing the last frame
+
+When a Bot's live screen ended — the computer stopped, or the socket failed — the message explaining
+why was drawn only by a component the take-the-wheel view does not mount, so the screen sat frozen on
+its last frame with nothing said. The reason is now shown where the live screen is.
+
+### A Bot's browser drops the automation flags a person needs gone to sign in
+
+The browser announced itself as automated (`navigator.webdriver`, the enable-automation switch),
+which sites like Google refuse even when a real person has taken the wheel. Those flags are now off
+at the source — the browser flag, not a script that patches `navigator.webdriver` and leaves the
+other tells. A headless build still reports `HeadlessChrome` in its user agent, which only running
+headed under a virtual display removes; that heavier change is tracked separately.
+
+### An empty model reply, or a run with no question, no longer ends in silence
+
+Two failures on strict OpenAI-compatible providers (z.ai GLM, Anthropic): a follow-up run that
+carried only tool deltas and no human turn was refused outright, and a reply with no text and no tool
+call ended the run with nothing on screen. A run with no human turn now carries a neutral
+continuation, and an empty reply ends on a visible line rather than in silence. OpenAI, which
+tolerated both, is unchanged.
+
### Embedded PostgreSQL initialises on a platform volume, and says so when it cannot
`EMBEDDED_POSTGRES=on` could not create its cluster on a platform whose persistent volume is an ext4
diff --git a/agent-computer/src/profiles.ts b/agent-computer/src/profiles.ts
index 4e6fe26c..c97f041c 100644
--- a/agent-computer/src/profiles.ts
+++ b/agent-computer/src/profiles.ts
@@ -96,6 +96,15 @@ const LAUNCH_ARGS = [
...(SANDBOX_ENABLED ? [] : ["--no-sandbox"]),
"--disable-dev-shm-usage",
"--password-store=basic",
+ // Drop the automation signals Chromium sets for itself, so a real person who takes the wheel can
+ // sign in to a site that refuses obvious automation (Google among them). This is the flag, not a
+ // JS patch of `navigator.webdriver`: the flag turns the property off at the source, where spoofing
+ // it from a script leaves the other tells a detector cross-checks. It does not change what the Bot
+ // may do; the governed path is unchanged. The larger tell — a headless build reporting
+ // `HeadlessChrome` in its user agent — is only removed by running headed under a virtual display,
+ // which is a heavier image change tracked separately; this reduces the signals it can reduce
+ // without one.
+ "--disable-blink-features=AutomationControlled",
];
console.info(
@@ -372,6 +381,11 @@ export function createProfiles(root: string, onClosed: BrowserClosed) {
const proxy = egressFor(botId, process.env);
const context = await chromium.launchPersistentContext(dir, {
args: LAUNCH_ARGS,
+ // Playwright launches with `--enable-automation`, which sets `navigator.webdriver` and the
+ // "controlled by automated software" banner. Dropped for the same reason as the flag above:
+ // a person who takes the wheel should be able to sign in. Named explicitly so the sandbox
+ // default args Playwright still supplies are otherwise left intact.
+ ignoreDefaultArgs: ["--enable-automation"],
// Playwright adds `--no-sandbox` on its own unless told otherwise, so leaving this out
// means the flag above decides nothing and a deployment that asked for the sandbox does
// not get one. Verified by reading the launched process arguments, not by trusting either.
diff --git a/agent-langgraph/src/history.ts b/agent-langgraph/src/history.ts
index ce16acf4..2f982528 100644
--- a/agent-langgraph/src/history.ts
+++ b/agent-langgraph/src/history.ts
@@ -103,9 +103,32 @@ export function toLangChainMessages(input: RunAgentInput): BaseMessage[] {
}
}
+ /*
+ * A run that carries no human turn is answered by OpenAI and refused by the strict providers.
+ *
+ * OpenAI tolerates a history that opens on an assistant or tool message. Anthropic and the strict
+ * OpenAI-compatible providers (z.ai GLM among them) require the first non-system message to be a
+ * human one, and will not answer a history that is only deltas: an assistant turn and its tool
+ * results with nothing a person said to respond to. A follow-up run continuing after a tool result
+ * is exactly that shape, so on those providers it came back empty and the run ended in silence.
+ *
+ * A neutral continuation turn gives them one to answer. It is appended only when the history holds
+ * no human turn at all, so a normal conversation is untouched, and OpenAI — which already answered
+ * the same history — sees no change beyond one trailing line asking it to continue.
+ */
+ const hasHumanTurn = messages.some(
+ (message): message is HumanMessage => message instanceof HumanMessage,
+ );
+ if (!hasHumanTurn) {
+ messages.push(new HumanMessage(CONTINUE_TURN));
+ }
+
return messages;
}
+/** The continuation a strict provider needs when a run carries only deltas. See toLangChainMessages. */
+const CONTINUE_TURN = "Continue from where the conversation above left off.";
+
function parseArguments(raw: string): Record<string, unknown> {
try {
const parsed = JSON.parse(raw || "{}");
diff --git a/agent-langgraph/src/index.ts b/agent-langgraph/src/index.ts
index 58a286b4..9b439a17 100644
--- a/agent-langgraph/src/index.ts
+++ b/agent-langgraph/src/index.ts
@@ -332,7 +332,9 @@ function buildGraph(input: RunAgentInput) {
return new StateGraph(MessagesAnnotation)
.addNode("answer", async (state) => ({
- messages: [await bound.invoke(state.messages)],
+ messages: [
+ withVisibleReply((await bound.invoke(state.messages)) as AIMessage),
+ ],
}))
.addNode("tools", async (state) => {
const last = state.messages.at(-1) as AIMessage;
@@ -384,6 +386,37 @@ function buildGraph(input: RunAgentInput) {
.compile();
}
+/**
+ * A reply with nothing in it ends the run in silence, so give it a line to end on.
+ *
+ * When a model returns no text and no tool call, the conditional edge sees no calls and returns END,
+ * and the person is left looking at a turn that produced no answer and no reason. Strict providers do
+ * this on a run they will not answer. Re-asking tends to get the same empty reply, so rather than
+ * loop, the run ends on a visible message saying what happened. Only a genuinely empty reply is
+ * touched: a reply with any text, or any tool call, is returned exactly as the model produced it.
+ */
+function withVisibleReply(reply: AIMessage): AIMessage {
+ const hasCall = (reply.tool_calls ?? []).length > 0;
+ if (hasCall || hasVisibleText(reply.content)) return reply;
+ return new AIMessage({ content: EMPTY_REPLY_FALLBACK });
+}
+
+function hasVisibleText(content: AIMessage["content"]): boolean {
+ if (typeof content === "string") return content.trim().length > 0;
+ if (Array.isArray(content)) {
+ return content.some((part) =>
+ typeof part === "string"
+ ? part.trim().length > 0
+ : typeof (part as { text?: unknown }).text === "string" &&
+ (part as { text: string }).text.trim().length > 0,
+ );
+ }
+ return false;
+}
+
+const EMPTY_REPLY_FALLBACK =
+ "The model returned an empty reply and the run ended without an answer. This can happen with a strict provider; try asking again.";
+
async function runAgent(input: RunAgentInput): Promise<Response> {
const encoder = new EventEncoder();
const stream = new ReadableStream<Uint8Array>({
diff --git a/app/src/components/computer/computer-view.tsx b/app/src/components/computer/computer-view.tsx
index 532c85b1..a0990604 100644
--- a/app/src/components/computer/computer-view.tsx
+++ b/app/src/components/computer/computer-view.tsx
@@ -640,11 +640,24 @@ export function ComputerView({
/>
</div>
) : showLiveScreen ? (
- <LiveScreen
- computerId={computerId}
- driving={driving}
- onProblem={setProblem}
- />
+ <div className="relative w-full" style={{ aspectRatio }}>
+ <LiveScreen
+ computerId={computerId}
+ driving={driving}
+ onProblem={setProblem}
+ />
+ {/*
+ A live screen that ends reports why through `onProblem`, and this is the
+ branch that is mounted when it does. Without drawing it here the message
+ landed in `problem`, which only the sibling `NothingToSee` reads, so the
+ screen ended with the stale last frame frozen on the canvas and nothing said.
+ */}
+ {problem ? (
+ <div className="absolute inset-0 flex items-center justify-center bg-background/85 p-4 text-center text-sm text-muted-foreground">
+ <span>{problem}</span>
+ </div>
+ ) : null}
+ </div>
) : (
<div className="relative w-full" style={{ aspectRatio }}>
<NothingToSee
diff --git a/docker/s6/scripts/postgres-init.sh b/docker/s6/scripts/postgres-init.sh
index ba9a1ffe..b40abf29 100755
--- a/docker/s6/scripts/postgres-init.sh
+++ b/docker/s6/scripts/postgres-init.sh
@@ -1,13 +1,20 @@
-#!/bin/sh
-# Create the cluster the first time, and only the first time.
+#!/command/with-contenv sh
+# Create the cluster the first time, and hand the API a password every time.
#
-# Bound to loopback and trust-auth on purpose: the only client is the process beside it, inside this
-# container, and a password would be a secret with nobody to keep it from. Publishing 5432 from this
-# container would change that, which is why nothing here does.
+# Bound to loopback, but no longer trust-auth. The process beside it is not the only client: the
+# Bot's shell runs in this same container, and under trust it could `psql -h 127.0.0.1 -U openbot`
+# with no password and reach the audit trail, the policy store, and the credential vault as the
+# instance owner. A generated password closes that: the shell has no way to learn it (its own
+# environment is an allow-list that does not carry the URL), and scram refuses a connection without
+# it. The password lives beside the data on the same volume, so it survives a restart the way the
+# data does.
set -eu
[ "${EMBEDDED_POSTGRES:-off}" = "on" ] || exit 0
DATA=/var/lib/postgresql/data
+PW_FILE=/var/lib/postgresql/pgpassword
+BIN=/usr/lib/postgresql/16/bin
+
if [ ! -s "$DATA/PG_VERSION" ]; then
# Created and owned here, as root, because this is the only step in a position to do it.
#
@@ -33,9 +40,35 @@ if [ ! -s "$DATA/PG_VERSION" ]; then
exit 1
fi
- s6-setuidgid postgres /usr/lib/postgresql/16/bin/initdb -D "$DATA" -A trust -U openbot >/dev/null
- s6-setuidgid postgres /usr/lib/postgresql/16/bin/pg_ctl -D "$DATA" -o "-c listen_addresses=127.0.0.1" -w start >/dev/null
- s6-setuidgid postgres /usr/lib/postgresql/16/bin/createdb -U openbot openbot
- s6-setuidgid postgres /usr/lib/postgresql/16/bin/psql -U openbot -d openbot -c 'CREATE EXTENSION IF NOT EXISTS vector' >/dev/null
- s6-setuidgid postgres /usr/lib/postgresql/16/bin/pg_ctl -D "$DATA" -w stop >/dev/null
+ # The cluster's password, generated once and kept on the same volume as the data. openssl is not in
+ # this image; /dev/urandom is. 600 and owned by postgres, so the Bot's shell (which runs as pwuser)
+ # cannot read the file even if it goes looking.
+ PW="$(od -An -N32 -tx1 /dev/urandom | tr -d ' \n')"
+ ( umask 077; printf '%s' "$PW" > "$PW_FILE" )
+ chown postgres:postgres "$PW_FILE"
+
+ # initdb reads the superuser password from a file (never an argument, which would show in `ps`), and
+ # sets scram for both local and TCP connections. The temp file is removed the moment initdb returns.
+ PWTMP="$(mktemp)"
+ printf '%s' "$PW" > "$PWTMP"
+ chown postgres:postgres "$PWTMP"
+ s6-setuidgid postgres "$BIN/initdb" -D "$DATA" -A scram-sha-256 -U openbot --pwfile="$PWTMP" >/dev/null
+ rm -f "$PWTMP"
+
+ s6-setuidgid postgres "$BIN/pg_ctl" -D "$DATA" -o "-c listen_addresses=127.0.0.1" -w start >/dev/null
+ # Over TCP with the password now, since the cluster no longer trusts an unauthenticated connection.
+ PGPASSWORD="$PW" s6-setuidgid postgres "$BIN/createdb" -h 127.0.0.1 -U openbot openbot
+ PGPASSWORD="$PW" s6-setuidgid postgres "$BIN/psql" -h 127.0.0.1 -U openbot -d openbot -c 'CREATE EXTENSION IF NOT EXISTS vector' >/dev/null
+ s6-setuidgid postgres "$BIN/pg_ctl" -D "$DATA" -w stop >/dev/null
+fi
+
+# Every boot, not only the first: hand the password-bearing URL to the services that connect over TCP
+# (`api` and `migrate`, both `with-contenv`). The password persists with the cluster; the container
+# environment is fresh each boot, so this has to run outside the first-init guard above. The file is
+# root-written here into s6's own environment directory, which pwuser cannot write and the Bot's shell
+# does not read.
+if [ -s "$PW_FILE" ]; then
+ PW="$(cat "$PW_FILE")"
+ printf 'postgres://openbot:%[email protected]:5432/openbot' "$PW" \
+ > /run/s6/container_environment/DATABASE_URL
fi