Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Let a Bot visualize with Open Generative UI, not describe markup (#309)

Jerel Velarde57f5a77d7f93
diff --git a/.env.example b/.env.example
index cb3c4ac6..f46d803a 100644
--- a/.env.example
+++ b/.env.example
@@ -19,6 +19,23 @@ KEY_ENCRYPTION_KEY=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
PORT=3001
SERVER_PORT=3001
TENANT_PACKAGE_DIR=../examples/fintech
+# Let a Bot answer with an interface it wrote itself: markup, styles and a script it generates for
+# that one answer, streamed into the transcript and rendered in a sandboxed iframe with no
+# same-origin access to this app. Off unless you set this to `true` or `1`.
+#
+# Asked for rather than inherited, unlike most of the switches in this file. It decides whether a
+# model may put code it wrote on somebody's screen, so a deployment should choose it rather than
+# acquire it by upgrading — including a deployment that builds its default branch automatically.
+#
+# This is not the component catalogue. A component is something this deployment holds and an
+# administrator grants per Bot; this has nothing to grant, because the Bot writes it on the spot and
+# it is gone when the conversation moves on. It is a deployment switch rather than a per-Bot grant
+# because the SDK offers no seam for one — see DeploymentConfig.generativeUi in server/src/config.ts.
+#
+# What the interface can reach is what the sandbox hands it, and this deployment hands it nothing: no
+# session, no same-origin access, and no route into your data. It can load libraries from a CDN, so a
+# deployment that must not reach the public internet from a browser tab should leave this off.
+# OPENBOT_GENERATIVE_UI=true
# What this deployment calls itself, when more than one shares an Intelligence project. A copy of a
# deployment made for development uses the same project key, and threads are listed per Bot with
# nothing to say which deployment a conversation came from. The name goes into every thread id this
diff --git a/app/src/components/channels/chat-messages.ts b/app/src/components/channels/chat-messages.ts
index aba4af7e..7dce9acf 100644
--- a/app/src/components/channels/chat-messages.ts
+++ b/app/src/components/channels/chat-messages.ts
@@ -1,4 +1,4 @@
-import type { Message, ToolCall } from "@ag-ui/core";
+import type { ActivityMessage, Message, ToolCall } from "@ag-ui/core";
/**
* Transcript projection that pairs assistant tool calls with later tool-result messages.
@@ -12,7 +12,25 @@ export type VisibleChatItem =
toolCall: ToolCall;
/** The result, once there is one. Absent means the call is still in flight. */
result?: string;
- };
+ }
+ /**
+ * Something a Bot is drawing rather than saying.
+ *
+ * Carried whole rather than projected into fields of our own, because what is inside an activity
+ * belongs to whoever renders it: an interface a Bot generated arrives here as partial HTML that
+ * grows on every chunk, and the renderer that paints it is the one that knows what a half-finished
+ * one looks like. Reshaping it on the way past would mean this file had to understand every
+ * activity type anybody registers.
+ */
+ | { kind: "activity"; id: string; message: ActivityMessage };
+
+/**
+ * The SDK's own tool for drawing an interface, whose output is an activity rather than a result.
+ *
+ * Named here rather than imported because the SDK exports the renderer and the argument schema but
+ * not the tool name; it is the string the runtime middleware matches on to emit the activity.
+ */
+const GENERATE_SANDBOXED_UI = "generateSandboxedUi";
/** A tool result, as it arrives, its own message, pointing back at the call it answers. */
type ToolResultMessage = { role: "tool"; toolCallId: string; content?: string };
@@ -44,6 +62,15 @@ export function toVisibleChatItems(
});
}
for (const toolCall of message.toolCalls ?? []) {
+ /*
+ * The call that draws an interface is not a row of its own; the interface is.
+ *
+ * Its renderer shows the waiting message and then returns nothing, so once the interface has
+ * arrived this leaves an empty item behind — invisible in itself, but still a child of a
+ * `gap-6` column, so every generated interface gained a stray gap under it. The activity
+ * beside it already shows its own progress while it is being written.
+ */
+ if (toolCall.function.name === GENERATE_SANDBOXED_UI) continue;
items.push({
kind: "tool",
// One assistant message can carry multiple tool calls.
@@ -57,6 +84,18 @@ export function toVisibleChatItems(
return items;
}
+ /*
+ * Activities are their own messages, in order, beside the prose.
+ *
+ * Kept rather than dropped, which is what this projection used to do with every role it did not
+ * name. A Bot that draws its own interface says nothing in `content` and calls no tool the
+ * transcript can pair a result with — the whole answer is the activity. Falling through to the
+ * bail below meant the turn rendered as silence.
+ */
+ if (message.role === "activity") {
+ return [{ kind: "activity", id: message.id, message }];
+ }
+
if (message.role !== "user") return [];
const text =
diff --git a/app/src/components/channels/chat-transcript.tsx b/app/src/components/channels/chat-transcript.tsx
index 548bc600..2a6a1fdd 100644
--- a/app/src/components/channels/chat-transcript.tsx
+++ b/app/src/components/channels/chat-transcript.tsx
@@ -1,5 +1,8 @@
-import type { Message } from "@ag-ui/core";
-import { useRenderToolCall } from "@copilotkit/react-core/v2";
+import type { ActivityMessage, Message } from "@ag-ui/core";
+import {
+ useRenderActivityMessage,
+ useRenderToolCall,
+} from "@copilotkit/react-core/v2";
import { IconBox } from "@tabler/icons-react";
import { motion, useReducedMotion } from "motion/react";
import { memo, useEffect, useMemo, useRef } from "react";
@@ -464,6 +467,52 @@ const TranscriptMessage = memo(function TranscriptMessage({
);
});
+/**
+ * What a failed activity is called on screen.
+ *
+ * An `activityType` is a protocol name and reads like one, so the boundary's sentence gets a phrase
+ * a person can read instead. An unknown type falls back to its own name rather than to something
+ * vague: whoever registered it will recognise it, and nobody else can act on either wording.
+ */
+function activityName(activityType: string): string {
+ return activityType === "open-generative-ui"
+ ? "This Bot's generated interface"
+ : activityType;
+}
+
+/**
+ * One activity, drawn by whichever renderer claims its type.
+ *
+ * The renderer comes from the SDK's registry, so an activity nobody registered draws nothing and
+ * this returns null rather than an empty row — unlike a tool call, which always has a line to fall
+ * back to because a call that happened is worth reporting even undrawn. An activity is the drawing;
+ * with no renderer there is nothing to say about it.
+ *
+ * The memo boundary earns its place differently here than on a tool call. It cannot spare this
+ * component its own churn — a generated interface streams, and `content` is a new object on every
+ * chunk, which is exactly when it must re-render — but it does stop the sentence being typed after it
+ * from re-rendering a mounted iframe.
+ */
+const TranscriptActivity = memo(function TranscriptActivity({
+ delay,
+ message,
+}: {
+ delay: number;
+ message: ActivityMessage;
+}) {
+ const { renderActivityMessage } = useRenderActivityMessage();
+ const drawn = renderActivityMessage(message);
+ if (!drawn) return null;
+
+ return (
+ <Arriving delay={delay}>
+ <ToolRenderBoundary name={activityName(message.activityType)}>
+ {drawn}
+ </ToolRenderBoundary>
+ </Arriving>
+ );
+});
+
/**
* One drawn tool call, memoised on the same terms.
*
@@ -653,6 +702,13 @@ export function ChatTranscript({
toolCallId={item.toolCall.id}
/>
</MessageScrollerItem>
+ ) : item.kind === "activity" ? (
+ <MessageScrollerItem key={item.id} messageId={item.id}>
+ <TranscriptActivity
+ delay={delays.delayFor(item.id, index, items.length)}
+ message={item.message}
+ />
+ </MessageScrollerItem>
) : (
<MessageScrollerItem
key={item.id}
diff --git a/app/src/lib/copilot/generative-ui.ts b/app/src/lib/copilot/generative-ui.ts
new file mode 100644
index 00000000..cf913fba
--- /dev/null
+++ b/app/src/lib/copilot/generative-ui.ts
@@ -0,0 +1,50 @@
+/**
+ * What a Bot is told about drawing an interface it wrote itself.
+ *
+ * The SDK ships a default set of guidelines, and they describe shadcn/ui: rounded cards, a violet
+ * accent, its own spacing scale. OpenBot does not look like that. The app's palette is deliberately
+ * without colour — every token in styles.css sits at chroma zero except the destructive red and the
+ * success teal — so the default guidance produces something that reads as a foreign widget dropped
+ * into the transcript rather than part of it.
+ *
+ * This is a prompt, so it is written for a model rather than for a person: concrete values it can
+ * copy, and the few rules that are actually load-bearing.
+ *
+ * WHY THE COLOURS ARE LITERAL. The generated interface renders inside a sandboxed iframe with no
+ * same-origin access to this app. It cannot reach the stylesheet, the theme class, or the CSS custom
+ * properties the rest of the UI is built from, so anything it should match has to be written out
+ * here in full. Referring it to `--muted-foreground` would produce an unstyled document.
+ *
+ * WHY prefers-color-scheme AND NOT THIS APP'S THEME. For the same reason: the iframe is a separate
+ * document and cannot see which theme the person picked. `prefers-color-scheme` is the only signal
+ * that crosses, so a generated interface follows the browser rather than the app's own switch. A
+ * person who has overridden their OS theme in OpenBot will see a generated interface that disagrees
+ * with the surface around it. That is a known limitation of the sandbox rather than something this
+ * text can fix.
+ */
+export const GENERATIVE_UI_DESIGN_SKILL = `You are generating a self-contained interface that renders inside a sandboxed iframe in OpenBot's chat transcript. It must look like it belongs to OpenBot, not like a widget from somewhere else.
+
+PALETTE. OpenBot is neutral by design. Use greys for structure and reserve colour for meaning.
+- Light: background #fafafa, surface #ffffff, text #0a0a0a, muted text #636363, border #e5e5e5.
+- Dark: background #0a0a0a, surface #171717, text #fafafa, muted text #a1a1a1, border rgba(255,255,255,0.10).
+- Only two accents, and only when they carry meaning: #e7000b destructive and #009689 success in light, #ff6467 and #00bba7 in dark.
+- Never introduce a brand hue, gradient, or coloured header. A purple or blue accent is wrong here.
+
+CHARTS. Series colours are steps of grey, not a rainbow: #d4d4d4, #737373, #525252, #404040, #262626. Distinguish series by ordering, direct labels, and shape rather than by hue. If a series means "bad", the destructive red is allowed for that one series.
+
+TYPE. font-family: Inter, ui-sans-serif, system-ui, sans-serif. Body 14px/1.5. Headings 15-16px, weight 600, no letter-spacing tricks. Numerals in tables and metrics: font-variant-numeric: tabular-nums.
+
+SHAPE AND SPACING. border-radius: 0.55rem on cards and controls, 0.375rem on small chips. 1px solid borders, never a drop shadow for elevation. Pad containers 12-16px. Space stacked blocks 8-12px.
+
+DARK MODE IS REQUIRED. Define the light palette first, then override inside @media (prefers-color-scheme: dark). Set an explicit background and colour on body — the iframe paints on nothing, so a transparent body shows through wrongly.
+
+LAYOUT. Assume a narrow column: roughly 320-680px wide, inside a chat message. Design for the narrow case first and let it grow. Never set a fixed pixel width on the outermost element; use max-width: 100%, flexbox or grid, and box-sizing: border-box everywhere. Anything wide — a table, a chart, a code block — scrolls inside its own container with overflow-x: auto. The page itself must never scroll sideways.
+
+HONESTY ABOUT DATA. You have no access to this deployment's data. Every number you render is one you were given or one you made up, so never present an invented figure as a reading from OpenBot. If you are illustrating rather than reporting, label it as an example on the interface itself.
+
+MECHANICS.
+- Keep it self-contained: inline the CSS and the JS. CDN <script> and <link> tags do load, so Chart.js, D3 and similar are available when a chart genuinely needs them; prefer plain SVG or CSS for anything simple.
+- No network calls to this deployment. The iframe has no session and no same-origin access; a fetch to /api will fail.
+- Guard every read of browser storage in try/catch. It throws outright in some contexts.
+- Interactive controls need a visible focus ring, real button elements, and hit targets of at least 32px.
+- Prefer one clear interface over a dashboard of panels. A single legible chart beats four cramped ones.`;
diff --git a/app/src/lib/copilot/provider.tsx b/app/src/lib/copilot/provider.tsx
index 14e0b196..46001b60 100644
--- a/app/src/lib/copilot/provider.tsx
+++ b/app/src/lib/copilot/provider.tsx
@@ -1,9 +1,12 @@
import { CopilotKitProvider } from "@copilotkit/react-core/v2";
+import { useQuery } from "@tanstack/react-query";
import type { ReactNode } from "react";
+import { deploymentCapabilitiesQueryOptions } from "@/lib/deployment/queries";
import { ActiveBotProvider } from "./active-bot";
import { ComputerTools } from "./computer-tools";
import { EscalationTool } from "./escalation-tool";
import { GalleryTools } from "./gallery-tools";
+import { GENERATIVE_UI_DESIGN_SKILL } from "./generative-ui";
import { HandoffTool } from "./handoff-tool";
import { SandboxedTools } from "./sandboxed-tools";
@@ -22,8 +25,30 @@ import { SandboxedTools } from "./sandboxed-tools";
* runtime rather than returning it).
*/
export function CopilotProvider({ children }: { children: ReactNode }) {
+ const { data: capabilities } = useQuery(deploymentCapabilitiesQueryOptions());
+
return (
- <CopilotKitProvider runtimeUrl="/api/copilotkit" credentials="include">
+ <CopilotKitProvider
+ runtimeUrl="/api/copilotkit"
+ credentials="include"
+ /*
+ * Passed only when this deployment actually has the capability, and this is the load-bearing
+ * part rather than a tidiness. The SDK reads generative UI as on when EITHER the runtime says
+ * so OR this prop is present at all, so passing it unconditionally would switch the browser
+ * half on in a deployment that had switched the server half off. The Bot would then be offered
+ * the tool, generate a whole interface, and nothing would draw it, because the events that
+ * paint one come from the runtime middleware this deployment declined to run.
+ *
+ * Absent, the SDK asks the runtime and believes the answer, which is the behaviour we want
+ * while this query is still in flight.
+ *
+ * The object carries guidance only. It does not turn anything on that the server has not
+ * already turned on; it replaces the SDK's shadcn-flavoured house style with OpenBot's.
+ */
+ {...(capabilities?.generativeUi
+ ? { openGenerativeUI: { designSkill: GENERATIVE_UI_DESIGN_SKILL } }
+ : {})}
+ >
{/* Computer tools target the Bot declared by the mounted surface. */}
<ActiveBotProvider>
<ComputerTools />
diff --git a/app/src/lib/deployment/queries.ts b/app/src/lib/deployment/queries.ts
new file mode 100644
index 00000000..cee31dc1
--- /dev/null
+++ b/app/src/lib/deployment/queries.ts
@@ -0,0 +1,59 @@
+import { queryOptions } from "@tanstack/react-query";
+import { client } from "@/lib/client";
+
+/**
+ * What this deployment can do, as the server is willing to say it.
+ *
+ * A projection, not the runtime. `/api/capabilities` is reachable before anybody has signed in, so
+ * only fields somebody may know unauthenticated appear on it; the Intelligence contract and every
+ * deployment secret stay on the server. See server/src/app.ts.
+ */
+export type DeploymentCapabilities = {
+ /**
+ * Whether a Bot may answer with an interface it wrote itself.
+ *
+ * Read by the browser because the browser owns half of this capability: the SDK's provider is what
+ * offers the model the tool that generates one. A deployment that turned the server half off while
+ * the browser kept offering the tool would have Bots writing whole interfaces that nothing draws,
+ * so both halves read this one answer.
+ */
+ generativeUi: boolean;
+};
+
+export const deploymentKeys = {
+ all: ["deployment"] as const,
+ capabilities: () => ["deployment", "capabilities"] as const,
+};
+
+/**
+ * What this deployment can do.
+ *
+ * From the server rather than from the build, like the sign-in options beside it: the container image
+ * is built once and knows nothing about the deployment that will run it, so a capability compiled
+ * into the bundle can only ever describe the build machine.
+ *
+ * Absent fields read as off. A server too old to answer, or one that failed to, is a server this app
+ * should not assume a capability of — and for generated interfaces the fail-closed direction is the
+ * safe one, because claiming it wrongly is what makes a Bot generate something nothing renders.
+ */
+export function deploymentCapabilitiesQueryOptions() {
+ return queryOptions({
+ queryKey: deploymentKeys.capabilities(),
+ // Configuration, not data. It cannot change without the process restarting.
+ staleTime: Number.POSITIVE_INFINITY,
+ queryFn: async (): Promise<DeploymentCapabilities> => {
+ /*
+ * The whole body, then one field off it. `/api/capabilities` answers with a bare object rather
+ * than the envelope most endpoints use, which is why this reads the Response itself instead of
+ * naming a key — the same shape the sign-in query uses.
+ */
+ const body = (await (
+ await client("/api/capabilities", {
+ fallback: "This deployment's capabilities could not be loaded.",
+ })
+ ).json()) as { generativeUi?: boolean };
+
+ return { generativeUi: body.generativeUi === true };
+ },
+ });
+}
diff --git a/app/src/styles.css b/app/src/styles.css
index 66d7cb19..2326b7d0 100644
--- a/app/src/styles.css
+++ b/app/src/styles.css
@@ -255,3 +255,37 @@ body {
[data-slot="item-description"] {
text-wrap: pretty;
}
+
+/*
+ * A generated interface is never allowed to occupy nothing.
+ *
+ * The SDK sizes the frame it draws into from a single measurement taken inside the sandbox, and it
+ * writes the result as an inline height on the frame's own container. Two races in that code make
+ * the number wrong, and both were reproduced against @copilotkit/react-core 1.69.0:
+ *
+ * - The measurement is queued alongside the expressions that build the interface. When it runs
+ * first the body it measures is still empty, the container is set to `height: 0px`, and because
+ * the container also clips, the interface is invisible. Nothing retries: the measurement is
+ * one-shot and its listener is removed after the first reply.
+ * - The effect that measures bails out when the sandbox has not finished loading, and its only
+ * dependency is "generation finished". An interface restored from thread history is already
+ * finished on its first render, while the sandbox is still being imported, so the effect bails
+ * and never runs again.
+ *
+ * A floor is the smallest thing that removes the failure that matters. `min-height` clamps above an
+ * inline `height`, so a collapsed measurement can no longer render nothing, and a measurement that
+ * came back correct and larger is left alone. The reader gets an interface that is present and may
+ * be cut off, instead of a turn that looks like the Bot said nothing at all.
+ *
+ * Selected through the frame rather than through a class of ours because both chat surfaces render
+ * this container and only one of them is ours: the packaged chat builds it too, and a rule that
+ * needed our wrapper would fix the transcript and leave the other surface broken. `websandbox__frame`
+ * is the class the sandbox library puts on the frame it creates.
+ *
+ * DELETE THIS WHEN THE SDK IS FIXED. It compensates for someone else's arithmetic, so it should not
+ * outlive the bug: once the measurement re-runs on sandbox-ready and after the queue drains, this
+ * rule can only make small interfaces taller than they asked to be.
+ */
+:where(div:has(> iframe.websandbox__frame)) {
+ min-height: 200px;
+}
diff --git a/app/tests/chat-messages.test.ts b/app/tests/chat-messages.test.ts
new file mode 100644
index 00000000..b637c3ec
--- /dev/null
+++ b/app/tests/chat-messages.test.ts
@@ -0,0 +1,157 @@
+import type { Message } from "@ag-ui/core";
+import { describe, expect, test } from "bun:test";
+import { toVisibleChatItems } from "../src/components/channels/chat-messages";
+
+/**
+ * What a channel transcript shows, out of the messages a run produced.
+ *
+ * The projection used to name the roles it understood and drop everything else, which was correct
+ * while every answer was prose or a tool call. It stopped being correct once a Bot could answer by
+ * drawing: a generated interface arrives as an activity message, says nothing in `content`, and
+ * pairs with no tool result — so the turn rendered as silence. These cases hold that shut.
+ */
+
+const PROSE: Message = {
+ id: "assistant-1",
+ role: "assistant",
+ content: "Here is how those issues group.",
+};
+
+/** A generated interface, mid-stream: the HTML grows on every chunk and `generating` is still true. */
+const DRAWING: Message = {
+ id: "activity-1",
+ role: "activity",
+ activityType: "open-generative-ui",
+ content: {
+ css: ".card { color: #0a0a0a }",
+ cssComplete: true,
+ html: ['<div class="card">'],
+ htmlComplete: false,
+ generating: true,
+ },
+};
+
+describe("toVisibleChatItems", () => {
+ test("keeps an activity, carrying the message whole", () => {
+ expect(toVisibleChatItems([DRAWING])).toEqual([
+ { kind: "activity", id: "activity-1", message: DRAWING },
+ ]);
+ });
+
+ /*
+ * The regression this projection had. A Bot that answers only by drawing produces exactly this
+ * one message, so dropping it left a turn that had plainly happened showing nothing at all.
+ */
+ test("does not render a drawing-only turn as silence", () => {
+ expect(toVisibleChatItems([DRAWING])).not.toEqual([]);
+ });
+
+ test("keeps an activity in its place beside the prose", () => {
+ expect(
+ toVisibleChatItems([PROSE, DRAWING]).map((item) => item.kind),
+ ).toEqual(["text", "activity"]);
+ });
+
+ /*
+ * An activity is a message in its own right, not something folded into the assistant turn that
+ * preceded it: it renders as its own row, and both survive.
+ */
+ test("draws prose and an activity as two items", () => {
+ const items = toVisibleChatItems([PROSE, DRAWING]);
+
+ expect(items).toHaveLength(2);
+ expect(items[0]).toEqual({
+ kind: "text",
+ id: "assistant-1",
+ role: "assistant",
+ text: "Here is how those issues group.",
+ });
+ });
+
+ // The roles this file already understood, so the addition above is not paid for elsewhere.
+ test("still pairs a tool call with the result that answers it", () => {
+ const called: Message = {
+ id: "assistant-2",
+ role: "assistant",
+ content: "",
+ toolCalls: [
+ {
+ id: "call-1",
+ type: "function",
+ function: { name: "botActivity", arguments: '{"days":7}' },
+ },
+ ],
+ };
+ const answered: Message = {
+ id: "result-1",
+ role: "tool",
+ toolCallId: "call-1",
+ content: "42",
+ };
+
+ expect(toVisibleChatItems([called, answered])).toEqual([
+ {
+ kind: "tool",
+ id: "call-1",
+ toolCall: called.toolCalls?.[0],
+ result: "42",
+ },
+ ]);
+ });
+
+ /*
+ * The call that produces a generated interface is not a row of its own.
+ *
+ * Its renderer shows the waiting message and then returns nothing, so keeping the item left an
+ * empty child in a `gap-6` column and every generated interface gained a stray gap beneath it.
+ */
+ test("drops the call that draws an interface, and keeps the interface", () => {
+ const drawing: Message = {
+ id: "assistant-3",
+ role: "assistant",
+ content: "",
+ toolCalls: [
+ {
+ id: "call-2",
+ type: "function",
+ function: { name: "generateSandboxedUi", arguments: "{}" },
+ },
+ ],
+ };
+
+ expect(toVisibleChatItems([drawing, DRAWING])).toEqual([
+ { kind: "activity", id: "activity-1", message: DRAWING },
+ ]);
+ });
+
+ // Every other tool still gets its row: only the one whose output is the activity is dropped.
+ test("keeps a call from any other tool", () => {
+ const other: Message = {
+ id: "assistant-4",
+ role: "assistant",
+ content: "",
+ toolCalls: [
+ {
+ id: "call-3",
+ type: "function",
+ function: { name: "botActivity", arguments: "{}" },
+ },
+ ],
+ };
+
+ expect(toVisibleChatItems([other]).map((item) => item.kind)).toEqual([
+ "tool",
+ ]);
+ });
+
+ // Roles the transcript has nothing to draw for are still dropped rather than rendered empty.
+ test("drops a role it has nothing to show", () => {
+ const thinking: Message = {
+ id: "reasoning-1",
+ role: "reasoning",
+ content: "considering the grouping",
+ };
+
+ expect(toVisibleChatItems([thinking])).toEqual([]);
+ });
+});
diff --git a/docs/configuration.md b/docs/configuration.md
index e9ec67d1..738d1a1d 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -56,6 +56,36 @@ at `agent-langgraph` on a laptop.
| `APP_DIST_DIR` | unset | Where the built app is, when this process serves it. Set inside the container image; unset in development, where Vite serves the app. |
| `AUDIT_RETENTION_DAYS` | unset | Whole number of days to keep audit rows; older ones are removed. Unset keeps the trail forever. |
| `WORKER_SHARED_SECRET` | unset; `start.sh` uses a fixed local default | The secret the routines worker presents to fire a due routine. Without it the server refuses every handoff, whether or not a worker exists to send one. |
+| `OPENBOT_GENERATIVE_UI` | unset (capability off) | `true` or `1` lets a Bot answer with an interface it wrote itself. |
+
+**`OPENBOT_GENERATIVE_UI`** turns on generated interfaces. Set it, and a Bot may answer by writing
+the markup, styles and script for an interface and streaming it into the transcript, where it renders
+in a sandboxed iframe. Left unset, a Bot answers in prose and with the components this deployment
+holds, as before.
+
+It is asked for rather than inherited, which is deliberate and unlike most switches here. This one
+decides whether a model may put code it wrote on somebody's screen and load libraries from a CDN to
+run it, so a deployment should choose it rather than acquire it by upgrading — including a deployment
+that builds its default branch automatically. Only `true` or `1` count as yes; anything else leaves it
+off.
+
+This is not the component catalogue. A component is something the deployment holds — compiled into
+the build or authored in the playground — and an administrator grants it per Bot. A generated
+interface has nothing to grant: it does not exist until the Bot writes it, and it is gone when the
+conversation moves on. That is also why this is one switch for the deployment rather than a grant per
+Bot. The interface is painted from activity events that only the runtime middleware emits, and the
+tool the model calls is registered by the browser for every Bot the moment that middleware runs, so
+enabling it for some Bots would leave the rest able to call the tool and draw nothing.
+
+The switch reaches both halves. The server passes `openGenerativeUI` to the runtime, and
+`/api/capabilities` reports the capability so the app offers the tool. The halves disagreeing is the
+one configuration worth avoiding: runtime-only means the tool is never offered, and browser-only
+means a Bot generates a whole interface that nothing renders.
+
+What a generated interface can reach is what the sandbox hands it, and this deployment hands it
+nothing — no session, no same-origin access to the app, no route into your data. It can load
+libraries from a CDN, which is the reason a deployment that must not reach the public internet from a
+browser tab should leave this unset.
**`AGENT_STALL_TIMEOUT_MS`** watches for the failure a Bot has that nothing else in the trail can
show: a stream that stops producing anything. Every other audit row is something that happened, and
diff --git a/server/src/app.ts b/server/src/app.ts
index 36617003..65bfd5ff 100644
--- a/server/src/app.ts
+++ b/server/src/app.ts
@@ -213,6 +213,16 @@ export function createApp(
context.json({
mode: config.runtime.mode,
durableHistory: config.runtime.durableHistory,
+ /*
+ * Whether a Bot may answer with an interface it wrote itself.
+ *
+ * Projected because the browser holds half of this capability. The runtime middleware turns a
+ * generated interface into the events that paint it, and the SDK's provider registers the tool
+ * that produces one; a deployment that switched the runtime half off while the browser went on
+ * offering the tool would have Bots writing interfaces nothing ever draws. One flag, read by
+ * both halves, so off means off.
+ */
+ generativeUi: config.generativeUi,
/*
* Which identity providers this deployment can sign somebody in with.
*
diff --git a/server/src/config.ts b/server/src/config.ts
index dbe95e5d..0bcfc9d2 100644
--- a/server/src/config.ts
+++ b/server/src/config.ts
@@ -221,6 +221,31 @@ export type DeploymentConfig = {
singleUser: boolean;
/** Names OpenBot on the analytics the runtime already sends. Off with OPENBOT_ACCESSIBILITY_DISABLED. */
accessibility: boolean;
+ /**
+ * Whether a Bot may answer with an interface it wrote itself.
+ *
+ * This is not the component catalogue. A component is something this deployment holds: it was
+ * either compiled into the build or authored in the playground, an administrator granted it to a
+ * Bot, and all a Bot decides is which of them to draw. Here there is nothing to grant, because
+ * there is nothing yet — the Bot writes the markup, the styles and the script for this one answer,
+ * and they are gone when the conversation moves on.
+ *
+ * A deployment switch rather than a per-Bot grant because the SDK offers no seam for one. The
+ * interface is painted from activity events that only the runtime middleware emits, and the tool
+ * the model calls is registered by the browser for every Bot the moment that middleware is on.
+ * Narrowing the middleware to some Bots would leave the rest able to call the tool and draw
+ * nothing at all, which is a worse answer than never offering it.
+ *
+ * Off until a deployment sets OPENBOT_GENERATIVE_UI. A capability that runs code a model wrote is
+ * one an operator should choose, not one they should discover after an upgrade — and a deployment
+ * that builds its default branch automatically would otherwise acquire it without a decision.
+ *
+ * What it runs is sandboxed by the SDK, in an iframe with no same-origin access to this app, so a
+ * generated interface reaches this deployment's data only through what the host hands it. This
+ * deployment hands it nothing. It can load libraries from a CDN, which is the part a deployment
+ * that must not reach the public internet from a browser tab needs to weigh.
+ */
+ generativeUi: boolean;
/**
* Where the built app is, when this process serves it.
*
@@ -770,6 +795,30 @@ function accessibilityEnabled(environment: Environment): boolean {
return off !== "true" && off !== "1";
}
+/**
+ * Whether a Bot may draw an interface it wrote itself.
+ *
+ * ASKED FOR, NOT INHERITED, which is the one place this deliberately breaks the symmetry with
+ * OPENBOT_ACCESSIBILITY_DISABLED above it. That flag names a deployment out of an analytics label,
+ * so defaulting it on costs a fork nothing it would mind. This one decides whether a model may put
+ * code it wrote on somebody's screen and pull libraries from a CDN to run it. Written as a disable
+ * switch, absence would be the permissive answer, and a deployment acquires the capability by
+ * upgrading rather than by choosing it — which is exactly how a deployment that auto-deploys its
+ * default branch would find out.
+ *
+ * Only "true" or "1" turn it on. Anything else is not a way of saying yes, and a value nobody
+ * intended should leave a capability off rather than on.
+ *
+ * The answer has to reach the browser as well as the runtime, which is why it ends up on
+ * /api/capabilities rather than staying server-side. Enabling only the runtime half would leave the
+ * browser never offering the tool; enabling only the browser half would have a Bot generate a whole
+ * interface that nothing renders. See DeploymentConfig.generativeUi.
+ */
+function generativeUiEnabled(environment: Environment): boolean {
+ const on = optional(environment, "OPENBOT_GENERATIVE_UI");
+ return on === "true" || on === "1";
+}
+
/**
* How long the audit trail is kept.
*
@@ -840,6 +889,7 @@ export function loadConfig(
configuredAuthProviders(auth).length > 0,
),
accessibility: accessibilityEnabled(environment),
+ generativeUi: generativeUiEnabled(environment),
...(optional(environment, "APP_DIST_DIR")
? { appDistDir: optional(environment, "APP_DIST_DIR") as string }
: {}),
diff --git a/server/src/copilot.ts b/server/src/copilot.ts
index 8e2f9c2f..547e329a 100644
--- a/server/src/copilot.ts
+++ b/server/src/copilot.ts
@@ -1095,6 +1095,23 @@ export function mountCopilotRuntime(
...(config.accessibility
? { telemetryProperties: { accessibility_title: "OpenBot" } }
: {}),
+ /*
+ * What lets a Bot answer with an interface it wrote itself.
+ *
+ * This one flag is the whole difference between a Bot that draws and a Bot that describes
+ * markup it cannot show. The middleware it turns on does not give the model the tool — the
+ * browser does that — it reads the arguments of the `generateSandboxedUi` call as they stream
+ * and re-emits them as `open-generative-ui` activity events. Those events are the only thing
+ * that paints: the tool's own renderer shows the waiting message and then returns nothing. So a
+ * deployment with the browser half and not this one has Bots generating whole interfaces that
+ * never appear, which is the shape this capability arrived in.
+ *
+ * `true` rather than a list of Bots. The list narrows only the event transform, and the tool
+ * stays offered to every Bot regardless, so naming some Bots here would leave the others able to
+ * call it and draw nothing. Whether the capability exists at all is the switch this deployment
+ * has; see DeploymentConfig.generativeUi.
+ */
+ ...(config.generativeUi ? { openGenerativeUI: true } : {}),
// `identifyUser` is the Intelligence projection of the same person `identifyActor` returns:
// one resolver decides both whose threads these are and whose coworkers exist.
agents: createRequestAgents(
diff --git a/server/tests/config.test.ts b/server/tests/config.test.ts
index 0c887811..394124fc 100644
--- a/server/tests/config.test.ts
+++ b/server/tests/config.test.ts
@@ -599,6 +599,58 @@ describe("accessibility", () => {
);
});
+/**
+ * Whether a Bot may answer with an interface it wrote itself.
+ *
+ * Same shape as accessibility above, and tested to the same bar for the same reason: the off switch
+ * has a second reader. It is projected on /api/capabilities so the browser stops offering the tool
+ * too, so a value that silently failed to mean "off" would leave Bots generating interfaces nothing
+ * renders rather than merely leaving a capability on.
+ */
+describe("generated interfaces", () => {
+ /*
+ * The default is the whole point of this block. Written as a disable switch, an upgrade would hand
+ * every existing deployment the ability to run code a model wrote, and a deployment that builds
+ * its default branch automatically would acquire it without anybody deciding to.
+ */
+ test("are off when nothing is set", () => {
+ expect(loadConfig(baseEnvironment).generativeUi).toBe(false);
+ });
+
+ test.each(["true", "1"])("are on for OPENBOT_GENERATIVE_UI=%p", (value) => {
+ expect(
+ loadConfig({ ...baseEnvironment, OPENBOT_GENERATIVE_UI: value })
+ .generativeUi,
+ ).toBe(true);
+ });
+
+ /*
+ * Anything else is not a way of saying yes. A value nobody intended — a stray "false", an empty
+ * variable left behind by a template — should leave the capability off, which is the direction
+ * that cannot surprise anybody.
+ */
+ test.each(["false", "no", "", "yes", "TRUE", "on"])(
+ "stay off for OPENBOT_GENERATIVE_UI=%p",
+ (value) => {
+ expect(
+ loadConfig({ ...baseEnvironment, OPENBOT_GENERATIVE_UI: value })
+ .generativeUi,
+ ).toBe(false);
+ },
+ );
+
+ // The old spelling was a disable switch. It must not still work, or a deployment that set it
+ // would read as having made a choice it has not made under the new name.
+ test("ignore the disable switch this replaced", () => {
+ expect(
+ loadConfig({
+ ...baseEnvironment,
+ OPENBOT_GENERATIVE_UI_DISABLED: "false",
+ }).generativeUi,
+ ).toBe(false);
+ });
+});
+
/**
* Naming the private addresses an agent may live at.
*
diff --git a/server/tests/health.test.ts b/server/tests/health.test.ts
index 24f112cc..47912f34 100644
--- a/server/tests/health.test.ts
+++ b/server/tests/health.test.ts
@@ -26,6 +26,9 @@ describe("runtime capabilities", () => {
await expect(response.json()).resolves.toEqual({
mode: "intelligence",
durableHistory: true,
+ // Off until a deployment asks for it. The browser reads this to decide whether to offer the
+ // tool that generates an interface, so it has to be here and not only in the runtime.
+ generativeUi: false,
// Names only. The sign-in screen reads this to know which buttons to draw.
authProviders: ["google"],
// A boolean, not a list: naming the registered providers would tell anybody who loads the
@@ -47,12 +50,34 @@ describe("runtime capabilities", () => {
expect(Object.keys(parsed)).toEqual([
"mode",
"durableHistory",
+ "generativeUi",
"authProviders",
"ssoConfigured",
]);
// The provider list is names, never the clients and secrets behind them.
expect(body).not.toContain("google-client-secret");
});
+
+ /*
+ * The answer has to reach the browser, not just the runtime.
+ *
+ * The app offers the model the tool that generates an interface, and it decides whether to from
+ * this field. The two halves disagreeing is the one configuration this capability must not be able
+ * to end up in: runtime-only means the tool is never offered, browser-only means a Bot writes a
+ * whole interface that nothing renders.
+ */
+ test("reports generated interfaces as on when the deployment asked for them", async () => {
+ const enabled = createApp(
+ loadConfig(testEnvironment({ OPENBOT_GENERATIVE_UI: "true" })),
+ );
+
+ const response = await enabled.request(
+ "http://openbot.local/api/capabilities",
+ );
+
+ expect(response.status).toBe(200);
+ expect((await response.json()).generativeUi).toBe(true);
+ });
});
describe("authentication availability", () => {