Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Make every Bot say where its answer came from (#153)

David McKay963f99d3d57f
diff --git a/docs/configuration.md b/docs/configuration.md
index 333424f7..95adf337 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -319,6 +319,16 @@ Each agent requires `id`, `name`, `title`, `role_description`, and `type`.
| `built-in` | `system_prompt` |
| `remote-ag-ui` | `endpoint` |
+The two types are told different amounts, which is easy to miss. A `built-in` agent gets its
+`system_prompt`; a `remote-ag-ui` agent has none, and its `role_description` is the only instruction
+it ever receives from the package. Write that sentence as the whole brief for the Bot, not as a
+label for a list.
+
+Both kinds are also told, by the deployment rather than by the package, to say where an answer came
+from: cite what a tool returned, and say plainly when the answer is from the model's own knowledge
+rather than from anything it read. That rule is not written per agent, so it cannot be missing from
+the next one somebody adds.
+
Any `${NAME}` in a package file is replaced with that environment variable, so one package works
against a local stack, a staging one and production. `${NAME:-fallback}` uses the fallback when the
name is unset or empty, which is how the example package points at the Bot in the box without
diff --git a/server/src/copilot.ts b/server/src/copilot.ts
index 8582a85d..d6920468 100644
--- a/server/src/copilot.ts
+++ b/server/src/copilot.ts
@@ -7,7 +7,10 @@ import {
} from "@copilotkit/runtime/v2";
import { createCopilotHonoHandler } from "@copilotkit/runtime/v2/hono";
import { z } from "zod";
-import { COMPUTER_GUIDANCE } from "../../shared/bot-prompt";
+import {
+ COMPUTER_GUIDANCE,
+ PROVENANCE_GUIDANCE,
+} from "../../shared/bot-prompt";
import { grantedToolGuidance } from "./plugins/tools";
import type { AgentActor } from "./agents/profile-types";
import type { StallGuard } from "./channels/stall-guard";
@@ -97,6 +100,14 @@ export function standingRoleMessage(
`You are ${profile.name}, ${profile.title}.`,
profile.roleDescription,
"This standing role applies in every channel. Treat channel messages as task-specific instructions within it.",
+ /*
+ * Here rather than in the package, because for a remote Bot the standing role is the only
+ * instruction there is: `role_description` is one sentence somebody wrote about what it is
+ * for, and nothing else reaches it. The compliance Bot that answered a filing question with
+ * thresholds and deadlines and no source was a `remote-ag-ui` agent whose entire prompt was
+ * "Investigate policies, transaction monitoring, and control evidence."
+ */
+ PROVENANCE_GUIDANCE,
].join("\n\n"),
};
}
@@ -210,6 +221,14 @@ export function builtInAgentConfiguration(
*/
prompt: [
agent.systemPrompt,
+ /*
+ * Unconditional, unlike the two below it.
+ *
+ * Those describe things a deployment may or may not have. This describes how to answer at all,
+ * and a Bot with no tools and no computer needs it most: it has nothing to read, so everything
+ * it says comes from its own knowledge, and saying so is the only honest move available.
+ */
+ PROVENANCE_GUIDANCE,
...(grantedToolGuidance(tools) ? [grantedToolGuidance(tools)] : []),
...(computerGuidance ? [computerGuidance] : []),
].join("\n\n"),
diff --git a/server/tests/copilot.test.ts b/server/tests/copilot.test.ts
index 87791b21..be3a4847 100644
--- a/server/tests/copilot.test.ts
+++ b/server/tests/copilot.test.ts
@@ -9,6 +9,7 @@ import {
resolveRuntimeAgents,
standingRoleMessage,
} from "../src/copilot";
+import { PROVENANCE_GUIDANCE } from "../../shared/bot-prompt";
import { grantedToolGuidance } from "../src/plugins/tools";
// Every agent row now joins its profile, so the row a coworker is built from always names it.
@@ -99,7 +100,9 @@ describe("registered Copilot agents", () => {
),
).toEqual({
model: "openai/gpt-4.1",
- prompt: "Be helpful.",
+ // The provenance rule is unconditional, so even a Bot with no tools and no computer carries
+ // it. That Bot needs it most: nothing it says was read anywhere.
+ prompt: `Be helpful.\n\n${PROVENANCE_GUIDANCE}`,
apiKey: "openai-secret",
});
});
@@ -324,6 +327,10 @@ describe("standing agent roles", () => {
"You are Expense Manager, Finance Operations.",
"Review receipts, categorize expenses, and prepare reimbursement reports.",
"This standing role applies in every channel. Treat channel messages as task-specific instructions within it.",
+ // For a remote Bot this message is the whole instruction, so the provenance rule has to
+ // travel in it or the Bot never hears it. Referenced rather than restated, so the assertion
+ // stays exact without pinning the wording twice.
+ PROVENANCE_GUIDANCE,
].join("\n\n"),
});
});
@@ -444,6 +451,7 @@ describe("standing agent roles", () => {
"You are Expense Manager, Finance Operations.",
"Reconcile corporate card statements.",
"This standing role applies in every channel. Treat channel messages as task-specific instructions within it.",
+ PROVENANCE_GUIDANCE,
].join("\n\n"),
);
});
@@ -586,3 +594,76 @@ describe("what a Bot is told it holds", () => {
);
});
});
+
+/**
+ * Where an answer came from, on every Bot rather than the ones somebody remembered.
+ *
+ * Asked what the obligation was for twelve cash deposits under the reporting threshold, the
+ * compliance Bot answered with a filing requirement, a dollar threshold, a thirty-day deadline and a
+ * five-year retention period. The audit trail for that turn holds exactly one row: the routing
+ * decision. No tool call, no source, and nothing saying the answer came from the model.
+ *
+ * One package's `knowledge` Bot had a rule against this in its YAML. The Bot whose entire subject is
+ * regulatory obligation did not, because a `remote-ag-ui` agent gets its role description and
+ * nothing else. That asymmetry is the bug: a rule this important living in one agent's YAML is a
+ * rule the next agent will not have.
+ *
+ * So both paths are asserted, because they are built by different functions and a fix to one is not
+ * a fix to the other.
+ */
+describe("where a Bot says its answer came from", () => {
+ test("a built-in Bot carries the rule even holding nothing at all", () => {
+ // The Bot that needs it most. No tools and no computer means nothing it says was read anywhere.
+ const prompt = builtInAgentConfiguration(
+ {
+ id: "general-assistant",
+ name: "General Assistant",
+ type: "built_in",
+ systemPrompt: "Be helpful.",
+ },
+ { provider: "openai", defaultModel: "gpt-4.1" },
+ "openai-secret",
+ ).prompt as string;
+
+ expect(prompt).toContain(PROVENANCE_GUIDANCE);
+ });
+
+ test("a remote Bot carries it in the only instruction it ever gets", () => {
+ const content = standingRoleMessage({
+ id: "risk-analyst",
+ name: "Risk Analyst",
+ title: "Risk & Compliance",
+ roleDescription:
+ "Investigate policies, transaction monitoring, and control evidence.",
+ }).content;
+
+ expect(content).toContain(PROVENANCE_GUIDANCE);
+ });
+
+ test("it does not send the Bot hunting for a source", () => {
+ /*
+ * The failure mode of the first attempt at this, which never left a branch. Told to find a
+ * source, Bots went reading the open web and looped on a government 404 page. An unsourced
+ * answer that says it is unsourced is honest; a search that never ends is a Bot that never
+ * answers.
+ */
+ const guidance = PROVENANCE_GUIDANCE.toLowerCase().replace(/\s+/g, " ");
+ expect(guidance).toContain("this is not an instruction to go looking");
+ expect(guidance).toContain("mark it plainly as unverified");
+ expect(guidance).toContain("do not go hunting the open web");
+ });
+
+ test("it names the answers that must not be stated without a source", () => {
+ // The general rule is easy to read past. The list is what makes it bite on the turn that
+ // produced this: a threshold, a deadline, a filing obligation, a figure.
+ const guidance = PROVENANCE_GUIDANCE.toLowerCase().replace(/\s+/g, " ");
+ for (const kind of [
+ "threshold",
+ "deadline",
+ "filing obligation",
+ "figure",
+ ]) {
+ expect(guidance).toContain(kind);
+ }
+ });
+});
diff --git a/shared/bot-prompt.ts b/shared/bot-prompt.ts
index 63897d0d..49a5c306 100644
--- a/shared/bot-prompt.ts
+++ b/shared/bot-prompt.ts
@@ -83,3 +83,57 @@ export const COMPUTER_GUIDANCE = COMPUTER_GUIDANCE_LINES.reduce<string[]>(
},
[""],
).join("\n\n");
+
+/**
+ * Where an answer came from, said out loud.
+ *
+ * Asked "a customer made 12 cash deposits just under the reporting threshold, what is our
+ * obligation", the compliance Bot answered at length and with confidence: file a SAR, $5,000 or
+ * more, within 30 calendar days of initial detection, retain for 5 years. The audit trail for that
+ * turn holds one row, the routing decision. No tool call, no source, and no sentence anywhere saying
+ * the answer came from the model rather than from anything this deployment can reach.
+ *
+ * Several of those numbers may well be right, and that is the problem. A confident, plausible,
+ * unsourced answer is indistinguishable from a confident, plausible, wrong one, and nothing marked
+ * the difference on a question about whether to file, against what threshold, inside what deadline.
+ *
+ * One package's `knowledge` Bot had a rule against exactly this, written into its YAML by whoever
+ * happened to think of it. The Bot whose whole subject is regulatory obligation did not, because
+ * `remote-ag-ui` gets its role description and nothing else. A rule that important sitting in one
+ * agent's YAML is a rule that will be missing from the next agent somebody adds, so it lives here
+ * and every Bot gets it.
+ *
+ * The last paragraph is not padding. An earlier attempt at this told Bots to go and find a source,
+ * and they went hunting the open web and looped on a government 404 page, which is worse than the
+ * problem: an unsourced answer marked as unsourced is honest, and a hunt for one is a Bot that
+ * never answers. The instruction is to say where the answer came from, not to go looking.
+ */
+const PROVENANCE_GUIDANCE_LINES = [
+ "Say where an answer came from. When you read it with one of your tools, cite what you read.",
+ "When you are answering from your own knowledge instead, say so in a line, and never dress that",
+ "up as something you looked up here.",
+ "",
+ "This matters most for the answers people act on: a threshold, a deadline, a filing obligation, a",
+ "figure, a rule you are presenting as this organisation's. Never state one of those as established",
+ "here without having read it somewhere you can name. Saying 'I have not checked this against your",
+ "own policy or the current regulation' costs you a sentence. Being confidently wrong about a",
+ "number somebody acts on costs them a great deal more.",
+ "",
+ "This is not an instruction to go looking. If nothing you can reach covers the question, answer as",
+ "well as you can and mark it plainly as unverified. Do not go hunting the open web for something",
+ "to cite, and do not keep retrying a page that is not giving you one: an unsourced answer that",
+ "says it is unsourced is honest, and a search that never ends is a Bot that never answers.",
+];
+
+export const PROVENANCE_GUIDANCE = PROVENANCE_GUIDANCE_LINES.reduce<string[]>(
+ (paragraphs, line) => {
+ if (line === "") {
+ paragraphs.push("");
+ return paragraphs;
+ }
+ const last = paragraphs.length - 1;
+ paragraphs[last] = paragraphs[last] ? `${paragraphs[last]} ${line}` : line;
+ return paragraphs;
+ },
+ [""],
+).join("\n\n");