Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.
Record why a message was not routed, not only where it went (#248)
Hotragn Pettugani50949d6eda31
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 25d106f0..5d41dd37 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -179,6 +179,31 @@ unknown session means "no opinion" and skips the generation check, so on exactlyshape it was written for, the check that stops a ref from a replaced computer resolving against alive one was silently absent. It now asks the supervisor when it does not know, by listing ratherthan by ensuring, so asking never starts a computer that had stopped.+### A routing trail says why a message was not routed, not only that it was not++Every untagged message writes a `channel.routed` row, and that row carried `fallback: true` for two+completely different situations: the router answering honestly that no specialist was a confident+match, which is the feature working, and the router not answering at all, which is an endpoint that is+down. Both read identically, so a deployment whose router had stopped working looked like one whose+messages were simply hard to route.++That is not hypothetical. The intent router spent an unknown period 404'ing on every deployment that+set `OPENAI_BASE_URL`, because a `/v1` was appended to a URL that already had one. It was fixed in+0.0.3, whose own note says untagged messages "silently stopped being routed and nothing said why".++The row now carries `undecided`, naming the cause: `unreachable`, `unparsed`, `off-roster`,+`unconfident`, or `one-candidate` — and `null` when the router did decide. Named values rather than a+sentence, because the useful question is how often, and a count needs something to group by.++Two smaller corrections came with it. A message routed to the only coworker that can reach the system+it names kept that as its reason and threw the cause away, so a router that had been down for a week+produced rows reading like reach-based routing working as intended; the cause now survives that path.+And an answer containing no JSON at all — a model replying in prose — was recorded as the router+naming a coworker off the roster, which sends whoever reads it to look at their roster rather than at+the model. It is now reported as unparsed, which is what it is.++Nothing changes about where a message goes. Every routing decision is the same decision it was.+### Notion joins the connector catalogueNotion is now a governed MCP connector, reached through Notion's own hosted server on thediff --git a/server/src/routing/classify.ts b/server/src/routing/classify.tsindex 5ae457b0..0286fa7c 100644--- a/server/src/routing/classify.ts+++ b/server/src/routing/classify.ts@@ -34,6 +34,35 @@ export type RoutingCandidate = {reaches?: readonly string[];};+/**+ * Why the router did not decide, when it did not.+ *+ * `fallback` says a message did not reach a coworker by an inferred match. It does not say whether+ * that was the router declining or the router failing, and those are different facts about a+ * deployment: "no specialist was a confident match" is the feature working, and "the router was+ * unreachable" is an endpoint that is down. Both landed on the same boolean and the same sentence.+ *+ * That mattered once already. #178 found the intent router appending `/v1` to a `OPENAI_BASE_URL`+ * that already carried one, so every call 404'd on every deployment that set the variable — and its+ * own changelog entry says untagged messages "silently stopped being routed and nothing said why".+ * The URL is fixed. The blindness that let it go unnoticed is this field.+ *+ * Named rather than free text so a trail can be counted: "this deployment routed nothing by inference+ * for a week" is a question `select ... where payload->>'undecided' = 'unreachable'` answers, and a+ * sentence is not.+ */+export type RoutingUndecided =+ /** The model call threw. An endpoint being down, a bad key, a gateway 404. */+ | "unreachable"+ /** It answered, and the answer was not JSON this could read. */+ | "unparsed"+ /** It named a coworker that is not on the roster it was given. */+ | "off-roster"+ /** It answered honestly that it was not sure. The feature working, not failing. */+ | "unconfident"+ /** Nothing to decide between, so no call was made. */+ | "one-candidate";+export type RoutingDecision = {agentId: string;name: string;@@ -41,6 +70,14 @@ export type RoutingDecision = {reason: string;/** True when this is the default rather than an inferred match: an honest "we were not sure". */fallback: boolean;+ /**+ * Why it was not decided, or null when it was.+ *+ * Survives the reach-based answer below. Landing on the one coworker that can reach the system a+ * message names is a good outcome and says nothing about whether the router answered, so replacing+ * this with that would hide exactly the failure it exists to count.+ */+ undecided: RoutingUndecided | null;};/** Below this the match is a guess, and a guess should defer to the default rather than surprise. */@@ -138,7 +175,10 @@ export function createIntentRouter(deps: {defaultId: string,): Promise<RoutingDecision> {const byId = new Map(candidates.map((c) => [c.id, c]));- const fallback = (reason: string): RoutingDecision => {+ const fallback = (+ reason: string,+ undecided: RoutingUndecided,+ ): RoutingDecision => {/** Before the default, ask whether the message named a system only one coworker can reach.*@@ -157,18 +197,33 @@ export function createIntentRouter(deps: {name: reachable.name,reason: `the only coworker that can reach ${reachable.system}`,fallback: true,+ // Carried through. Reach answered where the message went; it did not answer whether the+ // router did, and a router that has been down for a week must not read as this.+ undecided,};}const chosen = byId.get(defaultId) ?? candidates[0];return chosen- ? { agentId: chosen.id, name: chosen.name, reason, fallback: true }+ ? {+ agentId: chosen.id,+ name: chosen.name,+ reason,+ fallback: true,+ undecided,+ }: // No roster at all is a misconfiguration, not a routing outcome; surface the default id.- { agentId: defaultId, name: defaultId, reason, fallback: true };+ {+ agentId: defaultId,+ name: defaultId,+ reason,+ fallback: true,+ undecided,+ };};// Nothing to decide between: one coworker, or none but the default.if (candidates.length <= 1) {- return fallback("the only coworker available");+ return fallback("the only coworker available", "one-candidate");}let raw: string;@@ -177,17 +232,34 @@ export function createIntentRouter(deps: {} catch {return fallback("sent to your default while the router was unreachable",+ "unreachable",);}let parsed: { agentId?: unknown; reason?: unknown; confidence?: unknown };+ // The model is asked for bare JSON, but tolerate a fenced or padded answer. Named `jsonPart`+ // rather than `match`, which is the roster lookup a few lines below.+ const jsonPart = raw.match(/\{[\s\S]*\}/);+ /*+ * An answer with no object in it at all is unparsed, not off-roster.+ *+ * This used to fall through as `{}`, so a model replying in prose — "I think Risk Analyst is+ * best" — reached the roster check, found no id, and was recorded as the router having named a+ * coworker that does not exist. That points whoever reads it at their roster, when what is+ * wrong is that the model is not answering in the format it was asked for.+ */+ if (!jsonPart) {+ return fallback(+ "sent to your default; the router's answer did not parse",+ "unparsed",+ );+ }try {- // The model is asked for bare JSON, but tolerate a fenced or padded answer.- const match = raw.match(/\{[\s\S]*\}/);- parsed = match ? JSON.parse(match[0]) : {};+ parsed = JSON.parse(jsonPart[0]);} catch {return fallback("sent to your default; the router's answer did not parse",+ "unparsed",);}@@ -197,6 +269,7 @@ export function createIntentRouter(deps: {// A returned id that is not on the roster is the dangerous case: never act on it.return fallback("sent to your default; the router named no coworker on your roster",+ "off-roster",);}const confidence =@@ -204,6 +277,7 @@ export function createIntentRouter(deps: {if (confidence < MIN_CONFIDENCE) {return fallback("sent to your default; no specialist was a confident match",+ "unconfident",);}@@ -211,7 +285,14 @@ export function createIntentRouter(deps: {typeof parsed.reason === "string" && parsed.reason.trim()? parsed.reason.trim(): `matches ${match.name}`;- return { agentId: match.id, name: match.name, reason, fallback: false };+ // The router answered, on the roster, confidently. The only path where nothing was undecided.+ return {+ agentId: match.id,+ name: match.name,+ reason,+ fallback: false,+ undecided: null,+ };},// Split out so the prompt-build + call is one seam the tests can leave alone.diff --git a/server/src/routing/routes.ts b/server/src/routing/routes.tsindex 5d523c20..e8ae0720 100644--- a/server/src/routing/routes.ts+++ b/server/src/routing/routes.ts@@ -4,7 +4,11 @@ import type { AuditStore } from "../audit";import { recordAuditEvent } from "../audit";import type { AppVariables } from "../auth/guards";import type { AgentProfileStore } from "../agents/profile-store";-import type { IntentRouter, RoutingCandidate } from "./classify";+import type {+ IntentRouter,+ RoutingCandidate,+ RoutingUndecided,+} from "./classify";const DEV_ACTOR_EMAIL = "[email protected]";@@ -68,6 +72,14 @@ export function createRoutingRoutes(fallback: boolean,viaMention: boolean,candidates: readonly string[],+ /*+ * Why the router did not decide, when it did not.+ *+ * On the row rather than only in the sentence, because this is the field a deployment counts. A+ * router that has been unreachable for a week produced rows that read like ordinary+ * "no confident match" ones, which is how #178 went unnoticed for as long as it did.+ */+ undecided: RoutingUndecided | null,): Promise<void> {if (!auditStore) return;await recordAuditEvent(auditStore, {@@ -75,7 +87,7 @@ export function createRoutingRoutes(targetType: "agent",targetId: chosen,...(actorUserId ? { actorUserId } : {}),- payload: { chosen, reason, fallback, viaMention, candidates },+ payload: { chosen, reason, fallback, viaMention, candidates, undecided },});}@@ -126,7 +138,16 @@ export function createRoutingRoutes(* @zopeVaibhav had this right in #134.*/const reason = "named by the person asking";- await record(actorId(actor), chosen.id, reason, false, true, [chosen.id]);+ // The person chose. Nothing was left to the router, so nothing about it was undecided.+ await record(+ actorId(actor),+ chosen.id,+ reason,+ false,+ true,+ [chosen.id],+ null,+ );return context.json({agentId: chosen.id,name: chosen.name,@@ -164,6 +185,7 @@ export function createRoutingRoutes(decision.fallback,false,candidates.map((c) => c.id),+ decision.undecided,);return context.json({@@ -171,6 +193,7 @@ export function createRoutingRoutes(name: decision.name,reason: decision.reason,fallback: decision.fallback,+ undecided: decision.undecided,viaMention: false,});});diff --git a/server/tests/routing-classify.test.ts b/server/tests/routing-classify.test.tsindex 9407879b..f801f5f6 100644--- a/server/tests/routing-classify.test.ts+++ b/server/tests/routing-classify.test.ts@@ -269,3 +269,130 @@ describe("falling back to somebody who can actually answer", () => {expect(decision.fallback).toBe(false);});});++/**+ * Why a message was not routed, as a thing a deployment can count.+ *+ * `fallback` says an inferred match did not happen. It does not say whether the router declined or+ * the router failed, and only one of those is a deployment with something wrong with it. #178 was+ * exactly that: the router 404'd on every deployment that set `OPENAI_BASE_URL`, and its changelog+ * says untagged messages "silently stopped being routed and nothing said why". These pin the field+ * that answers it.+ */+describe("saying why a message was not routed", () => {+ test("a confident match leaves nothing undecided", async () => {+ const decision = await withAnswer(+ JSON.stringify({+ agentId: "risk-analyst",+ reason: "fraud",+ confidence: 0.9,+ }),+ ).route("is this transaction fraud", ROSTER, "general-assistant");++ expect(decision.fallback).toBe(false);+ expect(decision.undecided).toBeNull();+ });++ test("an unreachable router is named as unreachable", async () => {+ const decision = await throwing().route(+ "anything",+ ROSTER,+ "general-assistant",+ );+ expect(decision.undecided).toBe("unreachable");+ });++ test("an answer that does not parse is named as unparsed", async () => {+ const decision = await withAnswer('{ "agentId": }').route(+ "anything",+ ROSTER,+ "general-assistant",+ );+ expect(decision.undecided).toBe("unparsed");+ });++ test("prose with no JSON in it is unparsed, not off-roster", async () => {+ /*+ * A model answering in sentences is a model not following the format. Recorded as off-roster it+ * reads as a roster problem, and whoever investigates goes and looks at their roster.+ */+ const decision = await withAnswer("I think Risk Analyst is best.").route(+ "anything",+ ROSTER,+ "general-assistant",+ );+ expect(decision.undecided).toBe("unparsed");+ });++ test("an id that is not on the roster is named as off-roster", async () => {+ const decision = await withAnswer(+ JSON.stringify({ agentId: "somebody-else", confidence: 0.9 }),+ ).route("anything", ROSTER, "general-assistant");+ expect(decision.undecided).toBe("off-roster");+ });++ test("an honest low confidence is named as unconfident, not as a failure", async () => {+ // The one cause that is the feature working. Counting it with the failures would make the+ // number useless, which is the whole reason this is a named cause rather than a boolean.+ const decision = await withAnswer(+ JSON.stringify({ agentId: "risk-analyst", confidence: 0.2 }),+ ).route("anything", ROSTER, "general-assistant");+ expect(decision.undecided).toBe("unconfident");+ });++ test("a roster of one is named as one-candidate, and asks nothing", async () => {+ const decision = await throwing().route(+ "anything",+ [ROSTER[0] as RoutingCandidate],+ "general-assistant",+ );+ expect(decision.undecided).toBe("one-candidate");+ });++ test("the reach answer does not hide that the router never answered", async () => {+ /*+ * THE CASE THIS EXISTS FOR. Landing on the only coworker that can reach Google Drive is a good+ * outcome, and it says nothing about whether the router was up. Before this the reach sentence+ * replaced the failure entirely, so a deployment whose router had been down for a week produced+ * rows that read exactly like reach-based routing working as intended.+ */+ const reaching: RoutingCandidate[] = [+ { ...(ROSTER[0] as RoutingCandidate) },+ {+ ...(ROSTER[1] as RoutingCandidate),+ reaches: ["google-drive"],+ },+ ];++ const decision = await throwing().route(+ "find the PRD in google drive",+ reaching,+ "general-assistant",+ );++ // Still routed by reach, and still a fallback — both unchanged.+ expect(decision.agentId).toBe("knowledge");+ expect(decision.reason).toContain("google-drive");+ expect(decision.fallback).toBe(true);+ // And the router failure survives it.+ expect(decision.undecided).toBe("unreachable");+ });++ test("reach after a low-confidence answer says unconfident, not unreachable", async () => {+ // The two are told apart on the reach path as well, or the count is wrong wherever reach fires.+ const reaching: RoutingCandidate[] = [+ { ...(ROSTER[0] as RoutingCandidate) },+ {+ ...(ROSTER[1] as RoutingCandidate),+ reaches: ["google-drive"],+ },+ ];++ const decision = await withAnswer(+ JSON.stringify({ agentId: "knowledge", confidence: 0.1 }),+ ).route("find the PRD in google drive", reaching, "general-assistant");++ expect(decision.agentId).toBe("knowledge");+ expect(decision.undecided).toBe("unconfident");+ });+});diff --git a/server/tests/routing-routes.test.ts b/server/tests/routing-routes.test.tsindex 974996b2..b974d136 100644--- a/server/tests/routing-routes.test.ts+++ b/server/tests/routing-routes.test.ts@@ -4,7 +4,7 @@ import { Hono } from "hono";import type { AgentProfileStore } from "../src/agents/profile-store";import type { AuditStore } from "../src/audit";import type { AppVariables } from "../src/auth/guards";-import type { IntentRouter } from "../src/routing/classify";+import type { IntentRouter, RoutingUndecided } from "../src/routing/classify";import { createRoutingRoutes } from "../src/routing/routes";/**@@ -47,7 +47,7 @@ type Recorded = {payload: Record<string, unknown>;};-function app(options: { routed?: string } = {}) {+function app(options: { routed?: string; undecided?: RoutingUndecided } = {}) {const written: Recorded[] = [];/** Every call the router was asked to make, so "never asked" is an assertion and not a hope. */const asked: string[] = [];@@ -72,7 +72,8 @@ function app(options: { routed?: string } = {}) {agentId: chosen,name: ROSTER.find((a) => a.id === chosen)?.name ?? chosen,reason: "matches what it is for",- fallback: false,+ fallback: options.undecided !== undefined,+ undecided: options.undecided ?? null,};},} as unknown as IntentRouter;@@ -185,3 +186,47 @@ describe("recording which coworker a message went to", () => {expect(asked).toEqual(["hello"]);});});++/**+ * Why it was not decided, on the row rather than only in the sentence.+ *+ * The reason is prose for a person. This is the field a deployment counts, and counting is the point:+ * a router that has been unreachable for a week is invisible until somebody can ask how often.+ */+describe("recording why a message was not routed", () => {+ test("an unreachable router is on the row, not only in the sentence", async () => {+ const { server, written } = app({ undecided: "unreachable" });++ await post(server, { text: "what is our PTO policy" });++ expect(written[0]?.payload).toMatchObject({+ chosen: "knowledge",+ fallback: true,+ undecided: "unreachable",+ });+ });++ test("a decided routing records no cause", async () => {+ const { server, written } = app();++ await post(server, { text: "what is our PTO policy" });++ expect(written[0]?.payload).toMatchObject({+ fallback: false,+ undecided: null,+ });+ });++ test("a coworker the person named records no cause either", async () => {+ // Nothing was left to the router, so there is nothing about it to have failed. Recording a cause+ // here would count a person's own choice as a routing failure.+ const { server, written } = app();++ await post(server, { text: "hello", agentId: "risk-analyst" });++ expect(written[0]?.payload).toMatchObject({+ viaMention: true,+ undecided: null,+ });+ });+});