Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

Record why a message was not routed, not only where it went (#248)

Hotragn Pettugani50949d6eda31
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 25d106f0..5d41dd37 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -179,6 +179,31 @@ unknown session means "no opinion" and skips the generation check, so on exactly
shape it was written for, the check that stops a ref from a replaced computer resolving against a
live one was silently absent. It now asks the supervisor when it does not know, by listing rather
than by ensuring, so asking never starts a computer that had stopped.
+### A routing trail says why a message was not routed, not only that it was not
+
+Every untagged message writes a `channel.routed` row, and that row carried `fallback: true` for two
+completely different situations: the router answering honestly that no specialist was a confident
+match, which is the feature working, and the router not answering at all, which is an endpoint that is
+down. Both read identically, so a deployment whose router had stopped working looked like one whose
+messages were simply hard to route.
+
+That is not hypothetical. The intent router spent an unknown period 404'ing on every deployment that
+set `OPENAI_BASE_URL`, because a `/v1` was appended to a URL that already had one. It was fixed in
+0.0.3, whose own note says untagged messages "silently stopped being routed and nothing said why".
+
+The row now carries `undecided`, naming the cause: `unreachable`, `unparsed`, `off-roster`,
+`unconfident`, or `one-candidate` — and `null` when the router did decide. Named values rather than a
+sentence, because the useful question is how often, and a count needs something to group by.
+
+Two smaller corrections came with it. A message routed to the only coworker that can reach the system
+it names kept that as its reason and threw the cause away, so a router that had been down for a week
+produced rows reading like reach-based routing working as intended; the cause now survives that path.
+And an answer containing no JSON at all — a model replying in prose — was recorded as the router
+naming a coworker off the roster, which sends whoever reads it to look at their roster rather than at
+the model. It is now reported as unparsed, which is what it is.
+
+Nothing changes about where a message goes. Every routing decision is the same decision it was.
+
### Notion joins the connector catalogue
Notion is now a governed MCP connector, reached through Notion's own hosted server on the
diff --git a/server/src/routing/classify.ts b/server/src/routing/classify.ts
index 5ae457b0..0286fa7c 100644
--- a/server/src/routing/classify.ts
+++ b/server/src/routing/classify.ts
@@ -34,6 +34,35 @@ export type RoutingCandidate = {
reaches?: readonly string[];
};
+/**
+ * Why the router did not decide, when it did not.
+ *
+ * `fallback` says a message did not reach a coworker by an inferred match. It does not say whether
+ * that was the router declining or the router failing, and those are different facts about a
+ * deployment: "no specialist was a confident match" is the feature working, and "the router was
+ * unreachable" is an endpoint that is down. Both landed on the same boolean and the same sentence.
+ *
+ * That mattered once already. #178 found the intent router appending `/v1` to a `OPENAI_BASE_URL`
+ * that already carried one, so every call 404'd on every deployment that set the variable — and its
+ * own changelog entry says untagged messages "silently stopped being routed and nothing said why".
+ * The URL is fixed. The blindness that let it go unnoticed is this field.
+ *
+ * Named rather than free text so a trail can be counted: "this deployment routed nothing by inference
+ * for a week" is a question `select ... where payload->>'undecided' = 'unreachable'` answers, and a
+ * sentence is not.
+ */
+export type RoutingUndecided =
+ /** The model call threw. An endpoint being down, a bad key, a gateway 404. */
+ | "unreachable"
+ /** It answered, and the answer was not JSON this could read. */
+ | "unparsed"
+ /** It named a coworker that is not on the roster it was given. */
+ | "off-roster"
+ /** It answered honestly that it was not sure. The feature working, not failing. */
+ | "unconfident"
+ /** Nothing to decide between, so no call was made. */
+ | "one-candidate";
+
export type RoutingDecision = {
agentId: string;
name: string;
@@ -41,6 +70,14 @@ export type RoutingDecision = {
reason: string;
/** True when this is the default rather than an inferred match: an honest "we were not sure". */
fallback: boolean;
+ /**
+ * Why it was not decided, or null when it was.
+ *
+ * Survives the reach-based answer below. Landing on the one coworker that can reach the system a
+ * message names is a good outcome and says nothing about whether the router answered, so replacing
+ * this with that would hide exactly the failure it exists to count.
+ */
+ undecided: RoutingUndecided | null;
};
/** Below this the match is a guess, and a guess should defer to the default rather than surprise. */
@@ -138,7 +175,10 @@ export function createIntentRouter(deps: {
defaultId: string,
): Promise<RoutingDecision> {
const byId = new Map(candidates.map((c) => [c.id, c]));
- const fallback = (reason: string): RoutingDecision => {
+ const fallback = (
+ reason: string,
+ undecided: RoutingUndecided,
+ ): RoutingDecision => {
/*
* Before the default, ask whether the message named a system only one coworker can reach.
*
@@ -157,18 +197,33 @@ export function createIntentRouter(deps: {
name: reachable.name,
reason: `the only coworker that can reach ${reachable.system}`,
fallback: true,
+ // Carried through. Reach answered where the message went; it did not answer whether the
+ // router did, and a router that has been down for a week must not read as this.
+ undecided,
};
}
const chosen = byId.get(defaultId) ?? candidates[0];
return chosen
- ? { agentId: chosen.id, name: chosen.name, reason, fallback: true }
+ ? {
+ agentId: chosen.id,
+ name: chosen.name,
+ reason,
+ fallback: true,
+ undecided,
+ }
: // No roster at all is a misconfiguration, not a routing outcome; surface the default id.
- { agentId: defaultId, name: defaultId, reason, fallback: true };
+ {
+ agentId: defaultId,
+ name: defaultId,
+ reason,
+ fallback: true,
+ undecided,
+ };
};
// Nothing to decide between: one coworker, or none but the default.
if (candidates.length <= 1) {
- return fallback("the only coworker available");
+ return fallback("the only coworker available", "one-candidate");
}
let raw: string;
@@ -177,17 +232,34 @@ export function createIntentRouter(deps: {
} catch {
return fallback(
"sent to your default while the router was unreachable",
+ "unreachable",
);
}
let parsed: { agentId?: unknown; reason?: unknown; confidence?: unknown };
+ // The model is asked for bare JSON, but tolerate a fenced or padded answer. Named `jsonPart`
+ // rather than `match`, which is the roster lookup a few lines below.
+ const jsonPart = raw.match(/\{[\s\S]*\}/);
+ /*
+ * An answer with no object in it at all is unparsed, not off-roster.
+ *
+ * This used to fall through as `{}`, so a model replying in prose — "I think Risk Analyst is
+ * best" — reached the roster check, found no id, and was recorded as the router having named a
+ * coworker that does not exist. That points whoever reads it at their roster, when what is
+ * wrong is that the model is not answering in the format it was asked for.
+ */
+ if (!jsonPart) {
+ return fallback(
+ "sent to your default; the router's answer did not parse",
+ "unparsed",
+ );
+ }
try {
- // The model is asked for bare JSON, but tolerate a fenced or padded answer.
- const match = raw.match(/\{[\s\S]*\}/);
- parsed = match ? JSON.parse(match[0]) : {};
+ parsed = JSON.parse(jsonPart[0]);
} catch {
return fallback(
"sent to your default; the router's answer did not parse",
+ "unparsed",
);
}
@@ -197,6 +269,7 @@ export function createIntentRouter(deps: {
// A returned id that is not on the roster is the dangerous case: never act on it.
return fallback(
"sent to your default; the router named no coworker on your roster",
+ "off-roster",
);
}
const confidence =
@@ -204,6 +277,7 @@ export function createIntentRouter(deps: {
if (confidence < MIN_CONFIDENCE) {
return fallback(
"sent to your default; no specialist was a confident match",
+ "unconfident",
);
}
@@ -211,7 +285,14 @@ export function createIntentRouter(deps: {
typeof parsed.reason === "string" && parsed.reason.trim()
? parsed.reason.trim()
: `matches ${match.name}`;
- return { agentId: match.id, name: match.name, reason, fallback: false };
+ // The router answered, on the roster, confidently. The only path where nothing was undecided.
+ return {
+ agentId: match.id,
+ name: match.name,
+ reason,
+ fallback: false,
+ undecided: null,
+ };
},
// Split out so the prompt-build + call is one seam the tests can leave alone.
diff --git a/server/src/routing/routes.ts b/server/src/routing/routes.ts
index 5d523c20..e8ae0720 100644
--- a/server/src/routing/routes.ts
+++ b/server/src/routing/routes.ts
@@ -4,7 +4,11 @@ import type { AuditStore } from "../audit";
import { recordAuditEvent } from "../audit";
import type { AppVariables } from "../auth/guards";
import type { AgentProfileStore } from "../agents/profile-store";
-import type { IntentRouter, RoutingCandidate } from "./classify";
+import type {
+ IntentRouter,
+ RoutingCandidate,
+ RoutingUndecided,
+} from "./classify";
const DEV_ACTOR_EMAIL = "[email protected]";
@@ -68,6 +72,14 @@ export function createRoutingRoutes(
fallback: boolean,
viaMention: boolean,
candidates: readonly string[],
+ /*
+ * Why the router did not decide, when it did not.
+ *
+ * On the row rather than only in the sentence, because this is the field a deployment counts. A
+ * router that has been unreachable for a week produced rows that read like ordinary
+ * "no confident match" ones, which is how #178 went unnoticed for as long as it did.
+ */
+ undecided: RoutingUndecided | null,
): Promise<void> {
if (!auditStore) return;
await recordAuditEvent(auditStore, {
@@ -75,7 +87,7 @@ export function createRoutingRoutes(
targetType: "agent",
targetId: chosen,
...(actorUserId ? { actorUserId } : {}),
- payload: { chosen, reason, fallback, viaMention, candidates },
+ payload: { chosen, reason, fallback, viaMention, candidates, undecided },
});
}
@@ -126,7 +138,16 @@ export function createRoutingRoutes(
* @zopeVaibhav had this right in #134.
*/
const reason = "named by the person asking";
- await record(actorId(actor), chosen.id, reason, false, true, [chosen.id]);
+ // The person chose. Nothing was left to the router, so nothing about it was undecided.
+ await record(
+ actorId(actor),
+ chosen.id,
+ reason,
+ false,
+ true,
+ [chosen.id],
+ null,
+ );
return context.json({
agentId: chosen.id,
name: chosen.name,
@@ -164,6 +185,7 @@ export function createRoutingRoutes(
decision.fallback,
false,
candidates.map((c) => c.id),
+ decision.undecided,
);
return context.json({
@@ -171,6 +193,7 @@ export function createRoutingRoutes(
name: decision.name,
reason: decision.reason,
fallback: decision.fallback,
+ undecided: decision.undecided,
viaMention: false,
});
});
diff --git a/server/tests/routing-classify.test.ts b/server/tests/routing-classify.test.ts
index 9407879b..f801f5f6 100644
--- a/server/tests/routing-classify.test.ts
+++ b/server/tests/routing-classify.test.ts
@@ -269,3 +269,130 @@ describe("falling back to somebody who can actually answer", () => {
expect(decision.fallback).toBe(false);
});
});
+
+/**
+ * Why a message was not routed, as a thing a deployment can count.
+ *
+ * `fallback` says an inferred match did not happen. It does not say whether the router declined or
+ * the router failed, and only one of those is a deployment with something wrong with it. #178 was
+ * exactly that: the router 404'd on every deployment that set `OPENAI_BASE_URL`, and its changelog
+ * says untagged messages "silently stopped being routed and nothing said why". These pin the field
+ * that answers it.
+ */
+describe("saying why a message was not routed", () => {
+ test("a confident match leaves nothing undecided", async () => {
+ const decision = await withAnswer(
+ JSON.stringify({
+ agentId: "risk-analyst",
+ reason: "fraud",
+ confidence: 0.9,
+ }),
+ ).route("is this transaction fraud", ROSTER, "general-assistant");
+
+ expect(decision.fallback).toBe(false);
+ expect(decision.undecided).toBeNull();
+ });
+
+ test("an unreachable router is named as unreachable", async () => {
+ const decision = await throwing().route(
+ "anything",
+ ROSTER,
+ "general-assistant",
+ );
+ expect(decision.undecided).toBe("unreachable");
+ });
+
+ test("an answer that does not parse is named as unparsed", async () => {
+ const decision = await withAnswer('{ "agentId": }').route(
+ "anything",
+ ROSTER,
+ "general-assistant",
+ );
+ expect(decision.undecided).toBe("unparsed");
+ });
+
+ test("prose with no JSON in it is unparsed, not off-roster", async () => {
+ /*
+ * A model answering in sentences is a model not following the format. Recorded as off-roster it
+ * reads as a roster problem, and whoever investigates goes and looks at their roster.
+ */
+ const decision = await withAnswer("I think Risk Analyst is best.").route(
+ "anything",
+ ROSTER,
+ "general-assistant",
+ );
+ expect(decision.undecided).toBe("unparsed");
+ });
+
+ test("an id that is not on the roster is named as off-roster", async () => {
+ const decision = await withAnswer(
+ JSON.stringify({ agentId: "somebody-else", confidence: 0.9 }),
+ ).route("anything", ROSTER, "general-assistant");
+ expect(decision.undecided).toBe("off-roster");
+ });
+
+ test("an honest low confidence is named as unconfident, not as a failure", async () => {
+ // The one cause that is the feature working. Counting it with the failures would make the
+ // number useless, which is the whole reason this is a named cause rather than a boolean.
+ const decision = await withAnswer(
+ JSON.stringify({ agentId: "risk-analyst", confidence: 0.2 }),
+ ).route("anything", ROSTER, "general-assistant");
+ expect(decision.undecided).toBe("unconfident");
+ });
+
+ test("a roster of one is named as one-candidate, and asks nothing", async () => {
+ const decision = await throwing().route(
+ "anything",
+ [ROSTER[0] as RoutingCandidate],
+ "general-assistant",
+ );
+ expect(decision.undecided).toBe("one-candidate");
+ });
+
+ test("the reach answer does not hide that the router never answered", async () => {
+ /*
+ * THE CASE THIS EXISTS FOR. Landing on the only coworker that can reach Google Drive is a good
+ * outcome, and it says nothing about whether the router was up. Before this the reach sentence
+ * replaced the failure entirely, so a deployment whose router had been down for a week produced
+ * rows that read exactly like reach-based routing working as intended.
+ */
+ const reaching: RoutingCandidate[] = [
+ { ...(ROSTER[0] as RoutingCandidate) },
+ {
+ ...(ROSTER[1] as RoutingCandidate),
+ reaches: ["google-drive"],
+ },
+ ];
+
+ const decision = await throwing().route(
+ "find the PRD in google drive",
+ reaching,
+ "general-assistant",
+ );
+
+ // Still routed by reach, and still a fallback — both unchanged.
+ expect(decision.agentId).toBe("knowledge");
+ expect(decision.reason).toContain("google-drive");
+ expect(decision.fallback).toBe(true);
+ // And the router failure survives it.
+ expect(decision.undecided).toBe("unreachable");
+ });
+
+ test("reach after a low-confidence answer says unconfident, not unreachable", async () => {
+ // The two are told apart on the reach path as well, or the count is wrong wherever reach fires.
+ const reaching: RoutingCandidate[] = [
+ { ...(ROSTER[0] as RoutingCandidate) },
+ {
+ ...(ROSTER[1] as RoutingCandidate),
+ reaches: ["google-drive"],
+ },
+ ];
+
+ const decision = await withAnswer(
+ JSON.stringify({ agentId: "knowledge", confidence: 0.1 }),
+ ).route("find the PRD in google drive", reaching, "general-assistant");
+
+ expect(decision.agentId).toBe("knowledge");
+ expect(decision.undecided).toBe("unconfident");
+ });
+});
diff --git a/server/tests/routing-routes.test.ts b/server/tests/routing-routes.test.ts
index 974996b2..b974d136 100644
--- a/server/tests/routing-routes.test.ts
+++ b/server/tests/routing-routes.test.ts
@@ -4,7 +4,7 @@ import { Hono } from "hono";
import type { AgentProfileStore } from "../src/agents/profile-store";
import type { AuditStore } from "../src/audit";
import type { AppVariables } from "../src/auth/guards";
-import type { IntentRouter } from "../src/routing/classify";
+import type { IntentRouter, RoutingUndecided } from "../src/routing/classify";
import { createRoutingRoutes } from "../src/routing/routes";
/**
@@ -47,7 +47,7 @@ type Recorded = {
payload: Record<string, unknown>;
};
-function app(options: { routed?: string } = {}) {
+function app(options: { routed?: string; undecided?: RoutingUndecided } = {}) {
const written: Recorded[] = [];
/** Every call the router was asked to make, so "never asked" is an assertion and not a hope. */
const asked: string[] = [];
@@ -72,7 +72,8 @@ function app(options: { routed?: string } = {}) {
agentId: chosen,
name: ROSTER.find((a) => a.id === chosen)?.name ?? chosen,
reason: "matches what it is for",
- fallback: false,
+ fallback: options.undecided !== undefined,
+ undecided: options.undecided ?? null,
};
},
} as unknown as IntentRouter;
@@ -185,3 +186,47 @@ describe("recording which coworker a message went to", () => {
expect(asked).toEqual(["hello"]);
});
});
+
+/**
+ * Why it was not decided, on the row rather than only in the sentence.
+ *
+ * The reason is prose for a person. This is the field a deployment counts, and counting is the point:
+ * a router that has been unreachable for a week is invisible until somebody can ask how often.
+ */
+describe("recording why a message was not routed", () => {
+ test("an unreachable router is on the row, not only in the sentence", async () => {
+ const { server, written } = app({ undecided: "unreachable" });
+
+ await post(server, { text: "what is our PTO policy" });
+
+ expect(written[0]?.payload).toMatchObject({
+ chosen: "knowledge",
+ fallback: true,
+ undecided: "unreachable",
+ });
+ });
+
+ test("a decided routing records no cause", async () => {
+ const { server, written } = app();
+
+ await post(server, { text: "what is our PTO policy" });
+
+ expect(written[0]?.payload).toMatchObject({
+ fallback: false,
+ undecided: null,
+ });
+ });
+
+ test("a coworker the person named records no cause either", async () => {
+ // Nothing was left to the router, so there is nothing about it to have failed. Recording a cause
+ // here would count a person's own choice as a routing failure.
+ const { server, written } = app();
+
+ await post(server, { text: "hello", agentId: "risk-analyst" });
+
+ expect(written[0]?.payload).toMatchObject({
+ viaMention: true,
+ undecided: null,
+ });
+ });
+});