Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.
Say Blocked about a refusal the audit page had been calling Allowed (#302)
kevin91d8ae8f0cd5
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7a28adc4..a5354d72 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,15 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.## Unreleased+### The audit page no longer says "Allowed" about six kinds of refusal++A hop one Bot was not allowed to make, an endpoint this deployment would not dial, a rotation the+vault refused and a sign-in it turned away were all drawn as **Allowed**, in the muted colour every+ordinary row uses, and none of them appeared under **Blocked**. The same for a hop that ran out of+attempts and a question that reached nobody, which are "Did not happen" rather than allowed. The+page recognised six refusal types and the six added since were never added to it. Refusals now read+as refusals, the two saved views are built from the same lists the rows are labelled from, and a+refusal added later is added in one place or in none.### A conversation deleted while a server was reconnecting no longer lingers on the screenAnnouncements between servers travel as Postgres notifications, which reach whoever is subscribed atdiff --git a/app/src/lib/audit/outcome.ts b/app/src/lib/audit/outcome.tsnew file mode 100644index 00000000..be2af9e1--- /dev/null+++ b/app/src/lib/audit/outcome.ts@@ -0,0 +1,83 @@+/**+ * Whether a row in the trail is something this deployment turned away, something that was allowed+ * and then did not happen, or something that went through.+ *+ * THE TWO PLACES THAT DECIDE THIS HAVE TO BE ONE PLACE. The audit page asks the question twice: once+ * to colour and label a row, and once to build the `eventType` list behind the `Blocked` and+ * `Did not happen` saved views. They were two hand-written lists, and they had already drifted — a+ * refusal missing from the first is drawn as "Allowed", a refusal missing from the second is+ * missing from the view somebody clicks to ask what this deployment refused, and neither omission+ * says anything. So the lists live here, the page derives both from them, and a new refusal is added+ * in one place or in none.+ *+ * WHY "ALLOWED" IS THE ONE WRONG ANSWER. The page falls back to it for anything it does not+ * recognise, which is the right default for the many rows that are neither a refusal nor a failure —+ * a credential saved, a component published, a person's role changed. For a refusal it is not a+ * missing label, it is the opposite of what happened, on the screen an administrator opens to find+ * out what happened. A trail that is confidently wrong is worse than a silent one.+ */++/**+ * Refused: this deployment declined, and nothing was attempted.+ *+ * `mcp.callback_refused`, `routines.dispatch_refused` and `session.refused` are here even though no+ * policy judged anybody and, in the last two, no Bot was involved at all. Somebody filtering for+ * "what did this deployment turn away" wants them, and for each of the three this row is the only+ * evidence anywhere that anything was attempted: the wire answers all of them with the same opaque+ * refusal on purpose.+ */+export const REFUSED_EVENT_TYPES = [+ "computer.action_refused",+ "component.refused",+ "component.function_refused",+ "mcp.call_rejected",+ "mcp.callback_refused",+ "routines.dispatch_refused",+ /*+ * A hop one Bot was not allowed to make. `server/src/audit.ts` calls this "the more important of+ * the pair": a hop that happened is visible in the transcript anyway, and a hop that was refused+ * is invisible everywhere else.+ */+ "agent.handoff_refused",+ /** An endpoint a stored agent tried to reach and the deployment would not dial. */+ "agent.dial_refused",+ /** A rotation aimed at a key the credential does not belong to, or at a revoked one. */+ "credential.rotation_refused",+ /** A revoked person still holding a bookmark, or an address outside the deployment. */+ "session.refused",+] as const;++/**+ * Did not happen: nothing was refused, and nothing came of it either.+ *+ * Its own family because the difference is what somebody came to the row to find out. A boundary+ * holding and a Bot that was asked and never answered are different faults with different fixes, and+ * only one of them is the deployment working as configured.+ */+export const DID_NOT_HAPPEN_EVENT_TYPES = [+ "computer.action_failed",+ "agent.stream_stalled",+ /** A hop that was accepted, ran out of attempts, and never became the other Bot's turn. */+ "agent.handoff_failed",+ /** A question that reached nobody: the Bot stopped, and the person was never asked. */+ "agent.escalation_failed",+] as const;++export type AuditOutcome = "refused" | "did-not-happen" | "allowed";++const REFUSED = new Set<string>(REFUSED_EVENT_TYPES);+const DID_NOT_HAPPEN = new Set<string>(DID_NOT_HAPPEN_EVENT_TYPES);++/** What kind of thing this row is, for the label and the colour it is drawn in. */+export function outcomeOf(eventType: string): AuditOutcome {+ if (REFUSED.has(eventType)) return "refused";+ if (DID_NOT_HAPPEN.has(eventType)) return "did-not-happen";+ return "allowed";+}++/** The `eventType` query one of the saved views filters by. */+export function eventTypeFilter(+ types: readonly string[],+): `?eventType=${string}` {+ return `?eventType=${types.join(",")}`;+}diff --git a/app/src/routes/_authed/admin/audit.tsx b/app/src/routes/_authed/admin/audit.tsxindex 403294ad..18b4d98e 100644--- a/app/src/routes/_authed/admin/audit.tsx+++ b/app/src/routes/_authed/admin/audit.tsx@@ -9,6 +9,12 @@ import {} from "@/components/layout/page-shell";import { Button } from "@/components/ui/button";import { useBotNames } from "@/lib/agents/bot-names";+import {+ DID_NOT_HAPPEN_EVENT_TYPES,+ eventTypeFilter,+ outcomeOf,+ REFUSED_EVENT_TYPES,+} from "@/lib/audit/outcome";import { auditEventsQueryOptions } from "@/lib/audit/queries";import { silenceOf } from "@/lib/audit/silence";@@ -30,32 +36,21 @@ type AuditEvent = {createdAt: string;};+/*+ * The saved views, built from the same lists the row label and colour are decided by.+ *+ * Written out by hand here and again below, they had already drifted: a refusal on one list and not+ * the other is a row drawn as "Allowed" or a row missing from the view somebody clicks to ask what+ * this deployment refused. See `@/lib/audit/outcome`, which is now the only place either question is+ * answered.+ */const FILTERS = [{ label: "Everything", search: "" },{ label: "Computer actions", search: "?eventType=computer.action_allowed" },- {- label: "Blocked",- /*- * Include every refusal family, not only browser policy refusals.- *- * `mcp.callback_refused` is here because it is a refusal, even though nothing about a Bot was- * judged: a caller could not prove which Bot it was. Somebody filtering for what this deployment- * turned away wants that in the list, and it is the one refusal with no policy behind it, so- * leaving it out would hide the only evidence that anything was attempted.- *- * `routines.dispatch_refused` is the same shape one boundary over: the worker, not a Bot, and a- * stale or missing secret rather than a policy decision. The same reasoning that put- * `mcp.callback_refused` here applies unchanged — nobody was judged, something was still turned- * away, and the saved view a person clicks for "what did this deployment block" should show it.- */- search:- "?eventType=computer.action_refused,mcp.call_rejected,mcp.callback_refused,component.refused,component.function_refused,routines.dispatch_refused",- },+ { label: "Blocked", search: eventTypeFilter(REFUSED_EVENT_TYPES) },{label: "Did not happen",- // A stalled stream belongs here. It is the same complaint as an action that was allowed and then- // did not take: nothing was refused, and nothing came of it either.- search: "?eventType=computer.action_failed,agent.stream_stalled",+ search: eventTypeFilter(DID_NOT_HAPPEN_EVENT_TYPES),},] as const;@@ -148,19 +143,8 @@ function Row({| { role?: string; name?: string }| string| undefined;- const refused =- event.eventType === "computer.action_refused" ||- event.eventType === "component.refused" ||- event.eventType === "component.function_refused" ||- event.eventType === "mcp.call_rejected" ||- /*- * A caller that could not prove which Bot it was. Refused like the others, and it has to read- * that way here: the fallback below calls anything it does not recognise "Allowed", which for a- * refusal is the one wrong answer. A trail that is confidently wrong is worse than a silent one.- */- event.eventType === "mcp.callback_refused" ||- // The worker turned away at the door, same reasoning as the caller above.- event.eventType === "routines.dispatch_refused";+ const outcome = outcomeOf(event.eventType);+ const refused = outcome === "refused";const stalled = event.eventType === "agent.stream_stalled";/** Three different things, and the difference is what somebody comes to this row to find out.@@ -181,7 +165,7 @@ function Row({// Allowed by policy but not carried out. A stalled turn belongs in the same family: the Bot was// asked and the answer never arrived. Colour is how this table is read, and a row left in the// muted foreground reads as "Allowed", which a turn nobody ever got an answer to was not.- const failed = event.eventType === "computer.action_failed" || stalled;+ const failed = outcome === "did-not-happen";const silence = stalled ? silenceOf(payload) : null;return (diff --git a/app/tests/audit-outcome.test.ts b/app/tests/audit-outcome.test.tsnew file mode 100644index 00000000..8aa1e591--- /dev/null+++ b/app/tests/audit-outcome.test.ts@@ -0,0 +1,110 @@+import { describe, expect, test } from "bun:test";+import {+ DID_NOT_HAPPEN_EVENT_TYPES,+ eventTypeFilter,+ outcomeOf,+ REFUSED_EVENT_TYPES,+} from "../src/lib/audit/outcome";++/**+ * What the audit page says about a row, which for a refusal has exactly one wrong answer.+ *+ * The page falls back to "Allowed" for anything it does not recognise, which is right for the many+ * rows that are neither a refusal nor a failure. For a refusal it is the opposite of what happened,+ * on the screen somebody opens to find out what happened.+ */++describe("what the trail says a row was", () => {+ test("names a hop a boundary refused as a refusal", () => {+ expect(outcomeOf("agent.handoff_refused")).toBe("refused");+ });++ test("names a sign-in turned away as a refusal", () => {+ expect(outcomeOf("session.refused")).toBe("refused");+ });++ test("names a rotation the vault refused as a refusal", () => {+ expect(outcomeOf("credential.rotation_refused")).toBe("refused");+ });++ test("names an endpoint this deployment would not dial as a refusal", () => {+ expect(outcomeOf("agent.dial_refused")).toBe("refused");+ });++ test("keeps the refusals that were already named", () => {+ for (const eventType of [+ "computer.action_refused",+ "component.refused",+ "component.function_refused",+ "mcp.call_rejected",+ "mcp.callback_refused",+ "routines.dispatch_refused",+ ]) {+ expect(outcomeOf(eventType)).toBe("refused");+ }+ });++ test("tells a hop that never landed from one that was refused", () => {+ // Nothing was refused: the hop was accepted, tried, and ran out of attempts.+ expect(outcomeOf("agent.handoff_failed")).toBe("did-not-happen");+ // And a question that reached nobody, which nothing else anywhere records.+ expect(outcomeOf("agent.escalation_failed")).toBe("did-not-happen");+ expect(outcomeOf("computer.action_failed")).toBe("did-not-happen");+ expect(outcomeOf("agent.stream_stalled")).toBe("did-not-happen");+ });++ test("still calls something that went through allowed", () => {+ for (const eventType of [+ "computer.action_allowed",+ "mcp.call_succeeded",+ "agent.handoff_delivered",+ "agent.escalated",+ "credential.created",+ "session.signed_in",+ ]) {+ expect(outcomeOf(eventType)).toBe("allowed");+ }+ });++ test("does not call an unknown row a refusal", () => {+ // The fallback has to stay open: a row type this build has never heard of is not a refusal, and+ // claiming otherwise would be the same fault in the other direction.+ expect(outcomeOf("something.nobody.has.written.yet")).toBe("allowed");+ });+});++describe("the saved views ask the same question the rows do", () => {+ /*+ * The drift this exists to stop. Two hand-written lists meant a refusal could be drawn correctly+ * on the row and be absent from the view somebody clicks to ask what this deployment refused —+ * which is the harder failure to notice, because the view is not empty, it is just short.+ */+ test("Blocked filters by every event type drawn as a refusal", () => {+ const filtered = eventTypeFilter(REFUSED_EVENT_TYPES)+ .replace("?eventType=", "")+ .split(",");++ expect(filtered).toEqual([...REFUSED_EVENT_TYPES]);+ for (const eventType of filtered) {+ expect(outcomeOf(eventType)).toBe("refused");+ }+ });++ test("Did not happen filters by every event type drawn that way", () => {+ const filtered = eventTypeFilter(DID_NOT_HAPPEN_EVENT_TYPES)+ .replace("?eventType=", "")+ .split(",");++ expect(filtered).toEqual([...DID_NOT_HAPPEN_EVENT_TYPES]);+ for (const eventType of filtered) {+ expect(outcomeOf(eventType)).toBe("did-not-happen");+ }+ });++ test("no event type is in both families", () => {+ const refused = new Set<string>(REFUSED_EVENT_TYPES);+ for (const eventType of DID_NOT_HAPPEN_EVENT_TYPES) {+ expect(refused.has(eventType)).toBe(false);+ }+ });+});