Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

[3/7] test(eve): add rules about how HITL prompts should render to the conformance suite (#4170)

Ben Pankow2f9dddbbd871
diff --git a/.oxfmtrc.json b/.oxfmtrc.json
index cc447abb6..8c6f3ce87 100644
--- a/.oxfmtrc.json
+++ b/.oxfmtrc.json
@@ -2,6 +2,7 @@
"$schema": "./node_modules/oxfmt/configuration_schema.json",
"ignorePatterns": [
"packages/eve/CHANGELOG.md",
+ "packages/eve/src/internal/testing/channel-conformance/MATRIX.md",
"apps/docs/registry.json",
"packages/eve/src/setup/scaffold/create/web-template.ts"
]
diff --git a/packages/eve/src/internal/testing/channel-conformance/MATRIX.md b/packages/eve/src/internal/testing/channel-conformance/MATRIX.md
new file mode 100644
index 000000000..32d99c7ba
--- /dev/null
+++ b/packages/eve/src/internal/testing/channel-conformance/MATRIX.md
@@ -0,0 +1,85 @@
+# HITL conformance matrix
+
+<!-- Generated from conformance.ts by matrix.test.ts. Do not edit by hand. -->
+
+Each channel and client against each rule in [`contract.ts`](./contract.ts), as
+[`conformance.ts`](./conformance.ts) records it. The suite holds every cell to this
+table: a ✅ cell must pass, and a ❌ cell passes only while the rule fails with its
+recorded symptom. Regenerate it after changing either file:
+
+```sh
+pnpm --filter eve exec vitest run --config vitest.unit.config.ts channel-conformance/matrix -u
+```
+
+✅ passes · ❌ broken · — not supported (the platform lacks a capability the rule
+needs, or the client declines it below)
+
+| Rule | `chat-sdk` | `chat-sdk-text` | `discord` | `github` | `linear` | `linq` | `slack` | `teams` | `telegram` | `tui` | `twilio` |
+| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| a rendered question shows every option a person can choose | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| pressing a rendered option answers the pending question with that option | ✅ | — | ✅ | — | — | — | ✅ | ✅ | ✅ | ✅ | — |
+| a text reply matching an option answers the only pending question | ✅ | ✅ | — | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| a text reply that matches no option answers the question with the person's words | ✅ | ✅ | — | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| a text reply answers an open-ended question with the person's words | ✅ | ✅ | — | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| pressing an option of an answered question sends it to the agent as new input | ✅ | — | ✅ | — | — | — | ✅ | ✅ | ✅ | — | — |
+| pressing options of two pending questions answers each with its own option | ✅ | — | ✅ | — | — | — | ✅ | ✅ | ✅ | ✅ | — |
+| a text reply matching an option does not answer either of two pending questions | ✅ | ✅ | — | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| a tool approval shows a choice to approve and one to cancel | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| pressing Approve runs the gated tool | ✅ | — | ✅ | — | — | — | ✅ | ✅ | ✅ | ✅ | — |
+| pressing Approve twice runs the gated tool once | ✅ | — | ✅ | — | — | — | ✅ | ✅ | ✅ | ✅ | — |
+| pressing Approve on one of two pending approvals runs only that tool | ✅ | — | ✅ | — | — | — | ✅ | ✅ | ✅ | ✅ | — |
+| answering an approval and a question pending together settles both | ✅ | — | ✅ | — | — | — | ✅ | ✅ | ✅ | ✅ | — |
+| pressing Cancel stops the gated tool without running it | ✅ | — | ✅ | — | — | — | ✅ | ✅ | ✅ | ✅ | — |
+| a text reply of approve runs the gated tool | ✅ | ✅ | — | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — | ✅ |
+| a text reply of cancel stops the gated tool without running it | ✅ | ✅ | — | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | — | ✅ |
+| pressing an option clears the question's buttons | ❌ | — | ❌ | — | — | — | ✅ | ❌ | ❌ | ✅ | — |
+| answering a question by text clears its buttons | ❌ | — | — | — | — | — | ❌ | ❌ | ❌ | ✅ | — |
+| pressing an option names who answered on the question | ❌ | — | ❌ | — | — | — | ✅ | ❌ | ❌ | — | — |
+| answering a question by text names who answered on the question | ❌ | — | — | — | — | — | ❌ | ❌ | ❌ | — | — |
+| pressing Approve clears the approval's buttons | ❌ | — | ❌ | — | — | — | ✅ | ✅ | ❌ | ✅ | — |
+| approving by text clears the approval's buttons | ❌ | — | — | — | — | — | ✅ | ❌ | ❌ | — | — |
+| pressing Approve names who approved on the approval | ❌ | — | ❌ | — | — | — | ✅ | ✅ | ❌ | — | — |
+| approving by text names who approved on the approval | ❌ | — | — | — | — | — | ❌ | ❌ | ❌ | — | — |
+
+## Broken
+
+- **chat-sdk**, pressing an option clears the question's buttons: the bridge never edits an answered prompt
+- **chat-sdk**, answering a question by text clears its buttons: the bridge never edits an answered prompt
+- **chat-sdk**, pressing an option names who answered on the question: the bridge never edits an answered prompt
+- **chat-sdk**, answering a question by text names who answered on the question: the bridge never edits an answered prompt
+- **chat-sdk**, pressing Approve clears the approval's buttons: the bridge never edits an answered prompt
+- **chat-sdk**, approving by text clears the approval's buttons: the bridge never edits an answered prompt
+- **chat-sdk**, pressing Approve names who approved on the approval: the bridge never edits an answered prompt
+- **chat-sdk**, approving by text names who approved on the approval: the bridge never edits an answered prompt
+- **discord**, pressing an option clears the question's buttons: a press gets a deferred update and the message is never edited
+- **discord**, pressing an option names who answered on the question: a press gets a deferred update and the message is never edited
+- **discord**, pressing Approve clears the approval's buttons: a press gets a deferred update and the message is never edited
+- **discord**, pressing Approve names who approved on the approval: a press gets a deferred update and the message is never edited
+- **slack**, answering a question by text clears its buttons: only the button interaction handler edits a question; a typed answer leaves it
+- **slack**, answering a question by text names who answered on the question: only the button interaction handler edits a question; a typed answer leaves it
+- **slack**, approving by text names who approved on the approval: the card loses its buttons after a typed approval but doesn't say who approved
+- **teams**, pressing an option clears the question's buttons: only a pressed approval card is recorded for editing; questions and typed approvals are not
+- **teams**, answering a question by text clears its buttons: only a pressed approval card is recorded for editing; questions and typed approvals are not
+- **teams**, pressing an option names who answered on the question: only a pressed approval card is recorded for editing; questions and typed approvals are not
+- **teams**, answering a question by text names who answered on the question: only a pressed approval card is recorded for editing; questions and typed approvals are not
+- **teams**, approving by text clears the approval's buttons: only a pressed approval card is recorded for editing; questions and typed approvals are not
+- **teams**, approving by text names who approved on the approval: only a pressed approval card is recorded for editing; questions and typed approvals are not
+- **telegram**, pressing an option clears the question's buttons: nothing edits an answered prompt; a press only answers the callback query
+- **telegram**, answering a question by text clears its buttons: nothing edits an answered prompt; a press only answers the callback query
+- **telegram**, pressing an option names who answered on the question: nothing edits an answered prompt; a press only answers the callback query
+- **telegram**, answering a question by text names who answered on the question: nothing edits an answered prompt; a press only answers the callback query
+- **telegram**, pressing Approve clears the approval's buttons: nothing edits an answered prompt; a press only answers the callback query
+- **telegram**, approving by text clears the approval's buttons: nothing edits an answered prompt; a press only answers the callback query
+- **telegram**, pressing Approve names who approved on the approval: nothing edits an answered prompt; a press only answers the callback query
+- **telegram**, approving by text names who approved on the approval: nothing edits an answered prompt; a press only answers the callback query
+
+## Declined
+
+- **tui**, pressing an option of an answered question sends it to the agent as new input: an answered question's drawer closes, so nothing is left to press
+- **tui**, a text reply of approve runs the gated tool: the approval drawer holds the keyboard; a person answers it with y or n
+- **tui**, a text reply of cancel stops the gated tool without running it: the approval drawer holds the keyboard; a person answers it with y or n
+- **tui**, pressing an option names who answered on the question: one person answers at their own terminal; there's nobody else to tell
+- **tui**, answering a question by text names who answered on the question: one person answers at their own terminal; there's nobody else to tell
+- **tui**, approving by text clears the approval's buttons: the approval drawer holds the keyboard; a person answers it with y or n
+- **tui**, pressing Approve names who approved on the approval: one person answers at their own terminal; there's nobody else to tell
+- **tui**, approving by text names who approved on the approval: the approval drawer holds the keyboard; a person answers it with y or n
diff --git a/packages/eve/src/internal/testing/channel-conformance/chat-sdk-driver.ts b/packages/eve/src/internal/testing/channel-conformance/chat-sdk-driver.ts
index 8bcaea2b4..f35b16ffa 100644
--- a/packages/eve/src/internal/testing/channel-conformance/chat-sdk-driver.ts
+++ b/packages/eve/src/internal/testing/channel-conformance/chat-sdk-driver.ts
@@ -32,7 +32,32 @@ interface CardNode {
type Inbound =
| { readonly kind: "message"; readonly text: string }
- | { readonly kind: "action"; readonly actionId: string; readonly value?: string };
+ | {
+ readonly kind: "action";
+ readonly actionId: string;
+ /** The posted message holding the pressed button. */
+ readonly messageId: string;
+ readonly value?: string;
+ };
+
+interface PressHandle {
+ readonly button: CardNode;
+ readonly messageId: string;
+}
+
+function messageIdOf(call: PlatformCall): string {
+ return (call.response as { readonly id: string }).id;
+}
+
+/** A card's buttons, or only those under `prompt` when the card holds several requests. */
+function buttonsOf(call: PlatformCall, prompt?: string): { handle: PressHandle; label: string }[] {
+ const card = cardOf(call.body as AdapterPostableMessage);
+ if (card === undefined) return [];
+ const messageId = messageIdOf(call);
+ return nodes(prompt === undefined ? card : requestSection(card, prompt))
+ .filter((node) => node.type === "button" && node.id !== undefined)
+ .map((button) => ({ handle: { button, messageId }, label: button.label ?? "" }));
+}
/**
* Drives `chatSdkChannel` with a card-capable direct-message adapter: one
@@ -49,13 +74,26 @@ export function chatSdkDriver(): ChannelDriver {
if (!isPost(call)) return undefined;
const card = cardOf(call.body as AdapterPostableMessage);
if (card === undefined || !texts(card).includes(prompt)) return undefined;
- return nodes(requestSection(card, prompt))
- .filter((node) => node.type === "button" && node.id !== undefined)
- .map((button) => ({ handle: button, label: button.label ?? "" }));
+ return buttonsOf(call, prompt);
},
+ shownMessage(call) {
+ if (!isPost(call)) return undefined;
+ const card = cardOf(call.body as AdapterPostableMessage);
+ return {
+ id: messageIdOf(call),
+ options: buttonsOf(call),
+ text: card === undefined ? (driver.postedText(call) ?? "") : texts(card),
+ };
+ },
+ personShownAs: [PERSON.fullName, PERSON.userName],
press(option) {
- const button = option.handle as CardNode;
- return driver.inbound({ actionId: button.id!, kind: "action", value: button.value });
+ const { button, messageId } = option.handle as PressHandle;
+ return driver.inbound({
+ actionId: button.id!,
+ kind: "action",
+ messageId,
+ value: button.value,
+ });
},
};
}
@@ -171,7 +209,7 @@ function fakeAdapter(
{
actionId: body.actionId,
adapter,
- messageId: id,
+ messageId: body.messageId,
raw: body,
threadId,
user: PERSON,
diff --git a/packages/eve/src/internal/testing/channel-conformance/conformance.ts b/packages/eve/src/internal/testing/channel-conformance/conformance.ts
index 1691f61e0..5152f8d2c 100644
--- a/packages/eve/src/internal/testing/channel-conformance/conformance.ts
+++ b/packages/eve/src/internal/testing/channel-conformance/conformance.ts
@@ -26,6 +26,44 @@ export interface BrokenCell {
readonly symptom: RegExp;
}
+/** Rules that check how an answered prompt's message changes, by prompt kind and how it was answered. */
+const answeredPromptRules = {
+ approvalPress: [
+ "pressing Approve clears the approval's buttons",
+ "pressing Approve names who approved on the approval",
+ ],
+ approvalText: [
+ "approving by text clears the approval's buttons",
+ "approving by text names who approved on the approval",
+ ],
+ questionPress: [
+ "pressing an option clears the question's buttons",
+ "pressing an option names who answered on the question",
+ ],
+ questionText: [
+ "answering a question by text clears its buttons",
+ "answering a question by text names who answered on the question",
+ ],
+} as const satisfies Record<string, readonly HitlRule[]>;
+
+/** Answered prompts in `groups` are never edited, so they keep their buttons and never say who answered. */
+function staleAnsweredPrompts(
+ reason: string,
+ groups: readonly (keyof typeof answeredPromptRules)[],
+): Partial<Record<HitlRule, BrokenCell>> {
+ return Object.fromEntries(
+ groups
+ .flatMap((group) => answeredPromptRules[group])
+ .map((rule) => [
+ rule,
+ {
+ reason,
+ symptom: /the answered prompt (still offers \[".+\]|never names who answered)/,
+ },
+ ]),
+ );
+}
+
interface ConformanceChannel {
readonly driver: () => ChannelDriver | ClientDriver;
readonly broken?: Partial<Record<HitlRule, BrokenCell>>;
@@ -35,6 +73,7 @@ interface ConformanceChannel {
const TUI_TYPED_APPROVAL =
"the approval drawer holds the keyboard; a person answers it with y or n";
+const TUI_SINGLE_PERSON = "one person answers at their own terminal; there's nobody else to tell";
/**
* Every first-party channel's and client's place in the HITL contract, keyed by
@@ -49,14 +88,63 @@ const TUI_TYPED_APPROVAL =
* failure (such as harness breakage) both turn it red.
*/
const hitlConformance = {
- "chat-sdk": [{ driver: chatSdkDriver }, { driver: chatSdkTextDriver }],
- discord: [{ driver: discordDriver }],
+ "chat-sdk": [
+ {
+ driver: chatSdkDriver,
+ broken: staleAnsweredPrompts("the bridge never edits an answered prompt", [
+ "approvalPress",
+ "approvalText",
+ "questionPress",
+ "questionText",
+ ]),
+ },
+ { driver: chatSdkTextDriver },
+ ],
+ discord: [
+ {
+ driver: discordDriver,
+ broken: staleAnsweredPrompts(
+ "a press gets a deferred update and the message is never edited",
+ ["approvalPress", "questionPress"],
+ ),
+ },
+ ],
github: [{ driver: githubDriver }],
linear: [{ driver: linearDriver }],
linq: [{ driver: linqDriver }],
- slack: [{ driver: slackDriver }],
- teams: [{ driver: teamsDriver }],
- telegram: [{ driver: telegramDriver }],
+ slack: [
+ {
+ driver: slackDriver,
+ broken: {
+ ...staleAnsweredPrompts(
+ "only the button interaction handler edits a question; a typed answer leaves it",
+ ["questionText"],
+ ),
+ "approving by text names who approved on the approval": {
+ reason: "the card loses its buttons after a typed approval but doesn't say who approved",
+ symptom: /the answered prompt never names who answered/,
+ },
+ },
+ },
+ ],
+ teams: [
+ {
+ driver: teamsDriver,
+ broken: staleAnsweredPrompts(
+ "only a pressed approval card is recorded for editing; questions and typed approvals are not",
+ ["approvalText", "questionPress", "questionText"],
+ ),
+ },
+ ],
+ telegram: [
+ {
+ driver: telegramDriver,
+ broken: staleAnsweredPrompts(
+ "nothing edits an answered prompt; a press only answers the callback query",
+ ["approvalPress", "approvalText", "questionPress", "questionText"],
+ ),
+ },
+ ],
tui: [
{
driver: tuiDriver,
@@ -65,6 +153,11 @@ const hitlConformance = {
"an answered question's drawer closes, so nothing is left to press",
"a text reply of approve runs the gated tool": TUI_TYPED_APPROVAL,
"a text reply of cancel stops the gated tool without running it": TUI_TYPED_APPROVAL,
+ "approving by text clears the approval's buttons": TUI_TYPED_APPROVAL,
+ "approving by text names who approved on the approval": TUI_TYPED_APPROVAL,
+ "pressing Approve names who approved on the approval": TUI_SINGLE_PERSON,
+ "pressing an option names who answered on the question": TUI_SINGLE_PERSON,
+ "answering a question by text names who answered on the question": TUI_SINGLE_PERSON,
},
},
],
@@ -77,6 +170,23 @@ const hitlConformance = {
*/
const BROKEN_WAIT_TIMEOUT_MS = 3_000;
+/** How one channel or client stands on one rule. */
+type Cell =
+ | { readonly kind: "pass" }
+ | { readonly kind: "unsupported"; readonly reason?: string }
+ | { readonly kind: "broken"; readonly broken: BrokenCell };
+
+function cellOf(entry: ConformanceChannel, rule: (typeof hitlContract)[number]): Cell {
+ const { capabilities } = entry.driver();
+ if (!rule.requires.every((capability) => capabilities.includes(capability))) {
+ return { kind: "unsupported" };
+ }
+ const declined = entry.unsupported?.[rule.rule];
+ if (declined !== undefined) return { kind: "unsupported", reason: declined };
+ const broken = entry.broken?.[rule.rule];
+ return broken === undefined ? { kind: "pass" } : { kind: "broken", broken };
+}
+
/**
* Declares the HITL contract cells for one channel directory. Each channel gets
* its own test file because conversations can't overlap within a process, and
@@ -84,29 +194,25 @@ const BROKEN_WAIT_TIMEOUT_MS = 3_000;
*/
export function describeHitlConformance(channel: keyof typeof hitlConformance): void {
const entries: readonly ConformanceChannel[] = hitlConformance[channel];
- describe.each(entries.map((entry) => ({ ...entry, name: entry.driver().name })))(
+ describe.each(entries.map((entry) => ({ entry, name: entry.driver().name })))(
"$name HITL contract",
- ({ driver, broken, unsupported }) => {
- const { capabilities } = driver();
+ ({ entry }) => {
for (const rule of hitlContract) {
- const supported = rule.requires.every((capability) => capabilities.includes(capability));
- const known = broken?.[rule.rule];
- const declined = unsupported?.[rule.rule];
- if (!supported) {
- it.skip(`${rule.rule} (not supported)`, () => {});
- } else if (declined !== undefined) {
- it.skip(`${rule.rule} (not supported: ${declined})`, () => {});
- } else if (known === undefined) {
- it(rule.rule, () => withChannelConversation(driver(), (c) => rule.run(c)), 60_000);
+ const cell = cellOf(entry, rule);
+ const run = (options?: { readonly waitTimeoutMs: number }) =>
+ withChannelConversation(entry.driver(), (c) => rule.run(c), options);
+ if (cell.kind === "unsupported") {
+ const why = cell.reason === undefined ? "" : `: ${cell.reason}`;
+ it.skip(`${rule.rule} (not supported${why})`, () => {});
+ } else if (cell.kind === "pass") {
+ it(rule.rule, () => run(), 60_000);
} else {
it(
- `${rule.rule} (broken: ${known.reason})`,
+ `${rule.rule} (broken: ${cell.broken.reason})`,
() =>
- expect(
- withChannelConversation(driver(), (c) => rule.run(c), {
- waitTimeoutMs: BROKEN_WAIT_TIMEOUT_MS,
- }),
- ).rejects.toThrow(known.symptom),
+ expect(run({ waitTimeoutMs: BROKEN_WAIT_TIMEOUT_MS })).rejects.toThrow(
+ cell.broken.symptom,
+ ),
60_000,
);
}
@@ -114,3 +220,64 @@ export function describeHitlConformance(channel: keyof typeof hitlConformance):
},
);
}
+
+const MATRIX_SYMBOLS = { broken: "❌", pass: "✅", unsupported: "—" } as const;
+
+/**
+ * Renders every channel's and client's cell for every rule as Markdown. The
+ * suite holds each cell to what this table says, so the rendered matrix is
+ * current whenever the suite passes.
+ */
+export function renderHitlConformanceMatrix(): string {
+ const entries = Object.values(hitlConformance).flatMap(
+ (group): readonly ConformanceChannel[] => group,
+ );
+ const names = entries.map((entry) => entry.driver().name);
+ const escape = (text: string) => text.replaceAll("|", "\\|");
+ const row = (cells: readonly string[]) => `| ${cells.join(" | ")} |`;
+ const notes = (kind: "broken" | "unsupported") =>
+ entries.flatMap((entry, index) =>
+ hitlContract.flatMap((rule) => {
+ const cell = cellOf(entry, rule);
+ if (cell.kind !== kind) return [];
+ const reason = cell.kind === "broken" ? cell.broken.reason : cell.reason;
+ return reason === undefined ? [] : [`- **${names[index]}**, ${rule.rule}: ${reason}`];
+ }),
+ );
+ return [
+ "# HITL conformance matrix",
+ "",
+ "<!-- Generated from conformance.ts by matrix.test.ts. Do not edit by hand. -->",
+ "",
+ "Each channel and client against each rule in [`contract.ts`](./contract.ts), as",
+ "[`conformance.ts`](./conformance.ts) records it. The suite holds every cell to this",
+ "table: a ✅ cell must pass, and a ❌ cell passes only while the rule fails with its",
+ "recorded symptom. Regenerate it after changing either file:",
+ "",
+ "```sh",
+ "pnpm --filter eve exec vitest run --config vitest.unit.config.ts channel-conformance/matrix -u",
+ "```",
+ "",
+ "✅ passes · ❌ broken · — not supported (the platform lacks a capability the rule",
+ "needs, or the client declines it below)",
+ "",
+ row(["Rule", ...names.map((name) => `\`${name}\``)]),
+ row(["---", ...names.map(() => ":---:")]),
+ ...hitlContract.map((rule) =>
+ row([
+ // Non-breaking spaces keep each rule on one line; GitHub scrolls the table instead.
+ escape(rule.rule).replaceAll(" ", "\u00a0"),
+ ...entries.map((entry) => MATRIX_SYMBOLS[cellOf(entry, rule).kind]),
+ ]),
+ ),
+ "",
+ "## Broken",
+ "",
+ ...notes("broken").map(escape),
+ "",
+ "## Declined",
+ "",
+ ...notes("unsupported").map(escape),
+ "",
+ ].join("\n");
+}
diff --git a/packages/eve/src/internal/testing/channel-conformance/contract.ts b/packages/eve/src/internal/testing/channel-conformance/contract.ts
index 51328285e..4da04aacb 100644
--- a/packages/eve/src/internal/testing/channel-conformance/contract.ts
+++ b/packages/eve/src/internal/testing/channel-conformance/contract.ts
@@ -107,6 +107,47 @@ function expectAnsweredSaturday(output: unknown) {
expectAnswered(output, "Saturday");
}
+type Answer = "press" | "text";
+
+async function answerSaturday(conversation: ChannelConversation, by: Answer) {
+ const options = await askWhichDay(conversation);
+ if (by === "press") {
+ const saturday = options.find((option) => option.label === "Saturday");
+ expect(saturday, "a Saturday option to press").toBeDefined();
+ await conversation.press(saturday!);
+ } else {
+ await conversation.say("Saturday");
+ }
+ expectAnsweredSaturday(await conversation.waitForToolResult("ask_question"));
+}
+
+async function approveDeploy(conversation: ChannelConversation, by: Answer) {
+ const options = await askToDeploy(conversation);
+ if (by === "press") {
+ const approve = options.find((option) => APPROVE_LABELS.includes(option.label));
+ expect(approve, "an Approve option to press").toBeDefined();
+ await conversation.press(approve!);
+ } else {
+ await conversation.say("approve");
+ }
+ await expectDeployed(conversation);
+}
+
+/** Once answered, the prompt's message stops offering choices nobody can use anymore. */
+function expectButtonsCleared(conversation: ChannelConversation, prompt: string) {
+ const labels = conversation.shownPrompt(prompt).options.map((option) => option.label);
+ expect(labels, `the answered prompt still offers ${JSON.stringify(labels)}`).toEqual([]);
+}
+
+/** Once answered, the prompt's message shows everyone in the conversation who answered it. */
+function expectResponderNamed(conversation: ChannelConversation, prompt: string) {
+ const { text } = conversation.shownPrompt(prompt);
+ expect(
+ conversation.personShownAs.some((name) => text.includes(name)),
+ `the answered prompt never names who answered: ${JSON.stringify(text)}`,
+ ).toBe(true);
+}
+
export const hitlContract = [
{
rule: "a rendered question shows every option a person can choose",
@@ -308,6 +349,78 @@ export const hitlContract = [
await expectNotDeployed(conversation);
},
},
+ {
+ rule: "pressing an option clears the question's buttons",
+ source: "#1 (Slack's answered question card)",
+ requires: ["buttons"],
+ async run(conversation) {
+ await answerSaturday(conversation, "press");
+ expectButtonsCleared(conversation, PROMPT);
+ },
+ },
+ {
+ rule: "answering a question by text clears its buttons",
+ source: "#1 (Slack's answered question card)",
+ requires: ["buttons", "text-replies"],
+ async run(conversation) {
+ await answerSaturday(conversation, "text");
+ expectButtonsCleared(conversation, PROMPT);
+ },
+ },
+ {
+ rule: "pressing an option names who answered on the question",
+ source: "#1 (Slack's answered question card)",
+ requires: ["buttons"],
+ async run(conversation) {
+ await answerSaturday(conversation, "press");
+ expectResponderNamed(conversation, PROMPT);
+ },
+ },
+ {
+ rule: "answering a question by text names who answered on the question",
+ source: "#1 (Slack's answered question card)",
+ requires: ["buttons", "text-replies"],
+ async run(conversation) {
+ await answerSaturday(conversation, "text");
+ expectResponderNamed(conversation, PROMPT);
+ },
+ },
+ {
+ rule: "pressing Approve clears the approval's buttons",
+ source: "#2212 (Slack's settled approval card)",
+ requires: ["buttons"],
+ async run(conversation) {
+ await approveDeploy(conversation, "press");
+ expectButtonsCleared(conversation, APPROVAL_PROMPT);
+ },
+ },
+ {
+ rule: "approving by text clears the approval's buttons",
+ source: "#2212 (Slack's settled approval card)",
+ requires: ["buttons", "text-replies"],
+ async run(conversation) {
+ await approveDeploy(conversation, "text");
+ expectButtonsCleared(conversation, APPROVAL_PROMPT);
+ },
+ },
+ {
+ rule: "pressing Approve names who approved on the approval",
+ source: "#2212 (Slack's settled approval card)",
+ requires: ["buttons"],
+ async run(conversation) {
+ await approveDeploy(conversation, "press");
+ expectResponderNamed(conversation, APPROVAL_PROMPT);
+ },
+ },
+ {
+ rule: "approving by text names who approved on the approval",
+ source: "#2212 (Slack's settled approval card)",
+ requires: ["buttons", "text-replies"],
+ async run(conversation) {
+ await approveDeploy(conversation, "text");
+ expectResponderNamed(conversation, APPROVAL_PROMPT);
+ },
+ },
] as const satisfies readonly ContractRule[];
export type HitlRule = (typeof hitlContract)[number]["rule"];
diff --git a/packages/eve/src/internal/testing/channel-conformance/discord-driver.ts b/packages/eve/src/internal/testing/channel-conformance/discord-driver.ts
index 793f28906..e440a6d5d 100644
--- a/packages/eve/src/internal/testing/channel-conformance/discord-driver.ts
+++ b/packages/eve/src/internal/testing/channel-conformance/discord-driver.ts
@@ -9,6 +9,7 @@ import {
} from "#internal/testing/channel-conformance/harness.js";
let nextChannel = 0;
+const PERSON = { id: "U_CONFORMANCE", username: "alice" } as const;
/** A message component eve renders for a choice: a button (type 2) or a select menu (type 3). */
interface DiscordComponent {
@@ -41,6 +42,8 @@ export function discordDriver(): ChannelDriver {
const { privateKey, publicKeyHex } = testKeys();
let interactionId = 0;
let messageId = 0;
+ /** The message each interaction token's `@original` response is, once known. */
+ const originals = new Map<string, string>();
function signed(body: string): Request {
const timestamp = String(Math.floor(Date.now() / 1000));
@@ -63,14 +66,34 @@ export function discordDriver(): ChannelDriver {
if (path.endsWith("/typing")) {
return { body, method: `POST ${path}`, response: {} };
}
- messageId += 1;
return {
body,
method: `${request.method} ${path}`,
- response: { channel_id: channelId, id: `M_CONFORMANCE_${messageId}` },
+ response: { channel_id: channelId, id: writtenMessageId(request.method, path) },
};
}
+ /** Edits keep the id of the message they rewrite; everything else posts a new message. */
+ function writtenMessageId(method: string, path: string): string {
+ const original = /^\/api\/v\d+\/webhooks\/[^/]+\/([^/]+)\/messages\/@original$/u.exec(path);
+ if (original !== null) {
+ const token = original[1]!;
+ const known = originals.get(token);
+ if (known !== undefined) return known;
+ const created = newMessageId();
+ originals.set(token, created);
+ return created;
+ }
+ const edited = /\/messages\/([^/@]+)$/u.exec(path);
+ if (method === "PATCH" && edited !== null) return edited[1]!;
+ return newMessageId();
+ }
+
+ function newMessageId(): string {
+ messageId += 1;
+ return `M_CONFORMANCE_${messageId}`;
+ }
+
function nextInteraction(): string {
interactionId += 1;
return `I_CONFORMANCE_${interactionId}`;
@@ -95,7 +118,7 @@ export function discordDriver(): ChannelDriver {
id,
token: `tok-${id}`,
type: 2,
- user: { id: "U_CONFORMANCE", username: "alice" },
+ user: PERSON,
version: 1,
}),
);
@@ -107,14 +130,22 @@ export function discordDriver(): ChannelDriver {
readonly content?: string;
};
if (body.content?.includes(prompt) !== true) return undefined;
- const messageId = (call.response as { readonly id?: string }).id ?? "";
- return (body.components ?? []).flatMap((row) =>
- (row.components ?? []).flatMap((component) => visibleChoices(component, messageId)),
- );
+ return choicesOf(call);
+ },
+ shownMessage(call) {
+ if (!isMessageWrite(call)) return undefined;
+ return {
+ id: (call.response as { readonly id: string }).id,
+ options: choicesOf(call),
+ text: (call.body as { readonly content?: string }).content ?? "",
+ };
},
+ personShownAs: [`<@${PERSON.id}>`, PERSON.username],
press(option) {
const handle = option.handle as PressHandle;
const id = nextInteraction();
+ // A component interaction's `@original` response is the message holding the component.
+ originals.set(`tok-${id}`, handle.messageId);
return signed(
JSON.stringify({
application_id: "APP1",
@@ -127,7 +158,7 @@ export function discordDriver(): ChannelDriver {
message: { id: handle.messageId },
token: `tok-${id}`,
type: 3,
- user: { id: "U_CONFORMANCE", username: "alice" },
+ user: PERSON,
version: 1,
}),
);
@@ -139,6 +170,16 @@ export function discordDriver(): ChannelDriver {
};
}
+function choicesOf(call: PlatformCall): RenderedOption[] {
+ const body = call.body as {
+ readonly components?: readonly { readonly components?: readonly DiscordComponent[] }[];
+ };
+ const messageId = (call.response as { readonly id: string }).id;
+ return (body.components ?? []).flatMap((row) =>
+ (row.components ?? []).flatMap((component) => visibleChoices(component, messageId)),
+ );
+}
+
function visibleChoices(component: DiscordComponent, messageId: string): RenderedOption[] {
if (component.custom_id === undefined) return [];
const customId = component.custom_id;
diff --git a/packages/eve/src/internal/testing/channel-conformance/harness.ts b/packages/eve/src/internal/testing/channel-conformance/harness.ts
index 65ca56f56..3dbd4193f 100644
--- a/packages/eve/src/internal/testing/channel-conformance/harness.ts
+++ b/packages/eve/src/internal/testing/channel-conformance/harness.ts
@@ -42,6 +42,16 @@ export function numberedOptions(text: string): RenderedOption[] {
.map((label) => ({ handle: label, label }));
}
+/** One platform message as a person sees it after an outbound call posts or edits it. */
+export interface ShownMessage {
+ /** The platform's id for the message, stable across edits. */
+ readonly id: string;
+ /** Every piece of text the message shows, joined. */
+ readonly text: string;
+ /** The choices the message still lets a person press. */
+ readonly options: readonly RenderedOption[];
+}
+
/**
* What a platform can do for a person, independent of eve. A rule that needs a
* capability a driver lacks is skipped for that channel as "not supported".
@@ -82,6 +92,14 @@ export interface ChannelDriver {
press(option: RenderedOption): Request;
/** Text the bot posted in one outbound call, if any. */
postedText(call: PlatformCall): string | undefined;
+ /**
+ * The message one outbound call posts or edits, as a person sees it
+ * afterward. Required with the `buttons` capability, since rules check how
+ * an answered prompt's message changes.
+ */
+ shownMessage?(call: PlatformCall): ShownMessage | undefined;
+ /** How the person driving the conversation appears in the platform's text, in any form. */
+ readonly personShownAs?: readonly string[];
}
/** What a person can do and see in one channel conversation. Contract rules use only this. */
@@ -115,6 +133,14 @@ export interface ChannelConversation {
waitForRest(): Promise<void>;
/** How many times a gated tool actually executed, as its side effect would show. */
runsOf(tool: GatedTool): number;
+ /**
+ * The message that asked `prompt` as it stands now, after every edit so far.
+ * Rules read it once an answer has settled, by which point the bot has had
+ * every chance to update it.
+ */
+ shownPrompt(prompt: string): ShownMessage;
+ /** How the person appears in the platform's text, in any form. */
+ readonly personShownAs: readonly string[];
}
/**
@@ -145,6 +171,10 @@ export interface ClientView {
press(option: RenderedOption): Promise<void>;
/** The bot replies the client shows now. */
replies(): readonly string[];
+ /** The message that asked `prompt` as it stands now; see {@link ChannelConversation.shownPrompt}. */
+ shownPrompt?(prompt: string): ShownMessage;
+ /** How the person appears in the client's text, in any form. */
+ readonly personShownAs?: readonly string[];
/** What the client shows now, for timeout errors. */
describe(): string;
/** Stops the client and releases anything it holds, such as its event stream. */
@@ -292,6 +322,20 @@ function webhookView(
describe,
),
replies: () => calls.flatMap((call) => driver.postedText(call) ?? []),
+ shownPrompt(prompt) {
+ const read = driver.shownMessage?.bind(driver);
+ if (read === undefined) throw new Error(`${driver.name} cannot read shown messages.`);
+ const asked = calls.find((call) => (driver.findOptions(call, prompt)?.length ?? 0) > 0);
+ const id = asked === undefined ? undefined : read(asked)?.id;
+ if (id === undefined) throw new Error(`The question "${prompt}" was never asked.`);
+ return calls
+ .flatMap((call) => {
+ const shown = read(call);
+ return shown?.id === id ? [shown] : [];
+ })
+ .at(-1)!;
+ },
+ personShownAs: driver.personShownAs ?? [],
describe,
close: async () => {},
};
@@ -489,6 +533,11 @@ async function converse(
}),
runsOf: (tool) => runs[tool],
waitForRest: () => waitForRest([...sessions.values()], wait),
+ shownPrompt(prompt) {
+ if (view.shownPrompt === undefined) throw new Error(`${label} cannot read shown messages.`);
+ return view.shownPrompt(prompt);
+ },
+ personShownAs: view.personShownAs ?? [],
};
/**
diff --git a/packages/eve/src/internal/testing/channel-conformance/matrix.test.ts b/packages/eve/src/internal/testing/channel-conformance/matrix.test.ts
new file mode 100644
index 000000000..1086a6ac7
--- /dev/null
+++ b/packages/eve/src/internal/testing/channel-conformance/matrix.test.ts
@@ -0,0 +1,7 @@
+import { expect, it } from "vitest";
+
+import { renderHitlConformanceMatrix } from "#internal/testing/channel-conformance/conformance.js";
+
+it("MATRIX.md shows the conformance table", async () => {
+ await expect(renderHitlConformanceMatrix()).toMatchFileSnapshot("./MATRIX.md");
+});
diff --git a/packages/eve/src/internal/testing/channel-conformance/slack-driver.ts b/packages/eve/src/internal/testing/channel-conformance/slack-driver.ts
index 8bd575139..3654a3af0 100644
--- a/packages/eve/src/internal/testing/channel-conformance/slack-driver.ts
+++ b/packages/eve/src/internal/testing/channel-conformance/slack-driver.ts
@@ -27,7 +27,7 @@ interface SlackOption {
/** The HITL widgets eve renders: one button per option, or one single-click radio/select. */
interface SlackElement {
- readonly action_id: string;
+ readonly action_id?: string;
readonly options?: readonly SlackOption[];
readonly text?: SlackText;
readonly type: string;
@@ -36,9 +36,17 @@ interface SlackElement {
interface PressHandle {
readonly action: Record<string, unknown>;
+ /** The pressed message's blocks, which Slack echoes in every `block_actions` payload. */
+ readonly blocks: readonly unknown[];
readonly messageTs: string;
}
+interface SlackMessageBody {
+ readonly blocks?: readonly unknown[];
+ readonly text?: string;
+ readonly ts?: string;
+}
+
/** Drives the Slack channel through its Events API and interactivity webhooks in a DM. */
export function slackDriver(): ChannelDriver {
// A fresh DM per driver keeps each test's session apart in the shared workflow world.
@@ -120,32 +128,35 @@ export function slackDriver(): ChannelDriver {
);
},
findOptions(call, prompt) {
- if (call.method !== "chat.postMessage" && call.method !== "chat.update") return undefined;
- const body = call.body as { readonly blocks?: readonly unknown[] };
- if (body.blocks === undefined || !JSON.stringify(body.blocks).includes(prompt))
- return undefined;
- const messageTs = (call.response as { readonly ts: string }).ts;
- const optionsIn = (blocks: readonly unknown[]) =>
- blocks.flatMap((block) =>
- // Questions use `actions` blocks (`elements`); approval cards keep buttons in `actions`.
- (
- (block as { readonly elements?: readonly SlackElement[] }).elements ??
- (block as { readonly actions?: readonly SlackElement[] }).actions ??
- []
- )
- .filter((element) => element.action_id.startsWith(HITL_ACTION_PREFIX))
- .flatMap((element) => renderedOptions(element, messageTs)),
- );
+ const body = call.body as SlackMessageBody;
+ if (!isMessageWrite(call) || body.blocks === undefined) return undefined;
+ if (!JSON.stringify(body.blocks).includes(prompt)) return undefined;
+ const messageTs = messageTsOf(call);
// A batch of approvals posts one card per request, each holding its own buttons.
- const card = optionsIn(body.blocks.filter((block) => JSON.stringify(block).includes(prompt)));
- return card.length > 0 ? card : optionsIn(body.blocks);
+ const card = hitlOptions(
+ body.blocks.filter((block) => JSON.stringify(block).includes(prompt)),
+ messageTs,
+ body.blocks,
+ );
+ return card.length > 0 ? card : hitlOptions(body.blocks, messageTs);
},
+ shownMessage(call) {
+ if (!isMessageWrite(call)) return undefined;
+ const body = call.body as SlackMessageBody;
+ const id = messageTsOf(call);
+ return {
+ id,
+ options: hitlOptions(body.blocks ?? [], id),
+ text: [body.text ?? "", ...blockTexts(body.blocks ?? [])].join("\n"),
+ };
+ },
+ personShownAs: [`<@${PERSON}>`],
press: (option) => {
- const { action, messageTs } = option.handle as PressHandle;
+ const { action, blocks, messageTs } = option.handle as PressHandle;
const payload = {
actions: [action],
channel: { id: CHANNEL },
- message: { thread_ts: threadTs, ts: messageTs },
+ message: { blocks, thread_ts: threadTs, ts: messageTs },
team: { id: TEAM },
type: "block_actions",
user: { id: PERSON, name: "alice", team_id: TEAM, username: "alice" },
@@ -169,7 +180,49 @@ export function slackDriver(): ChannelDriver {
};
}
-function renderedOptions(element: SlackElement, messageTs: string): RenderedOption[] {
+function isMessageWrite(call: PlatformCall): boolean {
+ return call.method === "chat.postMessage" || call.method === "chat.update";
+}
+
+/** A post's ts comes back from Slack; an update names the ts it rewrites. */
+function messageTsOf(call: PlatformCall): string {
+ return call.method === "chat.update"
+ ? (call.body as SlackMessageBody).ts!
+ : (call.response as { readonly ts: string }).ts;
+}
+
+/** HITL buttons in `blocks`, each pressed as part of the whole message `messageBlocks`. */
+function hitlOptions(
+ blocks: readonly unknown[],
+ messageTs: string,
+ messageBlocks: readonly unknown[] = blocks,
+): RenderedOption[] {
+ return blocks.flatMap((block) =>
+ // Questions use `actions` blocks (`elements`); approval cards keep buttons in `actions`.
+ (
+ (block as { readonly elements?: readonly SlackElement[] }).elements ??
+ (block as { readonly actions?: readonly SlackElement[] }).actions ??
+ []
+ )
+ .filter((element) => element.action_id?.startsWith(HITL_ACTION_PREFIX) === true)
+ .flatMap((element) => renderedOptions(element, messageBlocks, messageTs)),
+ );
+}
+
+/** Every `text` string in Block Kit, wherever a block nests it. */
+function blockTexts(value: unknown): string[] {
+ if (Array.isArray(value)) return value.flatMap(blockTexts);
+ if (typeof value !== "object" || value === null) return [];
+ return Object.entries(value).flatMap(([key, child]) =>
+ key === "text" && typeof child === "string" ? [child] : blockTexts(child),
+ );
+}
+
+function renderedOptions(
+ element: SlackElement,
+ blocks: readonly unknown[],
+ messageTs: string,
+): RenderedOption[] {
if (element.type === "button" && element.text !== undefined) {
const action = {
action_id: element.action_id,
@@ -177,11 +230,14 @@ function renderedOptions(element: SlackElement, messageTs: string): RenderedOpti
type: "button",
value: element.value,
};
- return [{ handle: { action, messageTs } satisfies PressHandle, label: element.text.text }];
+ return [
+ { handle: { action, blocks, messageTs } satisfies PressHandle, label: element.text.text },
+ ];
}
return (element.options ?? []).map((option) => ({
handle: {
action: { action_id: element.action_id, selected_option: option, type: element.type },
+ blocks,
messageTs,
} satisfies PressHandle,
label: option.text.text,
diff --git a/packages/eve/src/internal/testing/channel-conformance/teams-driver.ts b/packages/eve/src/internal/testing/channel-conformance/teams-driver.ts
index 4c35e90f9..325a3f671 100644
--- a/packages/eve/src/internal/testing/channel-conformance/teams-driver.ts
+++ b/packages/eve/src/internal/testing/channel-conformance/teams-driver.ts
@@ -20,6 +20,7 @@ interface Choice {
interface AdaptiveCard {
readonly body?: readonly {
readonly choices?: readonly Choice[];
+ readonly text?: string;
readonly type?: string;
}[];
readonly actions?: readonly {
@@ -39,6 +40,8 @@ interface ActivityBody {
}
interface PressHandle {
+ /** The card's activity, which Teams names as `replyToId` on the press. */
+ readonly activityId: string;
readonly data: Record<string, unknown>;
/** The chosen ChoiceSet value; unset when each option is its own Submit action. */
readonly value?: string;
@@ -75,11 +78,10 @@ export function teamsDriver(): ChannelDriver {
async function decode(request: Request): Promise<PlatformCall> {
const bodyText = await request.text();
const body = bodyText === "" ? {} : JSON.parse(bodyText);
- return {
- body,
- method: new URL(request.url).pathname.split("/").at(-1) ?? "",
- response: { id: nextActivityId() },
- };
+ const path = new URL(request.url).pathname;
+ // An update keeps the id of the activity it replaces.
+ const id = request.method === "PUT" ? path.split("/").at(-1)! : nextActivityId();
+ return { body, method: `${request.method} ${path}`, response: { id } };
}
return {
@@ -93,35 +95,28 @@ export function teamsDriver(): ChannelDriver {
message: (text) => activity({ text, type: "message" }),
findOptions(call, prompt) {
const body = call.body as ActivityBody;
- if (body.type === "typing" || body.type !== "message" || !body.text?.includes(prompt)) {
- return undefined;
- }
- const card = body.attachments?.find(
- (attachment) => attachment.contentType === "application/vnd.microsoft.card.adaptive",
- )?.content;
- const choiceSet = card?.body?.find((element) => element.type === "Input.ChoiceSet");
- const submits = (card?.actions ?? []).filter(
- (action) => action.type === "Action.Submit" && action.data !== undefined,
+ if (body.type !== "message" || !body.text?.includes(prompt)) return undefined;
+ return cardOptions(call);
+ },
+ shownMessage(call) {
+ const body = call.body as ActivityBody;
+ if (body.type !== "message") return undefined;
+ const texts = (adaptiveCard(body)?.body ?? []).flatMap((element) =>
+ element.type === "TextBlock" && element.text !== undefined ? [element.text] : [],
);
- // A select renders one ChoiceSet and a single Submit; other choices are one Submit each.
- if (choiceSet?.choices !== undefined) {
- const data = submits[0]?.data;
- if (data === undefined) return [];
- return choiceSet.choices.map((choice) => ({
- handle: { data, value: choice.value } satisfies PressHandle,
- label: choice.title,
- }));
- }
- return submits.map((action) => ({
- handle: { data: action.data! } satisfies PressHandle,
- label: action.title ?? "",
- }));
+ return {
+ id: activityIdOf(call),
+ options: cardOptions(call),
+ text: [body.text ?? "", ...texts].join("\n"),
+ };
},
+ personShownAs: [PERSON.name],
press: (option: RenderedOption) => {
- const { data, value } = option.handle as PressHandle;
+ const { activityId: cardActivityId, data, value } = option.handle as PressHandle;
return activity({
id: `INVOKE-${activityId + 1}`,
name: "adaptiveCard/action",
+ replyToId: cardActivityId,
type: "invoke",
value: { action: { data: value === undefined ? data : { ...data, eve_option: value } } },
});
@@ -133,3 +128,35 @@ export function teamsDriver(): ChannelDriver {
},
};
}
+
+function activityIdOf(call: PlatformCall): string {
+ return (call.response as { readonly id: string }).id;
+}
+
+function adaptiveCard(body: ActivityBody): AdaptiveCard | undefined {
+ return body.attachments?.find(
+ (attachment) => attachment.contentType === "application/vnd.microsoft.card.adaptive",
+ )?.content;
+}
+
+function cardOptions(call: PlatformCall): RenderedOption[] {
+ const card = adaptiveCard(call.body as ActivityBody);
+ const activityId = activityIdOf(call);
+ const choiceSet = card?.body?.find((element) => element.type === "Input.ChoiceSet");
+ const submits = (card?.actions ?? []).filter(
+ (action) => action.type === "Action.Submit" && action.data !== undefined,
+ );
+ // A select renders one ChoiceSet and a single Submit; other choices are one Submit each.
+ if (choiceSet?.choices !== undefined) {
+ const data = submits[0]?.data;
+ if (data === undefined) return [];
+ return choiceSet.choices.map((choice) => ({
+ handle: { activityId, data, value: choice.value } satisfies PressHandle,
+ label: choice.title,
+ }));
+ }
+ return submits.map((action) => ({
+ handle: { activityId, data: action.data! } satisfies PressHandle,
+ label: action.title ?? "",
+ }));
+}
diff --git a/packages/eve/src/internal/testing/channel-conformance/telegram-driver.ts b/packages/eve/src/internal/testing/channel-conformance/telegram-driver.ts
index 7d832c80d..adeadee8a 100644
--- a/packages/eve/src/internal/testing/channel-conformance/telegram-driver.ts
+++ b/packages/eve/src/internal/testing/channel-conformance/telegram-driver.ts
@@ -2,6 +2,7 @@ import { telegramChannel } from "#public/channels/telegram/index.js";
import {
type ChannelDriver,
type PlatformCall,
+ type RenderedOption,
recordingFetch,
} from "#internal/testing/channel-conformance/harness.js";
@@ -13,6 +14,42 @@ interface InlineButton {
readonly text: string;
}
+interface MessageBody {
+ readonly message_id?: number;
+ readonly reply_markup?: { readonly inline_keyboard?: readonly InlineButton[][] };
+ readonly text?: string;
+}
+
+interface PressHandle {
+ readonly data: string;
+ readonly messageId: number;
+}
+
+const MESSAGE_WRITES = new Set(["sendMessage", "editMessageText", "editMessageReplyMarkup"]);
+
+/** A send's id comes back from Telegram; an edit names the message it rewrites. */
+function messageIdOf(call: PlatformCall): number {
+ return call.method === "sendMessage"
+ ? (call.response as { readonly result: { readonly message_id: number } }).result.message_id
+ : (call.body as MessageBody).message_id!;
+}
+
+function inlineOptions(call: PlatformCall): RenderedOption[] {
+ const messageId = messageIdOf(call);
+ return ((call.body as MessageBody).reply_markup?.inline_keyboard ?? [])
+ .flat()
+ .flatMap((button) =>
+ button.callback_data === undefined
+ ? []
+ : [
+ {
+ handle: { data: button.callback_data, messageId } satisfies PressHandle,
+ label: button.text,
+ },
+ ],
+ );
+}
+
/** Drives the Telegram channel through its Bot API webhook in a private chat. */
export function telegramDriver(): ChannelDriver {
// A fresh chat per driver keeps each test's session apart in the shared workflow world.
@@ -56,26 +93,32 @@ export function telegramDriver(): ChannelDriver {
message: (text) =>
update({ message: { chat: CHAT, date: 0, from: PERSON, message_id: 1000 + updateId, text } }),
findOptions(call, prompt) {
- const body = call.body as {
- readonly reply_markup?: { readonly inline_keyboard?: readonly InlineButton[][] };
- readonly text?: string;
- };
+ const body = call.body as MessageBody;
if (call.method !== "sendMessage" || body.text?.includes(prompt) !== true) return undefined;
// An open-ended question posts a ForceReply prompt with no keyboard.
- return (body.reply_markup?.inline_keyboard ?? [])
- .flat()
- .filter((button) => button.callback_data !== undefined)
- .map((button) => ({ handle: button.callback_data, label: button.text }));
+ return inlineOptions(call);
},
- press: (option) =>
- update({
+ shownMessage(call) {
+ if (!MESSAGE_WRITES.has(call.method)) return undefined;
+ return {
+ id: String(messageIdOf(call)),
+ // Telegram drops a message's inline keyboard when an edit omits `reply_markup`.
+ options: inlineOptions(call),
+ text: (call.body as MessageBody).text ?? "",
+ };
+ },
+ personShownAs: [PERSON.first_name],
+ press: (option) => {
+ const { data, messageId } = option.handle as PressHandle;
+ return update({
callback_query: {
- data: option.handle,
+ data,
from: PERSON,
id: `callback-${updateId}`,
- message: { chat: CHAT, date: 0, message_id: 1 },
+ message: { chat: CHAT, date: 0, message_id: messageId },
},
- }),
+ });
+ },
postedText: (call: PlatformCall) =>
call.method === "sendMessage" || call.method === "editMessageText"
? (call.body as { readonly text?: string }).text
diff --git a/packages/eve/src/internal/testing/channel-conformance/tui-driver.ts b/packages/eve/src/internal/testing/channel-conformance/tui-driver.ts
index b3e03c710..46e5aedd0 100644
--- a/packages/eve/src/internal/testing/channel-conformance/tui-driver.ts
+++ b/packages/eve/src/internal/testing/channel-conformance/tui-driver.ts
@@ -88,6 +88,14 @@ export function tuiDriver(): ClientDriver {
keyboard.enter();
},
replies: () => readReplies(screen.snapshot()),
+ // The TUI asks in a drawer rather than a posted message, so an answered
+ // prompt shows no options once its drawer closes.
+ shownPrompt: (prompt) => ({
+ id: prompt,
+ options:
+ focusedRow(screen, prompt) === undefined ? [] : readOptions(screen, keyboard, prompt),
+ text: screen.snapshot(),
+ }),
describe,
async close() {
renderer.requestInterrupt();