Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

fix(slack): never silently drop the final reply when Slack rejects it (#4136)

Andrew Barba490ac0d6f298
diff --git a/.changeset/slack-final-reply-delivery.md b/.changeset/slack-final-reply-delivery.md
new file mode 100644
index 000000000..469a50dc5
--- /dev/null
+++ b/.changeset/slack-final-reply-delivery.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Slack no longer drops a final reply without a trace. When Slack refuses eve's default reply as too large or malformed, eve uploads it as a Markdown snippet. When a reply still isn't delivered, including when a custom renderer throws, eve posts a short notice with an error id, without the reply, and tells the model on the next message that the user never saw it. Custom renderers can opt in to the same fallbacks by posting through `postCompletedSlackReply` from `eve/channels/slack`, which tries their blocks, then native Markdown, then a snippet, using only the content they pass.
diff --git a/docs/channels/slack.mdx b/docs/channels/slack.mdx
index c21f6ade2..6d7233130 100644
--- a/docs/channels/slack.mdx
+++ b/docs/channels/slack.mdx
@@ -127,15 +127,16 @@ vercel connect attach slack/your-agent \
Check the delivery path in order so you can identify where a Slack event stops:
-| Symptom | Check | Next action |
-| --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `eve add channel/slack` stops before scaffolding the channel | Read the setup error for the failed Vercel login, project link, workspace install, connector lookup, or trigger registration step. | Resolve that step, then rerun `eve add channel/slack`. |
-| Setup says it could not remove an existing trigger destination | Check the connector's destinations in the Connect dashboard. `vercel connect detach` removes project token access, not trigger destinations. | Edit or remove the stale destination in the dashboard, then register `/eve/v1/slack` with `vercel connect attach`. |
-| Mentions never reach the deployment | Confirm that the connector trigger destination uses the intended project and branch, enables triggers, and points to `/eve/v1/slack`. | Edit or remove stale destinations in the Connect dashboard, then register the intended destination with `vercel connect attach`. |
-| Mentions work but DMs or thread replies do not | Check the Slack app's Event Subscriptions and OAuth scopes. DMs require `message.im` and `im:history`; channel replies require the matching message event and history scope. | Add the missing events or scopes, then reinstall the Slack app if Slack requires it. |
-| A long response cannot be uploaded as a snippet | Check whether the installed Slack app has the `files:write` bot scope. | Add `files:write`, then reinstall the Slack app so the new scope takes effect. |
-| A preview request receives an authentication page or protection error | Check whether the branch domain is exempt or the direct Slack request URL includes `x-vercel-protection-bypass`. Slack cannot complete an interactive Vercel Authentication challenge. | Add a Deployment Protection Exception for the branch domain, or add the automation bypass query parameter to a Slack app that uses environment-variable credentials. |
-| The webhook reaches the deployment but the agent does not reply | Open the project's Vercel runtime logs, then inspect the session in [**Agent Runs**](/docs/observability/agent-runs) when your team has access. | Use the first missing or failed stage to narrow the issue to channel dispatch, the agent run, or outbound Slack delivery. |
+| Symptom | Check | Next action |
+| --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `eve add channel/slack` stops before scaffolding the channel | Read the setup error for the failed Vercel login, project link, workspace install, connector lookup, or trigger registration step. | Resolve that step, then rerun `eve add channel/slack`. |
+| Setup says it could not remove an existing trigger destination | Check the connector's destinations in the Connect dashboard. `vercel connect detach` removes project token access, not trigger destinations. | Edit or remove the stale destination in the dashboard, then register `/eve/v1/slack` with `vercel connect attach`. |
+| Mentions never reach the deployment | Confirm that the connector trigger destination uses the intended project and branch, enables triggers, and points to `/eve/v1/slack`. | Edit or remove stale destinations in the Connect dashboard, then register the intended destination with `vercel connect attach`. |
+| Mentions work but DMs or thread replies do not | Check the Slack app's Event Subscriptions and OAuth scopes. DMs require `message.im` and `im:history`; channel replies require the matching message event and history scope. | Add the missing events or scopes, then reinstall the Slack app if Slack requires it. |
+| A long response cannot be uploaded as a snippet | Check whether the installed Slack app has the `files:write` bot scope. | Add `files:write`, then reinstall the Slack app so the new scope takes effect. |
+| The thread says the agent couldn't deliver its answer | Search the Vercel runtime logs for the error id in the notice. The `final reply was not delivered to Slack` line carries the error, including one thrown by an authored renderer. | Fix the cause, such as a missing `files:write` scope or a renderer that posts invalid blocks. A renderer can post through `postCompletedSlackReply` to get eve's fallbacks. Then ask the agent to resend the answer. |
+| A preview request receives an authentication page or protection error | Check whether the branch domain is exempt or the direct Slack request URL includes `x-vercel-protection-bypass`. Slack cannot complete an interactive Vercel Authentication challenge. | Add a Deployment Protection Exception for the branch domain, or add the automation bypass query parameter to a Slack app that uses environment-variable credentials. |
+| The webhook reaches the deployment but the agent does not reply | Open the project's Vercel runtime logs, then inspect the session in [**Agent Runs**](/docs/observability/agent-runs) when your team has access. | Use the first missing or failed stage to narrow the issue to channel dispatch, the agent run, or outbound Slack delivery. |
## Audience
@@ -469,7 +470,13 @@ Slack clears a thread status after two minutes without a message, so during a lo
Action labels follow the same rule as the task card titles described below, plus `+N more` when the model requests several actions at once. They leave out the model's own `task_wait` and `task_cancel` calls. Reasoning snippets build progressively: extensions of at least four characters appear immediately, while smaller streamed deltas use a five-second refresh interval to avoid one Slack request per token.
-Replies up to 12,000 characters render inline. Longer replies are uploaded unchanged as a Markdown snippet named `eve-response.md` with a short in-thread note, which requires the `files:write` bot scope. Upload failures are logged by the channel dispatcher; they do not trigger a model retry.
+Replies up to 12,000 characters render inline. Longer replies are uploaded unchanged as a Markdown snippet named `eve-response.md` with a short in-thread note, which requires the `files:write` bot scope.
+
+eve's default renderer also recovers when Slack refuses its reply as too large or malformed (`msg_too_long`, `msg_blocks_too_long`, `msg_blocks_too_many`, `invalid_blocks`, or `invalid_blocks_format`): a refused Markdown message is uploaded as the same `eve-response.md` snippet, without posting the message again. The default delivers the reply it receives, so a change a [renderer](#customize-rendering) passes to `next()`, such as a redaction, carries into the snippet. eve logs each fallback it takes, `inline` or `snippet`, with Slack's error code in a `Slack refused the final reply` warning. The log never includes the reply text.
+
+When a final reply still doesn't reach Slack, because the upload fails, Slack returns another error, or an authored renderer throws, eve logs the error and posts a short notice in the thread: `I finished, but couldn't deliver my answer in Slack (error id ...). Please ask me to resend it.` The error id matches the `final reply was not delivered to Slack` line in the Vercel runtime logs. The notice carries no reply content: eve never re-sends a reply that a renderer failed to post. The next message in the session tells the model that its previous reply was not delivered, so it doesn't assume the user read it.
+
+This applies to the turn's final reply only. A failed status update, tool-call narration, or other event is logged and skipped. Delivery failures don't trigger a model retry.
When the model replies while its tasks keep working, that reply posts like any other, and the reply after the results posts as another message. A schedule's session posts only the final reply.
@@ -548,6 +555,39 @@ export default slackChannel({
});
```
+A `message.completed` handler that posts the final reply itself, instead of calling `next()`, owns that delivery. If it throws, eve logs the error, posts the content-free notice described in [Delivery](#delivery), and tells the model on the next message. eve doesn't retry or re-send your reply, because your renderer may post elsewhere, change the content, or attach data eve can't reproduce. If you post the reply and then a separate follow-up message, as the feedback example above does, catch the follow-up's errors in your handler. Otherwise eve posts the notice even though the reply arrived.
+
+To get eve's fallbacks for a reply you render, post it with `postCompletedSlackReply`. You choose the content; eve makes the Slack calls:
+
+1. Your `blocks`, when given, with `text` as the notification text. `text` defaults to `markdown`.
+2. Your `markdown` as one native Markdown message, when it fits in `SLACK_MARKDOWN_TEXT_MAX_LENGTH` (12,000) characters. eve moves on from the blocks only when Slack refuses them as too large or malformed.
+3. Your `markdown` uploaded as the `eve-response.md` snippet, when it is longer or Slack refuses the Markdown message.
+
+Any other error, and a failed upload, is thrown to your handler. If your handler lets it propagate, eve posts the notice. eve only sends the `blocks`, `text`, and `markdown` you pass. Slack caps the combined text of all `markdown` blocks in one message at 12,000 characters, so a reply that fits in `markdown_text` can still be refused as blocks.
+
+```ts title="agent/channels/slack.ts"
+import { defineSlackRenderer, postCompletedSlackReply, slackChannel } from "eve/channels/slack";
+
+const sourcedReplies = defineSlackRenderer({
+ events: {
+ async "message.completed"(event, channel, _ctx, next) {
+ if (event.finishReason === "tool-calls" || !event.message) return next();
+ await postCompletedSlackReply(channel, {
+ blocks: [
+ { type: "markdown", text: event.message },
+ { type: "context", elements: [{ type: "mrkdwn", text: "Sources: internal docs" }] },
+ ],
+ markdown: event.message,
+ });
+ },
+ },
+});
+
+export default slackChannel({ renderers: [sourcedReplies] });
+```
+
+Pass a string, as in `postCompletedSlackReply(channel, reply)`, to deliver Markdown alone the way eve's default does.
+
`taskCard` runs for every turn that calls a tool, and receives the same view eve's default renders from:
- `state`: `blocked` while any task waits on a person, `working` while the turn or any other task works, then `finished`.
diff --git a/packages/eve/src/public/channels/slack/defaults.ts b/packages/eve/src/public/channels/slack/defaults.ts
index d5df94713..a8c531d62 100644
--- a/packages/eve/src/public/channels/slack/defaults.ts
+++ b/packages/eve/src/public/channels/slack/defaults.ts
@@ -24,8 +24,8 @@ import {
} from "#public/channels/slack/hitl.js";
import type { SlackMessage } from "#public/channels/slack/inbound.js";
import { deliverPrivateInputRequest } from "#public/channels/slack/private-approval-delivery.js";
+import { deliverCompletedSlackReply } from "#public/channels/slack/reply-delivery.js";
import {
- SLACK_MARKDOWN_TEXT_MAX_LENGTH,
SLACK_MAX_BLOCKS_PER_MESSAGE,
truncateMessageText,
truncateTypingStatus,
@@ -43,8 +43,6 @@ import type { InputRequest } from "#shared/input.js";
const log = createLogger("slack.defaults");
const REASONING_TYPING_REFRESH_INTERVAL_MS = 5_000;
const REASONING_TYPING_MIN_PROGRESS_CHARS = 4;
-const LONG_RESPONSE_FILENAME = "eve-response.md";
-const LONG_RESPONSE_NOTICE = "Here's a snippet with the full response.";
interface ReasoningAccumulator {
readonly stepIndex: number;
readonly text: string;
@@ -311,40 +309,6 @@ function groupInputRequestPostParts(
}));
}
-/**
- * Delivers a completed default Slack reply without sending content that
- * exceeds Slack's native Markdown limit. Long replies stay intact as one
- * Markdown snippet instead of being truncated or split across messages.
- */
-export async function postCompletedSlackReply(
- channel: SlackContext,
- message: string,
-): Promise<void> {
- if (message.length <= SLACK_MARKDOWN_TEXT_MAX_LENGTH) {
- await channel.thread.post(message);
- return;
- }
-
- const file = {
- data: new Blob([message], { type: "text/markdown" }),
- filename: LONG_RESPONSE_FILENAME,
- mimeType: "text/markdown",
- };
-
- const hasThread = channel.slack.threadTs.length > 0;
- if (!hasThread) {
- // Uploads cannot anchor proactive sessions; post the notice first.
- const anchor = await channel.thread.post(LONG_RESPONSE_NOTICE);
- if (!anchor.id || channel.slack.threadTs.length === 0) {
- throw new Error("Slack did not return a thread timestamp for the long response notice.");
- }
- }
- await channel.slack.uploadFiles([file], {
- initialComment: hasThread ? LONG_RESPONSE_NOTICE : undefined,
- snippetType: "markdown",
- });
-}
-
/**
* eve's default Slack event rendering: status lines, replies, errors, and the
* connection-authorization flow. It is the innermost link of every channel's
@@ -516,7 +480,7 @@ export const defaultEvents: SlackChannelInternalEvents = {
await channel.thread.startTyping();
return;
}
- await postCompletedSlackReply(channel, event.message);
+ await deliverCompletedSlackReply(channel, event.message, { turnId: event.turnId });
},
async "turn.failed"(event, channel, _ctx) {
diff --git a/packages/eve/src/public/channels/slack/index.ts b/packages/eve/src/public/channels/slack/index.ts
index 22ad41bef..482f162cc 100644
--- a/packages/eve/src/public/channels/slack/index.ts
+++ b/packages/eve/src/public/channels/slack/index.ts
@@ -75,6 +75,13 @@ export { type SlackFetch, type SlackTransportOptions } from "#public/channels/sl
export { defaultSlackAuth } from "#public/channels/slack/defaults.js";
+export {
+ postCompletedSlackReply,
+ type SlackCompletedReply,
+} from "#public/channels/slack/reply-delivery.js";
+
+export { SLACK_MARKDOWN_TEXT_MAX_LENGTH } from "#public/channels/slack/limits.js";
+
export {
defineSlackRenderer,
type SlackRenderer,
diff --git a/packages/eve/src/public/channels/slack/limits.ts b/packages/eve/src/public/channels/slack/limits.ts
index 2e4766af1..70334aa31 100644
--- a/packages/eve/src/public/channels/slack/limits.ts
+++ b/packages/eve/src/public/channels/slack/limits.ts
@@ -45,9 +45,11 @@ export const SLACK_CARD_SUBTEXT_MAX_LENGTH = 200;
export const SLACK_MESSAGE_TEXT_MAX_LENGTH = 40000;
/**
- * Slack's native `markdown_text` field on `chat.postMessage` is capped at
- * 12000 characters. Default agent replies use this field so headings, tables,
- * and other Markdown render natively.
+ * Slack caps a message's Markdown at 12,000 characters: the `markdown_text`
+ * field of `chat.postMessage`, and the combined text of every `markdown`
+ * block in one message. eve's default replies use `markdown_text` so
+ * headings, tables, and other Markdown render natively, and upload longer
+ * replies as a snippet.
*/
export const SLACK_MARKDOWN_TEXT_MAX_LENGTH = 12000;
diff --git a/packages/eve/src/public/channels/slack/reply-delivery.ts b/packages/eve/src/public/channels/slack/reply-delivery.ts
new file mode 100644
index 000000000..582d227dc
--- /dev/null
+++ b/packages/eve/src/public/channels/slack/reply-delivery.ts
@@ -0,0 +1,222 @@
+import type { DeliverPayload } from "#channel/types.js";
+import { createLogger, logError } from "#internal/logging.js";
+import {
+ SLACK_MARKDOWN_TEXT_MAX_LENGTH,
+ truncateMessageText,
+} from "#public/channels/slack/limits.js";
+import type {
+ SlackChannelInternalEvents,
+ SlackChannelState,
+ SlackContext,
+ SlackEventContext,
+} from "#public/channels/slack/slackChannel.js";
+
+const log = createLogger("slack.reply");
+const LONG_RESPONSE_FILENAME = "eve-response.md";
+const LONG_RESPONSE_NOTICE = "Here's a snippet with the full response.";
+const FALLBACK_LOG = "Slack refused the final reply; delivering it with a fallback";
+
+/**
+ * Slack error codes for a message payload Slack refused as too large or
+ * malformed. Slack posts nothing for a refused call, so a fallback post
+ * cannot duplicate it.
+ */
+const REFUSED_PAYLOAD_ERRORS = new Set([
+ "invalid_blocks",
+ "invalid_blocks_format",
+ "msg_blocks_too_long",
+ "msg_blocks_too_many",
+ "msg_too_long",
+]);
+
+type MessageCompletedHandler = NonNullable<SlackChannelInternalEvents["message.completed"]>;
+
+/**
+ * A completed reply for {@link postCompletedSlackReply}: Markdown alone, or
+ * Markdown with the Block Kit blocks to try first. `markdown` is the full
+ * reply eve falls back to; `text` is the notification text for the blocks,
+ * defaulting to `markdown`.
+ */
+export type SlackCompletedReply =
+ | string
+ | {
+ readonly markdown: string;
+ readonly blocks?: readonly unknown[];
+ readonly text?: string;
+ };
+
+/**
+ * Posts a completed reply to the bound Slack thread with eve's delivery
+ * fallbacks, using only the content you pass. eve's default renderer posts
+ * its replies through it; an authored renderer opts in by calling it instead
+ * of `channel.thread.post`.
+ *
+ * eve tries, in order:
+ *
+ * 1. Your `blocks`, when given, with `text` as the notification text.
+ * 2. `markdown` as one native Markdown message, when it fits in
+ * {@link SLACK_MARKDOWN_TEXT_MAX_LENGTH} characters. eve moves on from
+ * the blocks only when Slack refuses them as too large or malformed
+ * (`msg_too_long`, `msg_blocks_too_long`, `msg_blocks_too_many`,
+ * `invalid_blocks`, or `invalid_blocks_format`).
+ * 3. `markdown` uploaded unchanged as an `eve-response.md` snippet with a
+ * short note, when it is longer or Slack refuses the Markdown message.
+ * This requires the `files:write` bot scope. In a session without a
+ * thread yet, the note posts first and anchors the thread.
+ *
+ * Any other error, and a failed upload, is thrown so your renderer sees it.
+ * eve logs each fallback it takes with Slack's error code, never the reply.
+ *
+ * @example
+ * ```ts
+ * import { postCompletedSlackReply } from "eve/channels/slack";
+ *
+ * await postCompletedSlackReply(channel, {
+ * blocks: [{ type: "markdown", text: reply }, sourcesBlock],
+ * markdown: reply,
+ * });
+ * ```
+ */
+export async function postCompletedSlackReply(
+ channel: SlackContext,
+ reply: SlackCompletedReply,
+): Promise<void> {
+ await deliverCompletedSlackReply(channel, reply, {});
+}
+
+/**
+ * {@link postCompletedSlackReply} with fields for its fallback log, such as
+ * the turn id eve's default renderer knows.
+ */
+export async function deliverCompletedSlackReply(
+ channel: SlackContext,
+ reply: SlackCompletedReply,
+ logFields: { readonly turnId?: string },
+): Promise<void> {
+ const { blocks, markdown, text } = typeof reply === "string" ? { markdown: reply } : reply;
+ let slackError: string | undefined;
+ if (blocks !== undefined) {
+ try {
+ await channel.thread.post({ blocks, text: text ?? truncateMessageText(markdown) });
+ return;
+ } catch (error) {
+ slackError = refusedPayloadError(error);
+ if (slackError === undefined) throw error;
+ }
+ }
+ if (markdown.length <= SLACK_MARKDOWN_TEXT_MAX_LENGTH) {
+ if (slackError !== undefined) {
+ log.warn(FALLBACK_LOG, { fallback: "inline", slackError, ...logFields });
+ }
+ try {
+ await channel.thread.post(markdown);
+ return;
+ } catch (error) {
+ slackError = refusedPayloadError(error);
+ if (slackError === undefined) throw error;
+ }
+ }
+ if (slackError !== undefined) {
+ log.warn(FALLBACK_LOG, { fallback: "snippet", slackError, ...logFields });
+ }
+ await uploadReplySnippet(channel, markdown);
+}
+
+async function uploadReplySnippet(channel: SlackContext, message: string): Promise<void> {
+ const file = {
+ data: new Blob([message], { type: "text/markdown" }),
+ filename: LONG_RESPONSE_FILENAME,
+ mimeType: "text/markdown",
+ };
+
+ const hasThread = channel.slack.threadTs.length > 0;
+ if (!hasThread) {
+ // Uploads cannot anchor proactive sessions; post the notice first.
+ const anchor = await channel.thread.post(LONG_RESPONSE_NOTICE);
+ if (!anchor.id || channel.slack.threadTs.length === 0) {
+ throw new Error("Slack did not return a thread timestamp for the long response notice.");
+ }
+ }
+ await channel.slack.uploadFiles([file], {
+ initialComment: hasThread ? LONG_RESPONSE_NOTICE : undefined,
+ snippetType: "markdown",
+ });
+}
+
+/** Slack's error code when `error` is a refused-payload error, checking its causes. */
+function refusedPayloadError(error: unknown): string | undefined {
+ let current = error;
+ for (let depth = 0; depth < 5 && typeof current === "object" && current !== null; depth++) {
+ const candidate = current as {
+ readonly cause?: unknown;
+ readonly data?: { readonly error?: unknown };
+ readonly response?: { readonly error?: unknown };
+ };
+ // eve's Slack client reports the code on `response`; `@slack/web-api` on `data`.
+ const code = candidate.response?.error ?? candidate.data?.error;
+ if (typeof code === "string" && REFUSED_PAYLOAD_ERRORS.has(code)) return code;
+ current = candidate.cause;
+ }
+ return undefined;
+}
+
+/**
+ * Wraps a channel's composed `message.completed` chain, authored renderers
+ * included, so a final reply that fails is never lost without a trace.
+ *
+ * When the chain throws on a final reply, eve logs the error at error level,
+ * posts a content-free notice with its error id, and tells the model on the
+ * next delivery that its reply was not seen. eve never sends the reply from
+ * here: the chain owns the content, including any change a renderer made
+ * before `next`, and {@link postCompletedSlackReply} owns the fallbacks.
+ * Steps that end in tool calls and empty replies pass through.
+ */
+export function withFinalReplyDelivery(
+ render: MessageCompletedHandler | undefined,
+): MessageCompletedHandler {
+ return async (event, channel, ctx) => {
+ if (event.finishReason === "tool-calls" || !event.message) {
+ await render?.(event, channel, ctx);
+ return;
+ }
+ try {
+ await render?.(event, channel, ctx);
+ } catch (error) {
+ await reportUndeliveredReply(channel, event.turnId, error);
+ }
+ };
+}
+
+async function reportUndeliveredReply(
+ channel: SlackEventContext,
+ turnId: string,
+ error: unknown,
+): Promise<void> {
+ const errorId = logError(log, "final reply was not delivered to Slack", error, { turnId });
+ channel.state.undeliveredReplyErrorId = errorId;
+ try {
+ await channel.thread.post(
+ `I finished, but couldn't deliver my answer in Slack (error id \`${errorId}\`). Please ask me to resend it.`,
+ );
+ } catch (noticeError) {
+ logError(log, "undelivered reply notice failed", noticeError, { errorId, turnId });
+ }
+}
+
+/**
+ * Tells the model, ahead of the next message, that its previous reply never
+ * reached Slack, so it does not assume the user read that reply.
+ */
+export function withUndeliveredReplyNote<T extends DeliverPayload>(
+ state: SlackChannelState,
+ payload: T,
+): T {
+ const errorId = state.undeliveredReplyErrorId;
+ if (!errorId || payload.message === undefined) return payload;
+ state.undeliveredReplyErrorId = null;
+ const note = [
+ `Your previous reply was not delivered to Slack (error id ${errorId}).`,
+ "The user has not seen it. Do not assume they read it; send the answer again if they ask for it.",
+ ].join(" ");
+ return { ...payload, context: [note, ...(payload.context ?? [])] };
+}
diff --git a/packages/eve/src/public/channels/slack/slackChannel.test.ts b/packages/eve/src/public/channels/slack/slackChannel.test.ts
index 8ef2013d4..53ef8d82c 100644
--- a/packages/eve/src/public/channels/slack/slackChannel.test.ts
+++ b/packages/eve/src/public/channels/slack/slackChannel.test.ts
@@ -35,7 +35,7 @@ import {
SLACK_MESSAGE_TEXT_MAX_LENGTH,
SLACK_SECTION_TEXT_MAX_LENGTH,
} from "#public/channels/slack/limits.js";
-import { defaultSlackAuth } from "#public/channels/slack/index.js";
+import { defaultSlackAuth, postCompletedSlackReply } from "#public/channels/slack/index.js";
import {
composeSlackRenderers,
type SlackRenderChain,
@@ -666,7 +666,8 @@ describe("slackChannel() default event handlers", () => {
it.each([
["files.getUploadURLExternal", "missing_scope"],
["files.completeUploadExternal", "channel_not_found"],
- ])("message.completed propagates %s failure", async (method, error) => {
+ ])("message.completed posts a notice when the snippet's %s call fails", async (method, error) => {
+ const logs = captureLogRecords();
useSuccessfulSlackFileUpload(fetchMock);
const successfulFetch = fetchMock.getMockImplementation() as typeof fetch;
fetchMock.mockImplementation(async (input: string | URL | Request) => {
@@ -690,21 +691,32 @@ describe("slackChannel() default event handlers", () => {
const ctx = buildAdapterContext(adapter, stubAccessor());
const message = "x".repeat(SLACK_MARKDOWN_TEXT_MAX_LENGTH + 1);
- await expect(
- callCompletionHandler(
- adapter,
- makeEvent("message.completed", {
- finishReason: "stop",
- message,
- sequence: 0,
- stepIndex: 0,
- turnId: "t1",
- }),
- ctx,
- ),
- ).rejects.toThrow(error);
+ await callCompletionHandler(
+ adapter,
+ makeEvent("message.completed", {
+ finishReason: "stop",
+ message,
+ sequence: 0,
+ stepIndex: 0,
+ turnId: "t1",
+ }),
+ ctx,
+ );
- expect(fetchMock).toHaveBeenCalledTimes(method === "files.getUploadURLExternal" ? 1 : 3);
+ const failure = logs.records.find(
+ (record) => record.message === "final reply was not delivered to Slack",
+ );
+ expect(failure).toMatchObject({
+ fields: { error: { message: expect.stringContaining(error) } },
+ level: "error",
+ });
+ const { errorId } = failure!.fields!.error as { errorId: string };
+ expect(String(fetchMock.mock.calls.at(-1)![0])).toBe("https://slack.com/api/chat.postMessage");
+ const notice = parseSlackRequestBody(fetchMock.mock.calls.at(-1)![1] as RequestInit);
+ expect(notice).toMatchObject({ channel: "C01", thread_ts: "1700000000.000001" });
+ expect(notice.markdown_text).toContain("couldn't deliver my answer");
+ expect(notice.markdown_text).toContain(errorId);
+ expect(slackCalls(fetchMock, "chat.postMessage")).toHaveLength(1);
});
it("names one working task in the status while a turn waits, and counts several", async () => {
@@ -1832,6 +1844,408 @@ describe("slackChannel() default event handlers", () => {
});
});
+describe("slackChannel() final reply delivery", () => {
+ let fetchMock: ReturnType<typeof vi.fn>;
+ let uploadedBodies: string[];
+ beforeEach(() => {
+ fetchMock = vi.fn();
+ uploadedBodies = useSuccessfulSlackFileUpload(fetchMock);
+ vi.stubGlobal("fetch", fetchMock);
+ });
+
+ // Fails the next calls to each named Slack method with the queued error
+ // codes, in order; every other call succeeds.
+ function failSlackCalls(errors: Record<string, string[]>): void {
+ const succeed = fetchMock.getMockImplementation() as typeof fetch;
+ fetchMock.mockImplementation(async (input: string | URL | Request, init?: RequestInit) => {
+ const method = /^https:\/\/slack\.com\/api\/([\w.]+)$/.exec(String(input))?.[1];
+ const error = method === undefined ? undefined : errors[method]?.shift();
+ if (error !== undefined) {
+ return new Response(JSON.stringify({ error, ok: false }), {
+ headers: { "content-type": "application/json" },
+ });
+ }
+ return succeed(input, init);
+ });
+ }
+
+ function sourcedBlocks(message: string): unknown[] {
+ return [
+ { type: "markdown", text: message },
+ { type: "context", elements: [{ type: "mrkdwn", text: "Sources: 12 documents" }] },
+ ];
+ }
+
+ // Posts every reply as Block Kit with `channel.thread.post`, outside eve's helper.
+ const blockReplies: SlackRenderer = {
+ events: {
+ async "message.completed"(event, channel) {
+ await channel.thread.post({
+ blocks: sourcedBlocks(event.message),
+ text: event.message.slice(0, 100),
+ });
+ },
+ },
+ };
+
+ // Posts final replies as Block Kit through eve's helper, with the Markdown fallback.
+ function helperReplies(thread?: (channel: SlackEventContext) => SlackEventContext["thread"]) {
+ return {
+ events: {
+ async "message.completed"(event, channel, _ctx, next) {
+ if (event.finishReason === "tool-calls" || !event.message) return next();
+ await postCompletedSlackReply(
+ { slack: channel.slack, thread: thread?.(channel) ?? channel.thread },
+ { blocks: sourcedBlocks(event.message), markdown: event.message },
+ );
+ },
+ },
+ } satisfies SlackRenderer;
+ }
+
+ function adapterWith(renderers: readonly SlackRenderer[] = []) {
+ const adapter = withState(
+ getAdapter(slackChannel({ credentials: { botToken: "xoxb-test" }, renderers })),
+ THREAD_STATE,
+ );
+ return { adapter, ctx: buildAdapterContext(adapter, stubAccessor()) };
+ }
+
+ function finalReply(message: string, finishReason = "stop") {
+ return makeEvent("message.completed", {
+ finishReason,
+ message,
+ sequence: 0,
+ stepIndex: 0,
+ turnId: "t1",
+ });
+ }
+
+ function urls(): string[] {
+ return fetchMock.mock.calls.map(([input]) => String(input));
+ }
+
+ function slackBody(index: number): Record<string, unknown> {
+ return parseSlackRequestBody(fetchMock.mock.calls[index]![1] as RequestInit);
+ }
+
+ // Every byte eve sent to Slack: API bodies and uploaded files.
+ function everythingSent(): string {
+ const bodies = fetchMock.mock.calls
+ .filter(([input]) => String(input).startsWith("https://slack.com/api/"))
+ .map(([, init]) => JSON.stringify(parseSlackRequestBody(init as RequestInit)));
+ return [...bodies, ...uploadedBodies].join("\n");
+ }
+
+ function undeliveredErrorId(logs: { records: { message: string; fields?: unknown }[] }) {
+ const record = logs.records.find(
+ (entry) => entry.message === "final reply was not delivered to Slack",
+ );
+ return (record?.fields as { error: { errorId: string } } | undefined)?.error.errorId;
+ }
+
+ // The fallbacks eve logged, in order.
+ function fallbacks(logs: { records: { message: string; fields?: unknown }[] }) {
+ return logs.records
+ .filter(
+ (entry) => entry.message === "Slack refused the final reply; delivering it with a fallback",
+ )
+ .map((entry) => entry.fields);
+ }
+
+ const shortAnswer = `# Findings\n\n${"r".repeat(6_000)}`;
+ const longAnswer = `# Findings\n\n${"r".repeat(SLACK_MARKDOWN_TEXT_MAX_LENGTH)}`;
+
+ describe("postCompletedSlackReply", () => {
+ it("posts the caller's Markdown once when Slack refuses the blocks", async () => {
+ const logs = captureLogRecords();
+ failSlackCalls({ "chat.postMessage": ["msg_blocks_too_long"] });
+ const { adapter, ctx } = adapterWith([helperReplies()]);
+
+ await callCompletionHandler(adapter, finalReply(shortAnswer), ctx);
+
+ expect(slackOperations(fetchMock)).toEqual(["chat.postMessage", "chat.postMessage"]);
+ expect(slackBody(0)).toMatchObject({ blocks: sourcedBlocks(shortAnswer), text: shortAnswer });
+ expect(slackBody(1)).toMatchObject({
+ markdown_text: shortAnswer,
+ thread_ts: "1700000000.000001",
+ });
+ expect(slackBody(1).blocks).toBeUndefined();
+ expect(fallbacks(logs)).toEqual([{ fallback: "inline", slackError: "msg_blocks_too_long" }]);
+ expect(JSON.stringify(logs.records)).not.toContain(shortAnswer);
+ expect(ctx.state.undeliveredReplyErrorId).toBeUndefined();
+ });
+
+ it("uploads Markdown over the limit as a snippet when Slack refuses the blocks", async () => {
+ const logs = captureLogRecords();
+ failSlackCalls({ "chat.postMessage": ["msg_blocks_too_long"] });
+ const { adapter, ctx } = adapterWith([helperReplies()]);
+
+ await callCompletionHandler(adapter, finalReply(longAnswer), ctx);
+
+ expect(urls()).toEqual([
+ "https://slack.com/api/chat.postMessage",
+ "https://slack.com/api/files.getUploadURLExternal",
+ "https://files.slack.com/upload/F01",
+ "https://slack.com/api/files.completeUploadExternal",
+ ]);
+ expect(uploadedBodies).toEqual([longAnswer]);
+ expect(slackBody(3)).toMatchObject({
+ initial_comment: "Here's a snippet with the full response.",
+ thread_ts: "1700000000.000001",
+ });
+ expect(fallbacks(logs)).toEqual([{ fallback: "snippet", slackError: "msg_blocks_too_long" }]);
+ });
+
+ it("uploads a snippet when Slack refuses the Markdown message too", async () => {
+ const logs = captureLogRecords();
+ failSlackCalls({ "chat.postMessage": ["msg_blocks_too_long", "msg_too_long"] });
+ const { adapter, ctx } = adapterWith([helperReplies()]);
+
+ await callCompletionHandler(adapter, finalReply(shortAnswer), ctx);
+
+ expect(urls()).toEqual([
+ "https://slack.com/api/chat.postMessage",
+ "https://slack.com/api/chat.postMessage",
+ "https://slack.com/api/files.getUploadURLExternal",
+ "https://files.slack.com/upload/F01",
+ "https://slack.com/api/files.completeUploadExternal",
+ ]);
+ expect(uploadedBodies).toEqual([shortAnswer]);
+ expect(fallbacks(logs)).toEqual([
+ { fallback: "inline", slackError: "msg_blocks_too_long" },
+ { fallback: "snippet", slackError: "msg_too_long" },
+ ]);
+ });
+
+ it("throws an error other than a refusal without a fallback", async () => {
+ captureLogRecords();
+ failSlackCalls({ "chat.postMessage": ["not_in_channel"] });
+ const thrown: unknown[] = [];
+ const { adapter, ctx } = adapterWith([
+ {
+ events: {
+ async "message.completed"(event, channel, ctx, next) {
+ try {
+ await helperReplies().events["message.completed"](event, channel, ctx, next);
+ } catch (error) {
+ thrown.push(error);
+ throw error;
+ }
+ },
+ },
+ },
+ ]);
+
+ await callCompletionHandler(adapter, finalReply(shortAnswer), ctx);
+
+ expect(thrown).toEqual([
+ expect.objectContaining({ message: expect.stringContaining("not_in_channel") }),
+ ]);
+ // The refused blocks, then eve's notice: no Markdown post and no upload.
+ expect(slackOperations(fetchMock)).toEqual(["chat.postMessage", "chat.postMessage"]);
+ expect(String(slackBody(1).markdown_text)).toContain("couldn't deliver my answer");
+ });
+
+ it.each([
+ [
+ "`data.error`, as `@slack/web-api` throws it",
+ () =>
+ Object.assign(new Error("An API error occurred: msg_blocks_too_long"), {
+ code: "slack_webapi_platform_error",
+ data: { error: "msg_blocks_too_long", ok: false },
+ }),
+ ],
+ [
+ "a refusal wrapped in `cause`",
+ () =>
+ new Error("posting the reply failed", {
+ cause: Object.assign(new Error("Slack chat.postMessage failed"), {
+ response: { error: "msg_blocks_too_long", ok: false },
+ }),
+ }),
+ ],
+ ])("recognizes a refusal reported through %s", async (_name, refusal) => {
+ const logs = captureLogRecords();
+ // A thread whose first post throws the refusal; later posts reach Slack.
+ const refusingThread = (channel: SlackEventContext): SlackEventContext["thread"] => {
+ let refused = false;
+ return Object.create(channel.thread, {
+ post: {
+ value: (message: Parameters<SlackEventContext["thread"]["post"]>[0]) => {
+ if (refused) return channel.thread.post(message);
+ refused = true;
+ return Promise.reject(refusal());
+ },
+ },
+ });
+ };
+ const { adapter, ctx } = adapterWith([helperReplies(refusingThread)]);
+
+ await callCompletionHandler(adapter, finalReply(longAnswer), ctx);
+
+ expect(urls()).toEqual([
+ "https://slack.com/api/files.getUploadURLExternal",
+ "https://files.slack.com/upload/F01",
+ "https://slack.com/api/files.completeUploadExternal",
+ ]);
+ expect(uploadedBodies).toEqual([longAnswer]);
+ expect(fallbacks(logs)).toEqual([{ fallback: "snippet", slackError: "msg_blocks_too_long" }]);
+ });
+
+ it("uploads a string reply over the limit without trying a message", async () => {
+ const { adapter, ctx } = adapterWith([
+ {
+ events: {
+ async "message.completed"(event, channel) {
+ await postCompletedSlackReply(channel, event.message);
+ },
+ },
+ },
+ ]);
+
+ await callCompletionHandler(adapter, finalReply(longAnswer), ctx);
+
+ expect(urls()).toEqual([
+ "https://slack.com/api/files.getUploadURLExternal",
+ "https://files.slack.com/upload/F01",
+ "https://slack.com/api/files.completeUploadExternal",
+ ]);
+ expect(uploadedBodies).toEqual([longAnswer]);
+ });
+ });
+
+ describe("default renderer", () => {
+ it("uploads a refused Markdown reply as a snippet without posting it again", async () => {
+ const logs = captureLogRecords();
+ failSlackCalls({ "chat.postMessage": ["msg_too_long"] });
+ const { adapter, ctx } = adapterWith();
+
+ await callCompletionHandler(adapter, finalReply(shortAnswer), ctx);
+
+ expect(urls()).toEqual([
+ "https://slack.com/api/chat.postMessage",
+ "https://slack.com/api/files.getUploadURLExternal",
+ "https://files.slack.com/upload/F01",
+ "https://slack.com/api/files.completeUploadExternal",
+ ]);
+ expect(slackBody(0)).toMatchObject({ markdown_text: shortAnswer });
+ expect(uploadedBodies).toEqual([shortAnswer]);
+ expect(fallbacks(logs)).toEqual([
+ { fallback: "snippet", slackError: "msg_too_long", turnId: "t1" },
+ ]);
+ expect(ctx.state.undeliveredReplyErrorId).toBeUndefined();
+ });
+
+ it("falls back with the reply a renderer handed to `next`, never the original", async () => {
+ const logs = captureLogRecords();
+ failSlackCalls({ "chat.postMessage": ["msg_too_long"] });
+ const secret = "sk-live-4f9a2c";
+ const { adapter, ctx } = adapterWith([
+ {
+ events: {
+ async "message.completed"(event, _channel, _ctx, next) {
+ await next({ ...event, message: event.message.replaceAll(secret, "[redacted]") });
+ },
+ },
+ },
+ ]);
+
+ await callCompletionHandler(
+ adapter,
+ finalReply(`The deploy key is ${secret}.\n\n${"r".repeat(6_000)}`),
+ ctx,
+ );
+
+ expect(everythingSent()).not.toContain(secret);
+ expect(JSON.stringify(logs.records)).not.toContain(secret);
+ expect(uploadedBodies).toEqual([`The deploy key is [redacted].\n\n${"r".repeat(6_000)}`]);
+ });
+ });
+
+ describe("authored renderer failure", () => {
+ it("posts a notice instead of the reply, logs the error, and tells the model next turn", async () => {
+ const logs = captureLogRecords();
+ failSlackCalls({ "chat.postMessage": ["msg_blocks_too_long"] });
+ const { adapter, ctx } = adapterWith([blockReplies]);
+
+ await callCompletionHandler(adapter, finalReply(shortAnswer), ctx);
+
+ const errorId = undeliveredErrorId(logs);
+ expect(errorId).toEqual(expect.any(String));
+ // The renderer's refused post, then the notice: eve never re-sends the reply.
+ expect(urls()).toEqual([
+ "https://slack.com/api/chat.postMessage",
+ "https://slack.com/api/chat.postMessage",
+ ]);
+ expect(slackBody(1).markdown_text).toBe(
+ `I finished, but couldn't deliver my answer in Slack (error id \`${errorId}\`). Please ask me to resend it.`,
+ );
+ expect(slackBody(1).blocks).toBeUndefined();
+ expect(logs.records).toContainEqual(
+ expect.objectContaining({
+ fields: expect.objectContaining({
+ error: expect.objectContaining({
+ detail: expect.stringContaining("msg_blocks_too_long"),
+ name: "SlackApiError",
+ }),
+ turnId: "t1",
+ }),
+ level: "error",
+ message: "final reply was not delivered to Slack",
+ }),
+ );
+ expect(fallbacks(logs)).toEqual([]);
+ expect(ctx.state.undeliveredReplyErrorId).toBe(errorId);
+
+ const next = await adapter.deliver!({ context: ["thread context"], message: "Hello?" }, ctx);
+ expect(next?.context).toEqual([
+ expect.stringContaining(
+ `Your previous reply was not delivered to Slack (error id ${errorId})`,
+ ),
+ "thread context",
+ ]);
+ expect(next?.message).toBe("Hello?");
+ const after = await adapter.deliver!({ message: "And now?" }, ctx);
+ expect(after?.context).toBeUndefined();
+ });
+
+ it("logs and returns when the notice fails too", async () => {
+ const logs = captureLogRecords();
+ failSlackCalls({ "chat.postMessage": ["msg_blocks_too_long", "channel_not_found"] });
+ const { adapter, ctx } = adapterWith([blockReplies]);
+
+ await callCompletionHandler(adapter, finalReply(shortAnswer), ctx);
+
+ const errorId = undeliveredErrorId(logs);
+ expect(logs.records).toContainEqual(
+ expect.objectContaining({
+ fields: expect.objectContaining({ errorId }),
+ level: "error",
+ message: "undelivered reply notice failed",
+ }),
+ );
+ expect(slackOperations(fetchMock)).toEqual(["chat.postMessage", "chat.postMessage"]);
+ expect(ctx.state.undeliveredReplyErrorId).toBe(errorId);
+ });
+
+ it.each([
+ ["a step that ends in tool calls", "Checking the docs.", "tool-calls"],
+ ["an empty reply", "", "stop"],
+ ])("leaves a failure on %s to the adapter", async (_name, message, finishReason) => {
+ failSlackCalls({ "chat.postMessage": ["msg_blocks_too_long"] });
+ const { adapter, ctx } = adapterWith([blockReplies]);
+
+ await expect(
+ callCompletionHandler(adapter, finalReply(message, finishReason), ctx),
+ ).rejects.toThrow("msg_blocks_too_long");
+
+ expect(slackOperations(fetchMock)).toEqual(["chat.postMessage"]);
+ expect(ctx.state.undeliveredReplyErrorId).toBeUndefined();
+ });
+ });
+});
+
describe("rebuildSlackContext", () => {
beforeEach(() => {
vi.stubGlobal("fetch", vi.fn());
@@ -2038,22 +2452,32 @@ describe("rebuildSlackContext", () => {
},
);
const ctx = buildAdapterContext(adapter, stubAccessor());
+ const logs = captureLogRecords();
- await expect(
- callCompletionHandler(
- adapter,
- makeEvent("message.completed", {
- finishReason: "stop",
- message: "x".repeat(SLACK_MARKDOWN_TEXT_MAX_LENGTH + 1),
- sequence: 0,
- stepIndex: 0,
- turnId: "t1",
- }),
- ctx,
- ),
- ).rejects.toThrow("Slack did not return a thread timestamp");
+ await callCompletionHandler(
+ adapter,
+ makeEvent("message.completed", {
+ finishReason: "stop",
+ message: "x".repeat(SLACK_MARKDOWN_TEXT_MAX_LENGTH + 1),
+ sequence: 0,
+ stepIndex: 0,
+ turnId: "t1",
+ }),
+ ctx,
+ );
- expect(fetchMock).toHaveBeenCalledTimes(1);
+ // The anchor note, then the delivery notice; no upload without a thread.
+ expect(slackOperations(fetchMock)).toEqual(["chat.postMessage", "chat.postMessage"]);
+ expect(logs.records).toContainEqual(
+ expect.objectContaining({
+ fields: expect.objectContaining({
+ error: expect.objectContaining({
+ message: expect.stringContaining("Slack did not return a thread timestamp"),
+ }),
+ }),
+ level: "error",
+ }),
+ );
});
});
diff --git a/packages/eve/src/public/channels/slack/slackChannel.ts b/packages/eve/src/public/channels/slack/slackChannel.ts
index 360d7ce62..5ced6533b 100644
--- a/packages/eve/src/public/channels/slack/slackChannel.ts
+++ b/packages/eve/src/public/channels/slack/slackChannel.ts
@@ -47,6 +47,10 @@ import {
defaultOnMessage,
defaultReceived,
} from "#public/channels/slack/defaults.js";
+import {
+ withFinalReplyDelivery,
+ withUndeliveredReplyNote,
+} from "#public/channels/slack/reply-delivery.js";
import {
composeSlackRenderers,
type SlackRenderChain,
@@ -261,6 +265,11 @@ export interface SlackChannelState {
* `authorization.required` and approval events.
*/
slackUsersByPrincipal?: Record<string, string>;
+ /**
+ * Error id of a final reply that never reached Slack. The next delivered
+ * message tells the model the reply was not seen, then clears it.
+ */
+ undeliveredReplyErrorId?: string | null;
}
/**
@@ -824,6 +833,7 @@ export function slackChannel(config: SlackChannelConfig = {}): SlackChannel {
const events = withTaskCards(
{
...rendering.events,
+ "message.completed": withFinalReplyDelivery(rendering.events["message.completed"]),
async "turn.started"(data, channel, ctx) {
const triggeringUserId = slackUserIdFromAuthContext(ctx.session.auth.current);
if (triggeringUserId !== undefined) {
@@ -880,7 +890,7 @@ export function slackChannel(config: SlackChannelConfig = {}): SlackChannel {
// is the caller of this delivery.
deliver(payload, channel: SlackChannelContext & { readonly session: SessionHandle }) {
recordSlackPrincipal(channel.state, channel.session.auth.current, payload);
- return defaultDeliverResult(payload);
+ return defaultDeliverResult(withUndeliveredReplyNote(channel.state, payload));
},
routes: [