Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

fix(photon): download voice notes the webhook sends without an id (#5052)

Ben Pankow4d9fc9f563bf
diff --git a/.changeset/photon-undownloadable-attachments.md b/.changeset/photon-undownloadable-attachments.md
new file mode 100644
index 000000000..fd13aef8d
--- /dev/null
+++ b/.changeset/photon-undownloadable-attachments.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+iMessage voice notes now reach Photon agents. Photon's webhook leaves out a voice note's attachment id, so eve refetches the message by id to download it. Across Chat SDK channels, an attachment the platform gives no way to download now reaches the agent as a short note instead of being dropped, so an attachment-only message still gets a reply.
diff --git a/docs/channels/chat-sdk.mdx b/docs/channels/chat-sdk.mdx
index 3f16b99b6..6a89616cc 100644
--- a/docs/channels/chat-sdk.mdx
+++ b/docs/channels/chat-sdk.mdx
@@ -245,7 +245,7 @@ bot.onNewMention(async (thread, message) => {
});
```
-When the adapter can download an attachment (it sets `fetchData`, as the Slack, Google Chat, WhatsApp, and Twilio adapters do), eve downloads it with the adapter's credentials when the turn runs, not in the webhook, and passes the bytes. Platform file URLs usually need those credentials, so the model provider couldn't open them. eve rebuilds the download with the adapter's `rehydrateAttachment`; an adapter without one can't download after the webhook returns, so its files reach the agent as a short note. A failed download, or a file over 25 MB, also becomes a note. An attachment with only a URL is downloaded by eve when the URL is public `https:`; otherwise it becomes a note.
+When the adapter can download an attachment (it sets `fetchData`, as the Slack, Google Chat, WhatsApp, and Twilio adapters do), eve downloads it with the adapter's credentials when the turn runs, not in the webhook, and passes the bytes. Platform file URLs usually need those credentials, so the model provider couldn't open them. eve rebuilds the download with the adapter's `rehydrateAttachment`; an adapter without one can't download after the webhook returns, so its files reach the agent as a short note. A failed download, or a file over 25 MB, also becomes a note. An attachment with only a URL is downloaded by eve when the URL is public `https:`; otherwise it becomes a note. An attachment with neither a download nor a URL also becomes a note, so the agent still knows a file was sent.
See [File uploads](./custom#file-uploads) for how eve stages remote file URLs before the model call.
diff --git a/docs/channels/photon.mdx b/docs/channels/photon.mdx
index 6122fb161..08c86d559 100644
--- a/docs/channels/photon.mdx
+++ b/docs/channels/photon.mdx
@@ -34,8 +34,10 @@ Set `turnPolicy: "queue"` when every response should finish before Photon starts
Images and other files people send reach the agent as file parts. Photon's
webhook lists each file without its bytes, so eve downloads it from Photon by
-its attachment id with the project's credentials. A failed download, or a file
-over 25 MB, reaches the agent as a short note.
+its attachment id with the project's credentials. The webhook omits the id
+for voice notes, so eve refetches those messages from Photon to get it. A
+failed download, a file over 25 MB, or a file eve can't find an id for reaches
+the agent as a short note.
Customize inbound dispatch with `onMessage`. Return `null` to ignore a message, or return `title` alongside `auth` to set the title when the dispatch starts a run:
diff --git a/packages/eve/src/public/channels/chat-sdk/chatSdkChannel.test.ts b/packages/eve/src/public/channels/chat-sdk/chatSdkChannel.test.ts
index 65dfc4218..e0298a601 100644
--- a/packages/eve/src/public/channels/chat-sdk/chatSdkChannel.test.ts
+++ b/packages/eve/src/public/channels/chat-sdk/chatSdkChannel.test.ts
@@ -1136,36 +1136,35 @@ describe("messageToUserContent", () => {
expect((parts[1] as { data: URL }).data.href).toBe("https://example.com/a.pdf");
});
- it("skips attachments without a URL, keeping the text part", () => {
- const withUrllessAttachment = new Message({
- attachments: [{ name: "pasted", type: "image" }],
+ it("notes an attachment without a download in place, keeping the other parts", () => {
+ const mixed = new Message({
+ attachments: [
+ { name: "pasted", type: "image" },
+ {
+ mimeType: "application/pdf",
+ name: "a.pdf",
+ type: "file",
+ url: "https://example.com/a.pdf",
+ },
+ ],
author: author(),
- formatted: parseMarkdown("no url"),
+ formatted: parseMarkdown("see both"),
id: "message-3",
isMention: true,
metadata: metadata(),
- raw: { text: "no url" },
- text: "no url",
- threadId: THREAD_ID,
- });
-
- expect(messageToUserContent(withUrllessAttachment)).toEqual([{ text: "no url", type: "text" }]);
- });
-
- it("falls back to text when there are no usable parts", () => {
- const emptyWithUrllessAttachment = new Message({
- attachments: [{ name: "pasted", type: "image" }],
- author: author(),
- formatted: parseMarkdown(""),
- id: "message-4",
- isMention: true,
- metadata: metadata(),
- raw: { text: "" },
- text: "",
+ raw: { text: "see both" },
+ text: "see both",
threadId: THREAD_ID,
});
- expect(messageToUserContent(emptyWithUrllessAttachment)).toBe("");
+ expect(messageToUserContent(mixed)).toEqual([
+ { text: "see both", type: "text" },
+ {
+ text: "Attachment pasted was not retrieved: the platform did not provide a download for it.",
+ type: "text",
+ },
+ expect.objectContaining({ filename: "a.pdf", type: "file" }),
+ ]);
});
});
diff --git a/packages/eve/src/public/channels/chat-sdk/messageToUserContent.ts b/packages/eve/src/public/channels/chat-sdk/messageToUserContent.ts
index f394d9ceb..71fc8f187 100644
--- a/packages/eve/src/public/channels/chat-sdk/messageToUserContent.ts
+++ b/packages/eve/src/public/channels/chat-sdk/messageToUserContent.ts
@@ -20,7 +20,8 @@ type UserContentParts = Exclude<UserContent, string>;
* it. The channel rebuilds the download with the adapter's
* `rehydrateAttachment`. A file over 25 MB, or one that fails to download,
* reaches the agent as a short note. An attachment with only a URL is passed as
- * that URL. Attachments with neither are skipped.
+ * that URL. An attachment with neither also becomes a note, so a file the
+ * platform gave no way to download is never dropped silently.
*/
export function messageToUserContent(message: Message): string | UserContent {
const attachments = message.attachments ?? [];
@@ -35,19 +36,20 @@ export function messageToUserContent(message: Message): string | UserContent {
// Chat SDK thread ids start with the name of the adapter that owns them.
const adapter = message.threadId.split(":")[0]!;
for (const attachment of attachments) {
- const part = attachmentToPart(adapter, attachment);
- if (part !== null) parts.push(part);
+ parts.push(attachmentToPart(adapter, attachment));
}
- return parts.length > 0 ? parts : message.text;
+ return parts;
}
-function attachmentToPart(
- adapter: string,
- attachment: Attachment,
-): UserContentParts[number] | null {
+function attachmentToPart(adapter: string, attachment: Attachment): UserContentParts[number] {
const mediaType = attachment.mimeType ?? "application/octet-stream";
if (attachment.fetchData === undefined) {
- if (!attachment.url) return null;
+ if (!attachment.url) {
+ return {
+ text: `Attachment ${attachment.name ?? "file"} was not retrieved: the platform did not provide a download for it.`,
+ type: "text",
+ };
+ }
return { data: new URL(attachment.url), filename: attachment.name, mediaType, type: "file" };
}
if (attachment.size !== undefined && attachment.size > DEFAULT_UPLOAD_POLICY.maxBytes) {
diff --git a/packages/eve/src/public/channels/photon/attachments.test.ts b/packages/eve/src/public/channels/photon/attachments.test.ts
index 2062c9c80..150196c8f 100644
--- a/packages/eve/src/public/channels/photon/attachments.test.ts
+++ b/packages/eve/src/public/channels/photon/attachments.test.ts
@@ -19,7 +19,8 @@ function message(raw: unknown, names: readonly string[]): Message {
});
}
-function appServing(files: Record<string, string>) {
+/** A Photon app serving `files` by attachment id, whose `fetchMessage` returns `refetchedRaw`. */
+function appServing(files: Record<string, string>, refetchedRaw?: unknown) {
const getAttachment = vi.fn(async (_ctx: unknown, id: string, _phone?: string) =>
files[id] === undefined ? undefined : { read: async () => Buffer.from(files[id]!) },
);
@@ -31,10 +32,15 @@ function appServing(files: Record<string, string>) {
const app = {
__internal: { platforms: new Map([["iMessage", { ...runtime, store: "store" }]]) },
};
- const adapter: { app: unknown; rehydrateAttachment?(attachment: Attachment): Attachment } = {
- app,
- };
- return { adapter, getAttachment };
+ const fetchMessage = vi.fn(async (_threadId: string, _messageId: string) =>
+ refetchedRaw === undefined ? null : message(refetchedRaw, []),
+ );
+ const adapter: {
+ app: unknown;
+ fetchMessage: typeof fetchMessage;
+ rehydrateAttachment?(attachment: Attachment): Attachment;
+ } = { app, fetchMessage };
+ return { adapter, fetchMessage, getAttachment };
}
describe("addPhotonAttachmentDownloads", () => {
@@ -60,7 +66,7 @@ describe("addPhotonAttachmentDownloads", () => {
["one.jpg", "two.jpg"],
);
- addPhotonAttachmentDownloads(inbound, adapter);
+ await addPhotonAttachmentDownloads(inbound, adapter);
const [one, two] = inbound.attachments;
expect(await one?.fetchData?.()).toEqual(Buffer.from("first"));
@@ -72,11 +78,73 @@ describe("addPhotonAttachmentDownloads", () => {
);
});
+ it("refetches the message by id when the webhook omits a voice note's id", async () => {
+ const { adapter, fetchMessage, getAttachment } = appServing(
+ { "v-1": "audio" },
+ { content: { id: "v-1", type: "attachment" } },
+ );
+ const inbound = message(
+ { content: { type: "voice" }, space: { id: "chat", phone: "+15550100" } },
+ ["voice"],
+ );
+
+ await addPhotonAttachmentDownloads(inbound, adapter);
+
+ expect(fetchMessage).toHaveBeenCalledWith("thread-id", "message-id");
+ expect(inbound.attachments[0]?.fetchMetadata).toEqual({ id: "v-1", phone: "+15550100" });
+ expect(await inbound.attachments[0]?.fetchData?.()).toEqual(Buffer.from("audio"));
+ expect(getAttachment).toHaveBeenCalledWith(expect.anything(), "v-1", "+15550100");
+ });
+
+ it("keeps the webhook's ids and fills only the missing ones from the refetch", async () => {
+ // The refetched copy is shaped differently: its photo has no id.
+ const { adapter } = appServing(
+ { "p-1": "photo", "v-1": "audio" },
+ {
+ content: {
+ items: [
+ { content: { type: "attachment" } },
+ { content: { id: "v-1", type: "attachment" } },
+ ],
+ type: "group",
+ },
+ },
+ );
+ const inbound = message(
+ {
+ content: {
+ items: [
+ { content: { id: "p-1", name: "photo.jpg", type: "attachment" } },
+ { content: { type: "voice" } },
+ ],
+ type: "group",
+ },
+ },
+ ["photo.jpg", "voice"],
+ );
+
+ await addPhotonAttachmentDownloads(inbound, adapter);
+
+ const [photo, voice] = inbound.attachments;
+ expect(await photo?.fetchData?.()).toEqual(Buffer.from("photo"));
+ expect(await voice?.fetchData?.()).toEqual(Buffer.from("audio"));
+ });
+
+ it("leaves an attachment without a download when the refetch fails", async () => {
+ const { adapter, fetchMessage } = appServing({});
+ fetchMessage.mockRejectedValueOnce(new Error("Photon is unavailable"));
+ const inbound = message({ content: { type: "voice" } }, ["voice"]);
+
+ await addPhotonAttachmentDownloads(inbound, adapter);
+
+ expect(inbound.attachments[0]?.fetchData).toBeUndefined();
+ });
+
it("fails the download when iMessage no longer has the attachment", async () => {
const { adapter } = appServing({});
const inbound = message({ content: { id: "gone", type: "attachment" } }, ["gone.jpg"]);
- addPhotonAttachmentDownloads(inbound, adapter);
+ await addPhotonAttachmentDownloads(inbound, adapter);
await expect(inbound.attachments[0]?.fetchData?.()).rejects.toThrow(
"iMessage attachment gone was not found.",
@@ -89,7 +157,7 @@ describe("addPhotonAttachmentDownloads", () => {
{ content: { id: "a-1", type: "attachment" }, space: { id: "chat", phone: "+15550100" } },
["one.jpg"],
);
- addPhotonAttachmentDownloads(inbound, adapter);
+ await addPhotonAttachmentDownloads(inbound, adapter);
addPhotonAttachmentRehydration(adapter);
const { fetchData: _dropped, ...serialized } = inbound.attachments[0]!;
diff --git a/packages/eve/src/public/channels/photon/attachments.ts b/packages/eve/src/public/channels/photon/attachments.ts
index cff062183..cf3623ac7 100644
--- a/packages/eve/src/public/channels/photon/attachments.ts
+++ b/packages/eve/src/public/channels/photon/attachments.ts
@@ -45,6 +45,8 @@ interface SpectrumApp {
/** The adapter that owns the spectrum-ts app eve downloads attachments through. */
interface PhotonAdapter {
readonly app: unknown;
+ /** Asks Photon for the message by id; webhook deliveries are never cached, so this refetches. */
+ fetchMessage(threadId: string, messageId: string): Promise<Message | null>;
rehydrateAttachment?(attachment: Attachment): Attachment;
}
@@ -53,14 +55,24 @@ interface PhotonAdapter {
* webhook lists an attachment's name and type but never its bytes, so this
* records each attachment's id in its `fetchMetadata` and gives it a
* `fetchData`; the channel downloads it later, in the step, through
- * {@link addPhotonAttachmentRehydration}.
+ * {@link addPhotonAttachmentRehydration}. The webhook omits the id for some
+ * files, such as voice notes, so those come from the message refetched by id.
*/
-export function addPhotonAttachmentDownloads(message: Message, adapter: PhotonAdapter): void {
+export async function addPhotonAttachmentDownloads(
+ message: Message,
+ adapter: PhotonAdapter,
+): Promise<void> {
const attachments = message.attachments ?? [];
if (attachments.length === 0) return;
const raw = message.raw as { readonly content?: SpectrumContent; readonly space?: unknown };
// The adapter lists attachments in the order it walks the content, so their ids line up.
const ids = contentAttachmentIds(raw.content);
+ if (attachments.some((_, index) => ids[index] === undefined)) {
+ const refetched = await refetchedAttachmentIds(adapter, message);
+ attachments.forEach((_, index) => {
+ ids[index] ??= refetched[index];
+ });
+ }
const phone = spacePhone(raw.space);
attachments.forEach((attachment: Attachment, index) => {
const id = ids[index];
@@ -83,6 +95,21 @@ export function addPhotonAttachmentRehydration(adapter: PhotonAdapter): void {
};
}
+/** Attachment ids from the message as Photon serves it by id; empty when it can't. */
+async function refetchedAttachmentIds(
+ adapter: PhotonAdapter,
+ message: Message,
+): Promise<(string | undefined)[]> {
+ try {
+ const refetched = await adapter.fetchMessage(message.threadId, message.id);
+ const raw = refetched?.raw as { readonly content?: SpectrumContent } | undefined;
+ return contentAttachmentIds(raw?.content);
+ } catch {
+ // An attachment without an id reaches the agent as a note.
+ return [];
+ }
+}
+
async function downloadAttachment(app: unknown, id: string, phone?: string): Promise<Buffer> {
const runtime = (app as SpectrumApp | null)?.__internal?.platforms.get(IMESSAGE_PLATFORM);
const getAttachment = runtime?.definition.actions?.getAttachment;
diff --git a/packages/eve/src/public/channels/photon/inboundContent.test.ts b/packages/eve/src/public/channels/photon/inboundContent.test.ts
index fc1d80e24..7986f5c3a 100644
--- a/packages/eve/src/public/channels/photon/inboundContent.test.ts
+++ b/packages/eve/src/public/channels/photon/inboundContent.test.ts
@@ -25,18 +25,16 @@ describe("photonInboundContent", () => {
expect(photonInboundContent(message(" \n"))).toBeUndefined();
});
- it("drops attachment-only messages when Photon provides no attachment URL", () => {
+ it("notes an attachment-only message Photon sends without a download", () => {
expect(
photonInboundContent(
- message("", [
- {
- mimeType: "image/jpeg",
- name: "photo.jpg",
- size: 10,
- type: "image",
- },
- ]),
+ message("", [{ mimeType: "audio/x-caf", name: "voice", size: 10, type: "audio" }]),
),
- ).toBeUndefined();
+ ).toEqual([
+ {
+ text: "Attachment voice was not retrieved: the platform did not provide a download for it.",
+ type: "text",
+ },
+ ]);
});
});
diff --git a/packages/eve/src/public/channels/photon/photonIMessageChannel.ts b/packages/eve/src/public/channels/photon/photonIMessageChannel.ts
index 0bda13a52..d114a65d0 100644
--- a/packages/eve/src/public/channels/photon/photonIMessageChannel.ts
+++ b/packages/eve/src/public/channels/photon/photonIMessageChannel.ts
@@ -143,7 +143,7 @@ async function dispatchMessage(
if (result === null) return;
const adapter = bridge.bot.getAdapter("imessage");
await markReadBestEffort(adapter, thread, message);
- addPhotonAttachmentDownloads(message, adapter);
+ await addPhotonAttachmentDownloads(message, adapter);
const content = photonInboundContent(message);
if (content === undefined) return;
await bridge.send(content, {