Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

docs: add voice notes pattern for transcribing audio attachments (#5051)

Ben Pankow543820da89a0
diff --git a/docs/README.md b/docs/README.md
index b10ae11d6..002f0843a 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -25,6 +25,7 @@ Important naming note:
| Package a procedure the agent loads only when it applies | [Skills](./skills.mdx) |
| Carry state across turns, or shape what the model sees | [State](./concepts/state.md), [Context Control](./concepts/context-control.md) |
| Run commands or untrusted code in isolation | [Sandboxes](./sandbox/index.mdx) |
+| Answer voice notes and other audio attachments | [Voice Notes](./patterns/voice-notes.md) |
| Delegate work to a specialist child agent | [Subagents](./subagents/index.mdx) |
| Run work on a recurring schedule | [Schedules](./schedules.mdx) |
| Install an existing integration instead of writing one | [Add Integrations](./install-integrations.mdx) |
diff --git a/docs/channels/chat-sdk.mdx b/docs/channels/chat-sdk.mdx
index 6a89616cc..2cd6b2aa0 100644
--- a/docs/channels/chat-sdk.mdx
+++ b/docs/channels/chat-sdk.mdx
@@ -247,6 +247,8 @@ bot.onNewMention(async (thread, message) => {
When the adapter can download an attachment (it sets `fetchData`, as the Slack, Google Chat, WhatsApp, and Twilio adapters do), eve downloads it with the adapter's credentials when the turn runs, not in the webhook, and passes the bytes. Platform file URLs usually need those credentials, so the model provider couldn't open them. eve rebuilds the download with the adapter's `rehydrateAttachment`; an adapter without one can't download after the webhook returns, so its files reach the agent as a short note. A failed download, or a file over 25 MB, also becomes a note. An attachment with only a URL is downloaded by eve when the URL is public `https:`; otherwise it becomes a note. An attachment with neither a download nor a URL also becomes a note, so the agent still knows a file was sent.
+Audio attachments, such as voice notes, reach the model as a file label, not as sound. To answer them, give the agent a transcription tool; see [Voice notes](../patterns/voice-notes).
+
See [File uploads](./custom#file-uploads) for how eve stages remote file URLs before the model call.
## Configuration reference
diff --git a/docs/channels/slack.mdx b/docs/channels/slack.mdx
index 5d3d5392c..fff12a8e1 100644
--- a/docs/channels/slack.mdx
+++ b/docs/channels/slack.mdx
@@ -750,6 +750,8 @@ The agent receives the files attached to the message that reaches it. When a men
Images, PDFs, audio, video, and other files all reach the agent. See [Inbound attachments](../sandbox#inbound-attachments) for which files the model reads directly and which it opens with tools. Files shared through an integration such as Google Drive or Dropbox are links to the other service, not Slack files, so eve skips them; their link stays in the message text. A download that fails or takes longer than 30 seconds reaches the model as a note.
+Slack voice clips reach the model as an audio file label, not as sound. To answer them, give the agent a transcription tool; see [Voice notes](../patterns/voice-notes).
+
Private file downloads resolve credentials from the session's app installation workspace and require the Slack bot token's `files:read` scope. Add the scope and reinstall the Slack app before sending attachments.
Downloads cover Slack's own file hosts plus URLs under a configured [`api.fileBaseUrl`](#point-the-channel-at-a-different-slack-api-host), and travel on `api.fetch` when one is configured.
diff --git a/docs/patterns/meta.json b/docs/patterns/meta.json
index 65a9a1d3c..39875277c 100644
--- a/docs/patterns/meta.json
+++ b/docs/patterns/meta.json
@@ -6,6 +6,7 @@
"durable-cross-channel-notifications",
"multi-tenant-auth",
"multi-tenant-approvals",
+ "voice-notes",
"../protocols/ucp"
]
}
diff --git a/docs/patterns/voice-notes.md b/docs/patterns/voice-notes.md
new file mode 100644
index 000000000..a00735a7c
--- /dev/null
+++ b/docs/patterns/voice-notes.md
@@ -0,0 +1,67 @@
+---
+title: "Voice Notes"
+description: "Transcribe voice notes, voice messages, and other audio attachments with an AI Gateway transcription model so an eve agent can answer them."
+---
+
+Voice notes and other audio files reach your agent, but by default the model does not transcribe them. To give your agent the ability to answer a voice note, you will need to add a tool that transcribes the staged file.
+
+This works for any channel that delivers audio as a file, such as iMessage voice notes through [Photon](../channels/photon), [Slack](../channels/slack#attachments) voice clips, and files sent through the [eve](../channels/eve) and [Chat SDK](../channels/chat-sdk#attachments) channels. See [Inbound attachments](../sandbox#inbound-attachments) for how eve stages files.
+
+## Add a transcription tool
+
+```ts title="agent/tools/transcribe_audio.ts"
+import { posix } from "node:path";
+import { transcribe } from "ai";
+import { defineTool } from "eve/tools";
+import { z } from "zod";
+
+const TRANSCRIPTION_MODEL = "openai/whisper-1";
+const ATTACHMENTS_DIR = "/workspace/.eve/attachments/";
+
+export default defineTool({
+ description:
+ "Transcribe an audio attachment, such as a voice note, to text. Pass the path from its `Attached file` label.",
+ inputSchema: z.object({
+ path: z.string().describe(`Attachment path under ${ATTACHMENTS_DIR}`),
+ }),
+ outputSchema: z.object({ transcript: z.string() }),
+ async execute({ path }, ctx) {
+ const file = posix.normalize(path);
+ if (!file.startsWith(ATTACHMENTS_DIR)) {
+ throw new Error(`Only attachments under ${ATTACHMENTS_DIR} can be transcribed.`);
+ }
+
+ const sandbox = await ctx.getSandbox();
+ const audio = await sandbox.readBinaryFile({ path: file, abortSignal: ctx.abortSignal });
+ if (!audio) {
+ throw new Error(`Attachment not found: ${file}. Ask the user to send the voice note again.`);
+ }
+
+ try {
+ const { text } = await transcribe({
+ model: TRANSCRIPTION_MODEL,
+ audio,
+ abortSignal: ctx.abortSignal,
+ });
+ return { transcript: text.trim() };
+ } catch (error) {
+ const reason = error instanceof Error ? error.message : String(error);
+ throw new Error(
+ `Transcription failed (${reason}). Tell the user you couldn't understand the voice note and ask them to type it.`,
+ );
+ }
+ },
+});
+```
+
+`transcribe()` uses the same credentials as your eve agent.
+
+## Tell the agent when to transcribe
+
+```md title="agent/instructions.md"
+When a message has an audio attachment (`Attached file ... (audio/...)`), such as a voice note, call `transcribe_audio` with its path first and treat the transcript as what the user said. Do not mention the transcription step unless it fails.
+```
+
+## When transcription fails
+
+When `execute` throws, eve gives the error to the model as a tool error, which it can relay to the user. See [When a tool throws](../tools#when-a-tool-throws). A file over the channel's upload limit, 25 MB by default, or a file eve can't download reaches the agent as a short note instead of a file.
diff --git a/docs/sandbox/index.mdx b/docs/sandbox/index.mdx
index 37e599731..601a56f99 100644
--- a/docs/sandbox/index.mdx
+++ b/docs/sandbox/index.mdx
@@ -162,6 +162,8 @@ Before every model call, eve reads the referenced files back from the sandbox. E
Other files, such as CSV, HEIC, audio, and video, reach the model as the label alone. eve checks image and PDF media types against the file's bytes when it stages the file. A file whose bytes don't match its declared image or PDF type is staged as `application/octet-stream`, so a provider never receives bytes it would reject.
+To let the agent answer voice notes and other audio, add a tool that transcribes the staged file. See [Voice notes](/docs/patterns/voice-notes).
+
If a referenced file is missing, or its bytes no longer match what eve staged because something in the sandbox overwrote it, the model receives a `FileNotFound` note in its place.
A file that eve can't stage never enters session history. When a channel's `fetchFile` fails, or a file part's `data` is not bytes, base64, a `data:` URL, or a URL, the model receives a note that names the file instead.