Skip to content

armgit / letta

Public Imported from GitHub
0

Platform for stateful agents: AI with advanced memory that can learn and self-improve over time.

docs: fix multimodal content order in examples (#5843)

Camerone8eb2d53e93f
diff --git a/fern/pages/agents/multimodal.mdx b/fern/pages/agents/multimodal.mdx
new file mode 100644
index 000000000..ffadab61f
--- /dev/null
+++ b/fern/pages/agents/multimodal.mdx
@@ -0,0 +1,163 @@
+---
+title: "Multi-modal (image inputs)"
+subtitle: "Send images to your agents"
+slug: "multimodal"
+---
+
+<Note>
+Multi-modal features require compatible language models. Ensure your agent is configured with a multi-modal capable model.
+</Note>
+
+Letta agents support image inputs, enabling richer conversations and more powerful agent capabilities.
+
+## Model Support
+
+Multi-modal capabilities depend on the underlying language model.
+You can check which models from the API providers support image inputs by checking their individual model pages:
+
+- **[OpenAI](https://platform.openai.com/docs/models)**: GPT-4.1, o1/3/4, GPT-4o
+- **[Anthropic](https://docs.anthropic.com/en/docs/about-claude/models/overview)**: Claude Opus 4, Claude Sonnet 4
+- **[Gemini](https://ai.google.dev/gemini-api/docs/models)**: Gemini 2.5 Pro, Gemini 2.5 Flash
+
+If the provider you're using doesn't support image inputs, your images will still appear in the context window, but as a text message telling the agent that an image exists.
+
+## ADE Support
+
+You can pass images to your agents by drag-and-dropping them into the chat window, or clicking the image icon to select a manual file upload.
+
+<img className="light" src="/images/ade-mm.png" />
+<img className="dark" src="/images/ade-mm-dark.png" />
+
+## Usage Examples (SDK)
+
+### Sending an Image via URL
+
+<CodeGroup>
+```typescript TypeScript maxLines=100
+import { LettaClient } from '@letta-ai/letta-client';
+
+const client = new LettaClient({ token: "LETTA_API_KEY" });
+
+const response = await client.agents.messages.create(
+ agentState.id, {
+ messages: [
+ {
+ role: "user",
+ content: [
+ {
+ type: "text",
+ text: "Describe this image."
+ },
+ {
+ type: "image",
+ source: {
+ type: "url",
+ url: "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg",
+ },
+ }
+ ],
+ }
+ ],
+ }
+);
+```
+```python title="python" maxLines=100
+from letta_client import Letta
+
+client = Letta(token="LETTA_API_KEY")
+
+response = client.agents.messages.create(
+ agent_id=agent_state.id,
+ messages=[
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "text",
+ "text": "Describe this image."
+ },
+ {
+ "type": "image",
+ "source": {
+ "type": "url",
+ "url": "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg",
+ },
+ }
+ ],
+ }
+ ],
+)
+```
+</CodeGroup>
+
+### Sending an Image via Base64
+
+<CodeGroup>
+```typescript TypeScript maxLines=100
+import { LettaClient } from '@letta-ai/letta-client';
+
+const client = new LettaClient({ token: "LETTA_API_KEY" });
+
+const imageUrl = "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg";
+const imageResponse = await fetch(imageUrl);
+const imageBuffer = await imageResponse.arrayBuffer();
+const imageData = Buffer.from(imageBuffer).toString('base64');
+
+const response = await client.agents.messages.create(
+ agentState.id, {
+ messages: [
+ {
+ role: "user",
+ content: [
+ {
+ type: "text",
+ text: "Describe this image."
+ },
+ {
+ type: "image",
+ source: {
+ type: "base64",
+ mediaType: "image/jpeg",
+ data: imageData,
+ },
+ }
+ ],
+ }
+ ],
+ }
+);
+```
+```python title="python" maxLines=100
+import base64
+import httpx
+from letta_client import Letta
+
+client = Letta(token="LETTA_API_KEY")
+
+image_url = "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg"
+image_data = base64.standard_b64encode(httpx.get(image_url).content).decode("utf-8")
+
+response = client.agents.messages.create(
+ agent_id=agent_state.id,
+ messages=[
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "text",
+ "text": "Describe this image."
+ },
+ {
+ "type": "image",
+ "source": {
+ "type": "base64",
+ "media_type": "image/jpeg",
+ "data": image_data,
+ },
+ }
+ ],
+ }
+ ],
+)
+```
+</CodeGroup>