Platform for stateful agents: AI with advanced memory that can learn and self-improve over time.
docs: fix multimodal content order in examples (#5843)
Camerone8eb2d53e93f
diff --git a/fern/pages/agents/multimodal.mdx b/fern/pages/agents/multimodal.mdxnew file mode 100644index 000000000..ffadab61f--- /dev/null+++ b/fern/pages/agents/multimodal.mdx@@ -0,0 +1,163 @@+---+title: "Multi-modal (image inputs)"+subtitle: "Send images to your agents"+slug: "multimodal"+---++<Note>+Multi-modal features require compatible language models. Ensure your agent is configured with a multi-modal capable model.+</Note>++Letta agents support image inputs, enabling richer conversations and more powerful agent capabilities.++## Model Support++Multi-modal capabilities depend on the underlying language model.+You can check which models from the API providers support image inputs by checking their individual model pages:++- **[OpenAI](https://platform.openai.com/docs/models)**: GPT-4.1, o1/3/4, GPT-4o+- **[Anthropic](https://docs.anthropic.com/en/docs/about-claude/models/overview)**: Claude Opus 4, Claude Sonnet 4+- **[Gemini](https://ai.google.dev/gemini-api/docs/models)**: Gemini 2.5 Pro, Gemini 2.5 Flash++If the provider you're using doesn't support image inputs, your images will still appear in the context window, but as a text message telling the agent that an image exists.++## ADE Support++You can pass images to your agents by drag-and-dropping them into the chat window, or clicking the image icon to select a manual file upload.++<img className="light" src="/images/ade-mm.png" />+<img className="dark" src="/images/ade-mm-dark.png" />++## Usage Examples (SDK)++### Sending an Image via URL++<CodeGroup>+```typescript TypeScript maxLines=100+import { LettaClient } from '@letta-ai/letta-client';++const client = new LettaClient({ token: "LETTA_API_KEY" });++const response = await client.agents.messages.create(+ agentState.id, {+ messages: [+ {+ role: "user",+ content: [+ {+ type: "text",+ text: "Describe this image."+ },+ {+ type: "image",+ source: {+ type: "url",+ url: "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg",+ },+ }+ ],+ }+ ],+ }+);+```+```python title="python" maxLines=100+from letta_client import Letta++client = Letta(token="LETTA_API_KEY")++response = client.agents.messages.create(+ agent_id=agent_state.id,+ messages=[+ {+ "role": "user",+ "content": [+ {+ "type": "text",+ "text": "Describe this image."+ },+ {+ "type": "image",+ "source": {+ "type": "url",+ "url": "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg",+ },+ }+ ],+ }+ ],+)+```+</CodeGroup>++### Sending an Image via Base64++<CodeGroup>+```typescript TypeScript maxLines=100+import { LettaClient } from '@letta-ai/letta-client';++const client = new LettaClient({ token: "LETTA_API_KEY" });++const imageUrl = "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg";+const imageResponse = await fetch(imageUrl);+const imageBuffer = await imageResponse.arrayBuffer();+const imageData = Buffer.from(imageBuffer).toString('base64');++const response = await client.agents.messages.create(+ agentState.id, {+ messages: [+ {+ role: "user",+ content: [+ {+ type: "text",+ text: "Describe this image."+ },+ {+ type: "image",+ source: {+ type: "base64",+ mediaType: "image/jpeg",+ data: imageData,+ },+ }+ ],+ }+ ],+ }+);+```+```python title="python" maxLines=100+import base64+import httpx+from letta_client import Letta++client = Letta(token="LETTA_API_KEY")++image_url = "https://upload.wikimedia.org/wikipedia/commons/a/a7/Camponotus_flavomarginatus_ant.jpg"+image_data = base64.standard_b64encode(httpx.get(image_url).content).decode("utf-8")++response = client.agents.messages.create(+ agent_id=agent_state.id,+ messages=[+ {+ "role": "user",+ "content": [+ {+ "type": "text",+ "text": "Describe this image."+ },+ {+ "type": "image",+ "source": {+ "type": "base64",+ "media_type": "image/jpeg",+ "data": image_data,+ },+ }+ ],+ }+ ],+)+```+</CodeGroup>