Skip to content

Beautiful, AI-native markdown IDE and LLM wiki

Shrink oversized chat images to fit instead of refusing (PRD-8505) (#5093)

Abraham074a73bb8e14
diff --git a/.changeset/shrink-chat-images-to-fit.md b/.changeset/shrink-chat-images-to-fit.md
new file mode 100644
index 000000000..4bec10a64
--- /dev/null
+++ b/.changeset/shrink-chat-images-to-fit.md
@@ -0,0 +1,5 @@
+---
+"@inkeep/open-knowledge-app": patch
+---
+
+An image too large to attach to an agent chat whole, such as a full-screen screenshot, is now scaled down to fit instead of being refused.
diff --git a/docs/content/features/editor.mdx b/docs/content/features/editor.mdx
index 53fe12d88..ace9fdd2a 100644
--- a/docs/content/features/editor.mdx
+++ b/docs/content/features/editor.mdx
@@ -129,7 +129,7 @@ It's the primary in-editor way to hand work to an agent, and the freetext compan
New in-app agent sessions automatically receive the [OpenKnowledge project skill](./skills/built-in-skills.mdx#project-scope) on their first non-command message, even if you have not installed it into an editor. The added guidance stays out of your visible message text.
-The **Add to prompt (+)** menu beside the field is available for every selected target. It always offers **Mention a page** and, when selected comments are waiting to be attached, **Attach comments**. With an **In app** agent selected, it also offers **Attach files**; you can add files from the menu, drop a file on the field, or paste a screenshot into it. Each attachment appears as a removable chip above the input and travels with the prompt into the new conversation. Once the prompt is sent, the chip for a doc that's still in the project opens that doc. Images can come from anywhere. Other files inside the project are sent as a path reference that the agent reads itself. A text file from outside the project, or any text file attached in a browser (which can't see file paths), travels with the message instead. Other kinds of file can't travel with the message; mention them with @ once they're in the project. Images must be PNG, JPEG, GIF, or WebP, and the images and text files that travel with one message are capped at 700 KB combined; a larger file has to be in the project, where you can mention it with @. That budget is tight for screenshots — a full-screen Retina capture is usually several megabytes and will be refused, so crop the capture or resize it first. Pick a **Terminal** CLI or an **External app** instead and the composer refuses the drop with the reason — those targets have no attachment channel — and if you attach first and switch afterwards, the send is blocked with the same explanation rather than silently dropping the file. Attachments aren't available in note windows yet, and whether an in-app agent can actually read an image is up to that agent: if it can't, OpenKnowledge shows a notification saying so once the agent's capabilities are known. The empty-state create composer on a fresh project takes attachments the same way — drop, paste, or the attach button while an **In app** agent is picked — and refuses them with the reason for **Terminal** and **External apps** targets.
+The **Add to prompt (+)** menu beside the field is available for every selected target. It always offers **Mention a page** and, when selected comments are waiting to be attached, **Attach comments**. With an **In app** agent selected, it also offers **Attach files**; you can add files from the menu, drop a file on the field, or paste a screenshot into it. Each attachment appears as a removable chip above the input and travels with the prompt into the new conversation. Once the prompt is sent, the chip for a doc that's still in the project opens that doc. Images can come from anywhere. Other files inside the project are sent as a path reference that the agent reads itself. A text file from outside the project, or any text file attached in a browser (which can't see file paths), travels with the message instead. Other kinds of file can't travel with the message; mention them with @ once they're in the project. Images must be PNG, JPEG, GIF, or WebP, and the images and text files that travel with one message are capped at 700 KB combined; a larger text file has to be in the project, where you can mention it with @. An image too large for what's left of that budget, such as a full-screen Retina screenshot, is scaled down and re-encoded to fit when it can be, so the agent gets a smaller copy instead of a refusal. A GIF is never resized, since that would drop its animation, so one over 700 KB is still refused. Pick a **Terminal** CLI or an **External app** instead and the composer refuses the drop with the reason — those targets have no attachment channel — and if you attach first and switch afterwards, the send is blocked with the same explanation rather than silently dropping the file. Attachments aren't available in note windows yet, and whether an in-app agent can actually read an image is up to that agent: if it can't, OpenKnowledge shows a notification saying so once the agent's capabilities are known. The empty-state create composer on a fresh project takes attachments the same way — drop, paste, or the attach button while an **In app** agent is picked — and refuses them with the reason for **Terminal** and **External apps** targets.
The composer stays out of the way where it doesn't belong: it's hidden inside an AI editor's embedded view, while the [docked terminal](#terminal) is open (the terminal is its own AI entry point), and when no doc or folder is open (the empty state's create composer offers the same handoff, attachments included). It's also dismissible — an **Ask AI** badge in the status bar brings it back.
diff --git a/packages/app/src/components/acp/ThreadView.dom.test.tsx b/packages/app/src/components/acp/ThreadView.dom.test.tsx
index fc5db3872..14ce97218 100644
--- a/packages/app/src/components/acp/ThreadView.dom.test.tsx
+++ b/packages/app/src/components/acp/ThreadView.dom.test.tsx
@@ -17,11 +17,12 @@ import {
within,
} from '@testing-library/react';
import userEvent from '@testing-library/user-event';
-import { afterEach, describe, expect, test, vi } from 'vitest';
+import { afterEach, describe, expect, onTestFinished, test, vi } from 'vitest';
import { notifySignInTerminalExited } from '@/components/handoff/sign-in-terminal-events';
import { Dialog, DialogContent, DialogDescription, DialogTitle } from '@/components/ui/dialog';
import { TooltipProvider } from '@/components/ui/tooltip';
import { MAX_TOTAL_ATTACHMENT_BYTES } from '@/lib/acp/image-attachment';
+import { stubImageCanvas } from '@/lib/acp/image-canvas.test-helper';
import type { RemoteModelCandidate } from '@/lib/acp/model-candidates';
import type {
RenderedItem,
@@ -5794,6 +5795,63 @@ describe('ThreadView attachment budget and clear fence (PRD-8453)', () => {
const smallPng = (name: string) =>
new File([new Uint8Array([0x89, 0x50, 0x4e, 0x47])], name, { type: 'image/png' });
+ test('an image dropped on an agent that takes none is refused and logged', async () => {
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ onTestFinished(() => warn.mockRestore());
+ model = makeModel({ items: [], turnActive: false });
+ render(
+ <ThreadView info={makeInfo({ status: 'ready', promptCapabilities: { image: false } })} />,
+ );
+
+ fireDrop([smallPng('a.png')]);
+
+ await waitFor(() =>
+ expect(toastError).toHaveBeenCalledWith("Claude doesn't accept image attachments."),
+ );
+ expect(warn).toHaveBeenCalledWith(
+ '[acp-attachment] refused',
+ expect.objectContaining({ surface: 'agent-chat', reason: 'images-not-accepted' }),
+ );
+ });
+
+ test('a photo too big to send whole attaches as a copy shrunk to fit', async () => {
+ stubImageCanvas({ width: 4000, height: 3000 });
+ onTestFinished(() => vi.unstubAllGlobals());
+ model = makeModel({ items: [], turnActive: false });
+ render(
+ <ThreadView info={makeInfo({ status: 'ready', promptCapabilities: { image: true } })} />,
+ );
+
+ fireDrop([
+ new File([new Uint8Array(MAX_TOTAL_ATTACHMENT_BYTES * 3)], 'photo.png', {
+ type: 'image/png',
+ }),
+ ]);
+
+ await waitFor(() =>
+ expect(screen.queryAllByTestId('agent-thread-pending-image-preview')).toHaveLength(1),
+ );
+ expect(screen.getByTestId('agent-thread-drop-notice').textContent).toBe('');
+ });
+
+ test('a second photo too big to send whole is shrunk into what the first left of the budget', async () => {
+ stubImageCanvas({ width: 4000, height: 3000, bytesPerPixel: 0.15 });
+ onTestFinished(() => vi.unstubAllGlobals());
+ model = makeModel({ items: [], turnActive: false });
+ render(
+ <ThreadView info={makeInfo({ status: 'ready', promptCapabilities: { image: true } })} />,
+ );
+ const photo = (name: string) =>
+ new File([new Uint8Array(MAX_TOTAL_ATTACHMENT_BYTES * 3)], name, { type: 'image/png' });
+
+ fireDrop([photo('one.png'), photo('two.png')]);
+
+ await waitFor(() =>
+ expect(screen.queryAllByTestId('agent-thread-pending-image-preview')).toHaveLength(2),
+ );
+ expect(screen.getByTestId('agent-thread-drop-notice').textContent).toBe('');
+ });
+
test('interleaved drops cannot stack attachments past the aggregate byte budget', async () => {
model = makeModel({ items: [], turnActive: false });
render(
diff --git a/packages/app/src/components/acp/ThreadView.tsx b/packages/app/src/components/acp/ThreadView.tsx
index 1b61cdf3d..24e854ade 100644
--- a/packages/app/src/components/acp/ThreadView.tsx
+++ b/packages/app/src/components/acp/ThreadView.tsx
@@ -145,7 +145,9 @@ import {
embeddedAttachmentBytes,
fileToAttachment,
isAttachmentRefusal,
+ logAttachmentRejection,
MAX_TOTAL_ATTACHMENT_BYTES,
+ rejectedPart,
totalEmbeddedAttachmentBytes,
} from '@/lib/acp/image-attachment';
import { computeDiffRows } from '@/lib/acp/inline-diff';
@@ -489,6 +491,7 @@ export function ThreadView({
const isImage = (file.type || '').startsWith('image/');
if (isImage && !imagesAccepted) {
rejectedImageCount += 1;
+ logAttachmentRejection('agent-chat', file, { kind: 'images-not-accepted' });
} else {
accepted.push(file);
}
@@ -519,6 +522,8 @@ export function ThreadView({
absPathOf,
workspaceContentDir,
pathSeparator,
+ imageBudgetBytes:
+ MAX_TOTAL_ATTACHMENT_BYTES - totalEmbeddedAttachmentBytes(attachmentsRef.current),
});
setPendingUploads((previous) => previous.filter((p) => p.id !== placeholderId));
if (outcome.ok) {
@@ -528,13 +533,17 @@ export function ThreadView({
MAX_TOTAL_ATTACHMENT_BYTES
) {
tooLargeTotalCount += 1;
+ logAttachmentRejection('agent-chat', rejectedPart(outcome.part), {
+ kind: 'total-too-large',
+ limitBytes: MAX_TOTAL_ATTACHMENT_BYTES,
+ });
} else {
commitPendingAttachments([...current, outcome.part], generation);
}
- } else if (isAttachmentRefusal(outcome.error)) {
- refusals.push(outcome.error);
} else {
- report(describeImageError(outcome.error));
+ logAttachmentRejection('agent-chat', file, outcome.error);
+ if (isAttachmentRefusal(outcome.error)) refusals.push(outcome.error);
+ else report(describeImageError(outcome.error));
}
} catch (err) {
setPendingUploads((previous) => previous.filter((p) => p.id !== placeholderId));
diff --git a/packages/app/src/editor/use-composer-attachments.dom.test.tsx b/packages/app/src/editor/use-composer-attachments.dom.test.tsx
index 1885e6ec4..e041a661d 100644
--- a/packages/app/src/editor/use-composer-attachments.dom.test.tsx
+++ b/packages/app/src/editor/use-composer-attachments.dom.test.tsx
@@ -3,6 +3,7 @@ import { act, renderHook, waitFor } from '@testing-library/react';
import { afterEach, describe, expect, test, vi } from 'vitest';
import { useComposerAttachments } from '@/editor/use-composer-attachments';
import { MAX_TOTAL_ATTACHMENT_BYTES } from '@/lib/acp/image-attachment';
+import { stubImageCanvas } from '@/lib/acp/image-canvas.test-helper';
i18n.load('en', {});
i18n.activate('en');
@@ -20,6 +21,7 @@ function renderAttachments(onError: (message: string) => void = () => {}) {
afterEach(() => {
vi.restoreAllMocks();
+ vi.unstubAllGlobals();
});
describe('useComposerAttachments', () => {
@@ -169,6 +171,84 @@ describe('useComposerAttachments', () => {
);
});
+ test('photos too big to send whole attach as copies shrunk to fit the message', async () => {
+ stubImageCanvas({ width: 4000, height: 3000, bytesPerPixel: 0.15 });
+ const onError = vi.fn();
+ const { result } = renderAttachments(onError);
+ const photo = (name: string) =>
+ new File([new Uint8Array(MAX_TOTAL_ATTACHMENT_BYTES * 3)], name, { type: 'image/png' });
+
+ await act(async () => {
+ await result.current.ingestFiles([photo('one.png'), photo('two.png')]);
+ });
+
+ await waitFor(() => {
+ expect(result.current.pendingAttachments).toHaveLength(2);
+ });
+ const sizes = result.current.pendingAttachments.map((part) =>
+ part.kind === 'image' ? part.sizeBytes : undefined,
+ );
+ expect(
+ result.current.pendingAttachments.map((part) => part.kind === 'image' && part.mimeType),
+ ).toEqual(['image/webp', 'image/webp']);
+ expect(sizes.reduce((sum = 0, size = 0) => sum + size, 0)).toBeLessThanOrEqual(
+ MAX_TOTAL_ATTACHMENT_BYTES,
+ );
+ expect(onError).not.toHaveBeenCalled();
+ });
+
+ test('a shrunk copy refused by the message budget is logged as that copy', async () => {
+ stubImageCanvas({ width: 4000, height: 3000, bytesPerPixel: 1 });
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const onError = vi.fn();
+ const { result } = renderAttachments(onError);
+ const photoBytes = MAX_TOTAL_ATTACHMENT_BYTES * 3;
+
+ await act(async () => {
+ await result.current.ingestFiles([
+ new File(['a'.repeat(Math.ceil(MAX_TOTAL_ATTACHMENT_BYTES * 0.85))], 'notes.txt', {
+ type: 'text/plain',
+ }),
+ new File([new Uint8Array(photoBytes)], 'photo.png', { type: 'image/png' }),
+ ]);
+ });
+
+ await waitFor(() => {
+ expect(onError).toHaveBeenCalledTimes(1);
+ });
+ expect(result.current.pendingAttachments).toHaveLength(1);
+ const refusal = warn.mock.calls.find(
+ ([tag, detail]) =>
+ tag === '[acp-attachment] refused' &&
+ (detail as { reason?: string }).reason === 'total-too-large',
+ )?.[1] as { mimeType: string; sizeBytes: number } | undefined;
+ expect(refusal?.mimeType).toBe('image/webp');
+ expect(refusal?.sizeBytes).toBeLessThanOrEqual(MAX_TOTAL_ATTACHMENT_BYTES);
+ expect(refusal?.sizeBytes).not.toBe(photoBytes);
+ });
+
+ test('a refused file is logged with its size and reason', async () => {
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const { result } = renderAttachments(() => {});
+
+ await act(async () => {
+ await result.current.ingestFiles([
+ new File([new Uint8Array(MAX_TOTAL_ATTACHMENT_BYTES + 1)], 'huge.log', {
+ type: 'text/plain',
+ }),
+ ]);
+ });
+
+ expect(warn).toHaveBeenCalledWith(
+ '[acp-attachment] refused',
+ expect.objectContaining({
+ surface: 'composer',
+ reason: 'file-too-large',
+ sizeBytes: MAX_TOTAL_ATTACHMENT_BYTES + 1,
+ }),
+ );
+ });
+
test('an in-flight read shows a placeholder that clears once the attachment lands', async () => {
const { result } = renderAttachments();
diff --git a/packages/app/src/editor/use-composer-attachments.ts b/packages/app/src/editor/use-composer-attachments.ts
index 096d835ff..53ffc48d6 100644
--- a/packages/app/src/editor/use-composer-attachments.ts
+++ b/packages/app/src/editor/use-composer-attachments.ts
@@ -10,7 +10,9 @@ import {
embeddedAttachmentBytes,
fileToAttachment,
isAttachmentRefusal,
+ logAttachmentRejection,
MAX_TOTAL_ATTACHMENT_BYTES,
+ rejectedPart,
totalEmbeddedAttachmentBytes,
} from '@/lib/acp/image-attachment';
@@ -72,9 +74,12 @@ export function useComposerAttachments(
absPathOf: options.absPathOf,
workspaceContentDir: options.workspaceContentDir,
pathSeparator: options.pathSeparator,
+ imageBudgetBytes:
+ MAX_TOTAL_ATTACHMENT_BYTES - totalEmbeddedAttachmentBytes(attachmentsRef.current),
});
setPendingUploads((previous) => previous.filter((p) => p.id !== placeholderId));
if (!outcome.ok) {
+ logAttachmentRejection('composer', file, outcome.error);
if (isAttachmentRefusal(outcome.error)) refusals.push(outcome.error);
else report(describeImageError(outcome.error));
continue;
@@ -85,6 +90,10 @@ export function useComposerAttachments(
MAX_TOTAL_ATTACHMENT_BYTES
) {
tooLargeTotalCount += 1;
+ logAttachmentRejection('composer', rejectedPart(outcome.part), {
+ kind: 'total-too-large',
+ limitBytes: MAX_TOTAL_ATTACHMENT_BYTES,
+ });
continue;
}
commit([...current, outcome.part], generation);
diff --git a/packages/app/src/lib/acp/image-attachment.dom.test.ts b/packages/app/src/lib/acp/image-attachment.dom.test.ts
index 51d03aa4a..69b3e291b 100644
--- a/packages/app/src/lib/acp/image-attachment.dom.test.ts
+++ b/packages/app/src/lib/acp/image-attachment.dom.test.ts
@@ -1,5 +1,5 @@
import { i18n } from '@lingui/core';
-import { describe, expect, test } from 'vitest';
+import { afterEach, describe, expect, onTestFinished, test, vi } from 'vitest';
import {
ALLOWED_IMAGE_MIMES,
type AttachmentRefusal,
@@ -8,9 +8,14 @@ import {
describeImageError,
fileToAttachment,
fileToImageAttachment,
+ fitImageToBytes,
+ type ImageShrinker,
+ logAttachmentRejection,
MAX_EMBEDDED_FILE_BYTES,
MAX_IMAGE_BYTES,
+ shrinkImageToFit,
} from './image-attachment.ts';
+import { stubImageCanvas } from './image-canvas.test-helper.ts';
i18n.load('en', {});
i18n.activate('en');
@@ -54,18 +59,138 @@ describe('fileToImageAttachment', () => {
}
});
- test('refuses a file above the per-image cap without reading its bytes', async () => {
+ test('refuses an image over the per-image cap when nothing here can shrink it', async () => {
const bytes = new Uint8Array(MAX_IMAGE_BYTES + 1);
const file = makeFile(bytes, 'huge.png', 'image/png');
const result = await fileToImageAttachment(file);
+ expect(result).toEqual({
+ ok: false,
+ error: { kind: 'too-large', sizeBytes: bytes.length, limitBytes: MAX_IMAGE_BYTES },
+ });
+ });
+
+ test('an image over the cap is sent as a copy shrunk to fit', async () => {
+ const file = makeFile(new Uint8Array(MAX_IMAGE_BYTES * 3), 'photo.png', 'image/png');
+ const shrink = vi.fn<ImageShrinker>(async () => ({
+ blob: new Blob([new Uint8Array([7, 8, 9])], { type: 'image/webp' }),
+ width: 2048,
+ height: 1536,
+ }));
+
+ const result = await fileToImageAttachment(file, { shrink });
+
+ expect(shrink).toHaveBeenCalledWith(file, [MAX_IMAGE_BYTES]);
+ expect(result).toEqual({
+ ok: true,
+ part: {
+ kind: 'image',
+ data: 'BwgJ',
+ mimeType: 'image/webp',
+ name: 'photo.png',
+ sizeBytes: 3,
+ },
+ });
+ });
+
+ test('an image over the cap that cannot shrink to fit is refused', async () => {
+ const file = makeFile(new Uint8Array(MAX_IMAGE_BYTES * 3), 'photo.png', 'image/png');
+
+ const result = await fileToImageAttachment(file, { shrink: async () => null });
+
expect(result.ok).toBe(false);
- if (!result.ok) {
- expect(result.error.kind).toBe('too-large');
- if (result.error.kind === 'too-large') {
- expect(result.error.sizeBytes).toBe(bytes.length);
- expect(result.error.limitBytes).toBe(MAX_IMAGE_BYTES);
- }
- }
+ if (!result.ok) expect(result.error.kind).toBe('too-large');
+ });
+
+ test('an image over the cap that the browser cannot decode is refused as too large', async () => {
+ stubImageCanvas();
+ vi.stubGlobal(
+ 'createImageBitmap',
+ vi.fn(async () => {
+ throw new DOMException('The source image could not be decoded.', 'InvalidStateError');
+ }),
+ );
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ onTestFinished(() => {
+ warn.mockRestore();
+ vi.unstubAllGlobals();
+ });
+ const bytes = new Uint8Array(MAX_IMAGE_BYTES * 2);
+
+ const result = await fileToImageAttachment(makeFile(bytes, 'broken.png', 'image/png'));
+
+ expect(result).toEqual({
+ ok: false,
+ error: { kind: 'too-large', sizeBytes: bytes.length, limitBytes: MAX_IMAGE_BYTES },
+ });
+ });
+
+ test('a GIF over the cap is refused without shrinking, which would drop its animation', async () => {
+ const file = makeFile(new Uint8Array(MAX_IMAGE_BYTES + 1), 'loop.gif', 'image/gif');
+ const shrink = vi.fn<ImageShrinker>(async () => null);
+
+ const result = await fileToImageAttachment(file, { shrink });
+
+ expect(shrink).not.toHaveBeenCalled();
+ expect(result.ok).toBe(false);
+ });
+
+ test('an image within what is left of the message is sent as it is', async () => {
+ const file = makeFile(new Uint8Array([1, 2, 3, 4]), 'shot.png', 'image/png');
+ const shrink = vi.fn<ImageShrinker>(async () => null);
+
+ const result = await fileToImageAttachment(file, { budgetBytes: 10, shrink });
+
+ expect(shrink).not.toHaveBeenCalled();
+ expect(result.ok && result.part.kind === 'image' && result.part.mimeType).toBe('image/png');
+ });
+
+ test('an image that fits the cap but not what is left is shrunk to what is left', async () => {
+ const file = makeFile(new Uint8Array(1000), 'second.png', 'image/png');
+ const shrink = vi.fn<ImageShrinker>(async () => ({
+ blob: new Blob([new Uint8Array(400)], { type: 'image/webp' }),
+ width: 600,
+ height: 400,
+ }));
+
+ const result = await fileToImageAttachment(file, { budgetBytes: 500, shrink });
+
+ expect(shrink).toHaveBeenCalledWith(file, [500]);
+ expect(result.ok && result.part.kind === 'image' && result.part.sizeBytes).toBe(400);
+ });
+
+ test('an image that fits the cap but cannot shrink into what is left is sent as it is, for the caller to budget', async () => {
+ const file = makeFile(new Uint8Array(1000), 'second.png', 'image/png');
+ const shrink = vi.fn<ImageShrinker>(async () => null);
+
+ const result = await fileToImageAttachment(file, { budgetBytes: 500, shrink });
+
+ expect(shrink).toHaveBeenCalledTimes(1);
+ expect(result.ok && result.part.kind === 'image' && result.part.sizeBytes).toBe(1000);
+ });
+
+ test('an image over the cap that cannot shrink into what is left is shrunk to the cap, for the caller to budget', async () => {
+ const file = makeFile(new Uint8Array(MAX_IMAGE_BYTES * 2), 'photo.png', 'image/png');
+ const shrink = vi.fn<ImageShrinker>(async () => ({
+ blob: new Blob([new Uint8Array(900)], { type: 'image/webp' }),
+ width: 900,
+ height: 600,
+ }));
+
+ const result = await fileToImageAttachment(file, { budgetBytes: 100, shrink });
+
+ expect(shrink).toHaveBeenCalledWith(file, [100, MAX_IMAGE_BYTES]);
+ expect(result.ok && result.part.kind === 'image' && result.part.sizeBytes).toBe(900);
+ });
+
+ test('an image over the cap is decoded once, however many sizes it is fitted to', async () => {
+ stubImageCanvas({ width: 4000, height: 3000 });
+ onTestFinished(() => vi.unstubAllGlobals());
+ const file = makeFile(new Uint8Array(MAX_IMAGE_BYTES * 2), 'photo.png', 'image/png');
+
+ const result = await fileToImageAttachment(file, { budgetBytes: 100 });
+
+ expect(createImageBitmap).toHaveBeenCalledTimes(1);
+ expect(result).toMatchObject({ ok: true, part: { kind: 'image', mimeType: 'image/webp' } });
});
test('ALLOWED_IMAGE_MIMES intentionally excludes svg + bmp + tiff (agent-verified formats only)', () => {
@@ -79,6 +204,102 @@ describe('fileToImageAttachment', () => {
});
});
+describe('fitImageToBytes', () => {
+ const byArea = (bytesPerPixel: number) =>
+ vi.fn(
+ async (width: number, height: number) =>
+ new Blob([new Uint8Array(Math.round(width * height * bytesPerPixel))]),
+ );
+
+ test('caps the long edge first, then shrinks until the encoding fits', async () => {
+ const encode = byArea(0.2);
+
+ const fitted = await fitImageToBytes({ width: 4000, height: 3000 }, 300_000, encode);
+
+ expect(encode.mock.calls[0]).toEqual([2048, 1536]);
+ expect(fitted).not.toBeNull();
+ expect(fitted?.blob.size).toBeLessThanOrEqual(300_000);
+ expect(fitted?.width).toBeLessThan(2048);
+ expect((fitted?.width ?? 0) / (fitted?.height ?? 1)).toBeCloseTo(4 / 3, 1);
+ });
+
+ test('re-encodes a smaller image at its own size first', async () => {
+ const encode = byArea(0.1);
+
+ const fitted = await fitImageToBytes({ width: 1000, height: 800 }, 1_000_000, encode);
+
+ expect(encode).toHaveBeenCalledTimes(1);
+ expect(fitted).toMatchObject({ width: 1000, height: 800 });
+ });
+
+ test('gives up rather than shrink below a readable size', async () => {
+ const encode = byArea(1);
+
+ expect(await fitImageToBytes({ width: 4000, height: 3000 }, 10, encode)).toBeNull();
+ for (const [width, height] of encode.mock.calls) {
+ expect(Math.max(width, height)).toBeGreaterThanOrEqual(512);
+ }
+ });
+});
+
+describe('shrinkImageToFit', () => {
+ afterEach(() => {
+ vi.unstubAllGlobals();
+ });
+
+ test('encodes WebP where the browser can, and releases the decoded image', async () => {
+ const { close } = stubImageCanvas({ width: 3000, height: 2000, bytesPerPixel: 0.05 });
+ const file = makeFile(new Uint8Array(10), 'shot.png', 'image/png');
+
+ const shrunk = await shrinkImageToFit(file, [150_000]);
+
+ expect(shrunk?.blob.type).toBe('image/webp');
+ expect(shrunk?.blob.size).toBeLessThanOrEqual(150_000);
+ expect(close).toHaveBeenCalledTimes(1);
+ });
+
+ test('falls back to JPEG on white where the browser cannot encode WebP', async () => {
+ const canvas = stubImageCanvas({
+ encodes: (type) => (type === 'image/webp' ? 'image/png' : type),
+ });
+ const file = makeFile(new Uint8Array(10), 'shot.png', 'image/png');
+
+ const shrunk = await shrinkImageToFit(file, [150_000]);
+
+ expect(shrunk?.blob.type).toBe('image/jpeg');
+ expect(shrunk && canvas.encodedCanvas(shrunk.blob)).toEqual({
+ imageVisible: true,
+ transparentAreaColor: '#ffffff',
+ });
+ });
+
+ test('declines where there is no canvas to draw on', async () => {
+ vi.stubGlobal('OffscreenCanvas', undefined);
+ const file = makeFile(new Uint8Array(10), 'shot.png', 'image/png');
+
+ expect(await shrinkImageToFit(file, [150_000])).toBeNull();
+ });
+});
+
+describe('logAttachmentRejection', () => {
+ test('records why, how big and what type, but not the file name', () => {
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+ const file = makeFile(new Uint8Array(5), 'private-name.png', 'image/png');
+
+ logAttachmentRejection('composer', file, { kind: 'total-too-large', limitBytes: 700 });
+
+ expect(warn).toHaveBeenCalledWith('[acp-attachment] refused', {
+ surface: 'composer',
+ reason: 'total-too-large',
+ sizeBytes: 5,
+ mimeType: 'image/png',
+ limitBytes: 700,
+ });
+ expect(JSON.stringify(warn.mock.calls)).not.toContain('private-name');
+ warn.mockRestore();
+ });
+});
+
describe('describeImageError', () => {
test('unsupported-type calls out the mime it saw', () => {
expect(describeImageError({ kind: 'unsupported-type', mimeType: 'image/svg+xml' })).toContain(
diff --git a/packages/app/src/lib/acp/image-attachment.ts b/packages/app/src/lib/acp/image-attachment.ts
index 0999d3168..24e6f4cfc 100644
--- a/packages/app/src/lib/acp/image-attachment.ts
+++ b/packages/app/src/lib/acp/image-attachment.ts
@@ -144,12 +144,7 @@ export function describeAttachmentRefusals(refusals: readonly AttachmentRefusal[
return [...groups.values()].map(({ first, names }) => describeRefusedFiles(first, names));
}
-async function encodeImageFile(file: File): Promise<{
- readonly data: string;
- readonly mimeType: string;
- readonly name: string;
- readonly sizeBytes: number;
-}> {
+async function base64Of(blob: Blob): Promise<string> {
const dataUrl = await new Promise<string>((resolve, reject) => {
const reader = new FileReader();
reader.onerror = () => reject(reader.error ?? new Error('FileReader failed'));
@@ -161,20 +156,102 @@ async function encodeImageFile(file: File): Promise<{
}
resolve(result);
};
- reader.readAsDataURL(file);
+ reader.readAsDataURL(blob);
});
const comma = dataUrl.indexOf(',');
- const data = comma >= 0 ? dataUrl.slice(comma + 1) : dataUrl;
- return {
- data,
- mimeType: file.type || 'application/octet-stream',
- name: file.name || 'image',
- sizeBytes: file.size,
- };
+ return comma >= 0 ? dataUrl.slice(comma + 1) : dataUrl;
+}
+
+const RESIZE_MAX_EDGE = 2048;
+
+const RESIZE_MIN_EDGE = 512;
+
+const RESIZE_ATTEMPTS = 6;
+
+const RESIZE_QUALITY = 0.85;
+
+export interface ShrunkImage {
+ readonly blob: Blob;
+ readonly width: number;
+ readonly height: number;
+}
+
+export type ImageShrinker = (file: File, limits: readonly number[]) => Promise<ShrunkImage | null>;
+
+export async function fitImageToBytes(
+ size: { readonly width: number; readonly height: number },
+ maxBytes: number,
+ encode: (width: number, height: number) => Promise<Blob>,
+): Promise<ShrunkImage | null> {
+ const longEdge = Math.max(size.width, size.height);
+ const smallestEdge = Math.min(RESIZE_MIN_EDGE, longEdge);
+ let scale = Math.min(1, RESIZE_MAX_EDGE / longEdge);
+ for (let attempt = 0; attempt < RESIZE_ATTEMPTS; attempt += 1) {
+ const width = Math.max(1, Math.round(size.width * scale));
+ const height = Math.max(1, Math.round(size.height * scale));
+ if (Math.max(width, height) < smallestEdge) return null;
+ const blob = await encode(width, height);
+ if (blob.size <= maxBytes) return { blob, width, height };
+ scale *= Math.sqrt(maxBytes / blob.size) * 0.9;
+ }
+ return null;
+}
+
+async function encodeScaled(bitmap: ImageBitmap, width: number, height: number): Promise<Blob> {
+ const canvas = new OffscreenCanvas(width, height);
+ const context = canvas.getContext('2d');
+ if (context === null) throw new Error('OffscreenCanvas has no 2d context');
+ context.imageSmoothingQuality = 'high';
+ context.drawImage(bitmap, 0, 0, width, height);
+ const webp = await canvas.convertToBlob({ type: 'image/webp', quality: RESIZE_QUALITY });
+ if (webp.type === 'image/webp') return webp;
+ context.globalCompositeOperation = 'destination-over';
+ context.fillStyle = '#ffffff';
+ context.fillRect(0, 0, width, height);
+ return canvas.convertToBlob({ type: 'image/jpeg', quality: RESIZE_QUALITY });
+}
+
+async function fitBitmapToFirstLimit(
+ bitmap: ImageBitmap,
+ limits: readonly number[],
+): Promise<ShrunkImage | null> {
+ for (const maxBytes of limits) {
+ const fitted = await fitImageToBytes(bitmap, maxBytes, (width, height) =>
+ encodeScaled(bitmap, width, height),
+ );
+ if (fitted !== null) return fitted;
+ }
+ return null;
+}
+
+export const shrinkImageToFit: ImageShrinker = async (file, limits) => {
+ if (typeof createImageBitmap !== 'function' || typeof OffscreenCanvas !== 'function') {
+ return null;
+ }
+ let bitmap: ImageBitmap | undefined;
+ try {
+ bitmap = await createImageBitmap(file);
+ return await fitBitmapToFirstLimit(bitmap, limits);
+ } catch (error) {
+ console.warn(
+ '[acp-attachment] could not resize an image',
+ { sizeBytes: file.size, mimeType: file.type || '' },
+ error,
+ );
+ return null;
+ } finally {
+ bitmap?.close();
+ }
+};
+
+export interface ImageAttachmentOptions {
+ readonly budgetBytes?: number | undefined;
+ readonly shrink?: ImageShrinker;
}
export async function fileToImageAttachment(
file: File,
+ options: ImageAttachmentOptions = {},
): Promise<
| { readonly ok: true; readonly part: AttachmentPart }
| { readonly ok: false; readonly error: ImageAttachmentError }
@@ -183,25 +260,78 @@ export async function fileToImageAttachment(
if (!ALLOWED_IMAGE_MIMES.has(mimeType)) {
return { ok: false, error: { kind: 'unsupported-type', mimeType } };
}
+ const name = file.name || 'image';
+ const budget = Math.max(0, Math.min(MAX_IMAGE_BYTES, options.budgetBytes ?? MAX_IMAGE_BYTES));
+ if (file.size > budget && mimeType !== 'image/gif') {
+ const limits: number[] = budget > 0 ? [budget] : [];
+ if (file.size > MAX_IMAGE_BYTES && budget < MAX_IMAGE_BYTES) limits.push(MAX_IMAGE_BYTES);
+ const shrink = options.shrink ?? shrinkImageToFit;
+ const shrunk = limits.length > 0 ? await shrink(file, limits) : null;
+ if (shrunk !== null) {
+ console.info('[acp-attachment] resized an image to fit', {
+ fromBytes: file.size,
+ toBytes: shrunk.blob.size,
+ width: shrunk.width,
+ height: shrunk.height,
+ mimeType: shrunk.blob.type,
+ });
+ return {
+ ok: true,
+ part: {
+ kind: 'image',
+ data: await base64Of(shrunk.blob),
+ mimeType: shrunk.blob.type,
+ name,
+ sizeBytes: shrunk.blob.size,
+ },
+ };
+ }
+ }
if (file.size > MAX_IMAGE_BYTES) {
return {
ok: false,
error: { kind: 'too-large', sizeBytes: file.size, limitBytes: MAX_IMAGE_BYTES },
};
}
- const encoded = await encodeImageFile(file);
return {
ok: true,
- part: {
- kind: 'image',
- data: encoded.data,
- mimeType: encoded.mimeType,
- name: encoded.name,
- sizeBytes: encoded.sizeBytes,
- },
+ part: { kind: 'image', data: await base64Of(file), mimeType, name, sizeBytes: file.size },
};
}
+export type AttachmentSurface = 'agent-chat' | 'composer';
+
+export type AttachmentRejection =
+ | ImageAttachmentError
+ | { readonly kind: 'total-too-large'; readonly limitBytes: number }
+ | { readonly kind: 'images-not-accepted' };
+
+export interface RejectedAttachment {
+ readonly size: number;
+ readonly type: string;
+}
+
+export function rejectedPart(part: AttachmentPart): RejectedAttachment {
+ return {
+ size: embeddedAttachmentBytes(part),
+ type: part.kind === 'image' || part.kind === 'blob' ? part.mimeType : '',
+ };
+}
+
+export function logAttachmentRejection(
+ surface: AttachmentSurface,
+ rejected: RejectedAttachment,
+ rejection: AttachmentRejection,
+): void {
+ console.warn('[acp-attachment] refused', {
+ surface,
+ reason: rejection.kind,
+ sizeBytes: rejected.size,
+ mimeType: rejected.type || '',
+ ...('limitBytes' in rejection ? { limitBytes: rejection.limitBytes } : {}),
+ });
+}
+
export function collectImageFiles(dataTransfer: DataTransfer | null): File[] {
return collectFiles(dataTransfer, (file) => file.type.startsWith('image/'));
}
@@ -243,6 +373,7 @@ export interface FileToAttachmentDeps {
readonly absPathOf?: (file: File) => string | null;
readonly workspaceContentDir?: string;
readonly pathSeparator?: '/' | '\\';
+ readonly imageBudgetBytes?: number;
}
function workspaceRelativePath(abs: string, root: string, sep: '/' | '\\'): string | null {
@@ -315,7 +446,9 @@ export async function fileToAttachment(
| { readonly ok: false; readonly error: FileAttachmentError }
> {
const mime = file.type || '';
- if (mime.startsWith('image/')) return fileToImageAttachment(file);
+ if (mime.startsWith('image/')) {
+ return fileToImageAttachment(file, { budgetBytes: deps.imageBudgetBytes });
+ }
const found = projectPathOf(file, deps);
if (typeof found === 'string') return fileToEmbeddedAttachment(file, found);
return {
diff --git a/packages/app/src/lib/acp/image-canvas.test-helper.ts b/packages/app/src/lib/acp/image-canvas.test-helper.ts
new file mode 100644
index 000000000..0e4505de8
--- /dev/null
+++ b/packages/app/src/lib/acp/image-canvas.test-helper.ts
@@ -0,0 +1,74 @@
+import { vi } from 'vitest';
+
+export interface EncodedCanvas {
+ readonly imageVisible: boolean;
+ readonly transparentAreaColor: string;
+}
+
+export function stubImageCanvas(
+ options: {
+ readonly width?: number;
+ readonly height?: number;
+ readonly bytesPerPixel?: number;
+ readonly encodes?: (type: string) => string;
+ } = {},
+): {
+ readonly close: ReturnType<typeof vi.fn>;
+ readonly encodedCanvas: (blob: Blob) => EncodedCanvas | undefined;
+} {
+ const close = vi.fn();
+ const bytesPerPixel = options.bytesPerPixel ?? 0.05;
+ const encodes = options.encodes ?? ((type: string) => type);
+ const encodedCanvases = new WeakMap<Blob, EncodedCanvas>();
+ vi.stubGlobal(
+ 'createImageBitmap',
+ vi.fn(async () => ({ width: options.width ?? 3000, height: options.height ?? 2000, close })),
+ );
+ vi.stubGlobal(
+ 'OffscreenCanvas',
+ class {
+ readonly width: number;
+ readonly height: number;
+ image: 'none' | 'drawn' | 'covered' = 'none';
+ transparentArea = 'transparent';
+ constructor(width: number, height: number) {
+ this.width = width;
+ this.height = height;
+ }
+ getContext() {
+ const context = {
+ globalCompositeOperation: 'source-over',
+ fillStyle: '#000000',
+ drawImage: () => {
+ this.image = 'drawn';
+ },
+ fillRect: () => {
+ if (context.globalCompositeOperation === 'destination-over') {
+ if (this.transparentArea === 'transparent') this.transparentArea = context.fillStyle;
+ return;
+ }
+ if (this.image === 'drawn') this.image = 'covered';
+ this.transparentArea = context.fillStyle;
+ },
+ };
+ return context;
+ }
+ async convertToBlob({ type }: { type: string }) {
+ const encoded = encodes(type);
+ const blob = new Blob(
+ [new Uint8Array(Math.round(this.width * this.height * bytesPerPixel))],
+ { type: encoded },
+ );
+ encodedCanvases.set(blob, {
+ imageVisible: this.image === 'drawn',
+ transparentAreaColor:
+ encoded === 'image/jpeg' && this.transparentArea === 'transparent'
+ ? '#000000'
+ : this.transparentArea,
+ });
+ return blob;
+ }
+ },
+ );
+ return { close, encodedCanvas: (blob) => encodedCanvases.get(blob) };
+}