Skip to content

Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.

14.6 KB
Raw Blame
/**
 * What may be attached to a message, and how much of it.
 *
 * One declaration read from both sides, the same as `routine-firing.ts`: the composer refuses a
 * file before uploading it and the server refuses it again on arrival, and those two refusals have
 * to agree. A limit written twice is a limit that drifts, and the drift shows up as a file the
 * composer accepted and the server threw away with no explanation.
 */

export const MAX_ATTACHMENTS_PER_MESSAGE = 8;

/**
 * A hard ceiling, not a downscaling threshold: nothing in this path resizes
 * or re-encodes an image, so a file under this limit is stored and sent
 * whole. `resolvePart` (`server/src/channels/attachment-parts.ts`) then
 * base64-encodes those bytes into a single content part, which runs about
 * 4/3 the byte size — a file at this ceiling becomes a part of roughly
 * 10.7 MB.
 *
 * This deployment only ever targets `openai` (`tenant-package.ts` refuses
 * to load a package whose `model.provider` is anything else), and OpenAI's
 * vision input limit is 20 MB per image. 8 MB was chosen, not the 15 MB
 * that would sit exactly at that line, because it is the encoded form that
 * has to fit under the provider's number, not the stored one, and because
 * it is not certain from that number alone which side of the encoding it
 * was measured on — so this sits at roughly half of it either way.
 *
 * Downscaling is deliberately not built yet: a file over this ceiling is
 * refused at pick time rather than shrunk, and building the shrink step is
 * a known follow-up, not something this constant can stand in for.
 */
export const MAX_IMAGE_BYTES = 8 * 1024 * 1024;

/**
 * How large a text file may be UPLOADED. Not how much of it the model reads.
 *
 * Text files are not downscaled, so this is a hard refusal like `MAX_IMAGE_BYTES`.
 *
 * THIS IS NOT THE LIMIT THAT DECIDES WHAT REACHES THE MODEL, AND THE TWO NUMBERS ARE FAR APART.
 * `MAX_EXTRACTED_CHARACTERS` below is 120,000, and this is 1,048,576. A file accepted at this
 * ceiling is stored whole and then handed to the model as its first 120,000 characters — for
 * ASCII-ish text, where a byte is a character, roughly its first EIGHTH, with about 89% cut. Text
 * in a script that costs several bytes per character loses proportionally less, because the
 * ceilings are counted in different units, but a file anywhere near this size is cut.
 *
 * The two are deliberately different numbers because they bound different things, and neither can
 * stand in for the other:
 *
 *   - this one bounds what is UPLOADED, STORED and SERVED — bandwidth, disk, and the size of the
 *     response `/api/attachments/:id` has to produce;
 *   - `MAX_EXTRACTED_CHARACTERS` bounds what one attachment may spend of a shared CONTEXT WINDOW,
 *     which is a budget split with the conversation and with up to seven other attachments.
 *
 * REJECTED: lowering this to 120,000 so the two agree. It would refuse files the app can already
 * do something useful with — the first 120,000 characters of a large CSV usually answers the
 * question that was asked of it — and would trade a partial read for no read at all.
 *
 * REJECTED: raising `MAX_EXTRACTED_CHARACTERS` to match this. A megabyte of text is a few hundred
 * thousand tokens, which is the context window this deployment targets spent entirely on one
 * attachment. That is the outcome that constant exists to prevent.
 *
 * So the gap stays, and what it costs is stated here rather than left for somebody to derive by
 * dividing one constant by the other. What is NOT yet resolved is that the person who uploads the
 * file is not told — see `MAX_EXTRACTED_CHARACTERS`.
 */
export const MAX_FILE_BYTES = 1024 * 1024;

/**
 * How much extracted text may reach the model from one file.
 *
 * The real limit on a text attachment is tokens, not bytes: a one-megabyte file is a few hundred
 * thousand tokens and would fill the window on its own. 120,000 characters is roughly 30,000
 * tokens, which every model this deployment targets can hold alongside a conversation. Text past
 * this point is cut and the part says so, rather than being silently dropped.
 *
 * THE PART SAYS SO TO THE MODEL. NOBODY SAYS SO TO THE PERSON WHO ATTACHED THE FILE.
 *
 * `extractDocumentText` (`server/src/channels/attachment-parts.ts`) appends
 * `[attachment truncated at 120000 characters]` to what it sends, so the model is never left
 * answering questions about a file it read only part of without knowing. That is half the promise.
 * The other half is not kept: this ceiling is 120,000 while `MAX_FILE_BYTES` above is 1,048,576, so
 * a 1 MB CSV is accepted at pick time, uploaded whole, shown as a `1.0MB` tile — and then read by
 * the model as roughly its first eighth, with nothing anywhere in the UI saying so. Somebody who
 * asks "how many rows have status=failed" gets a confident answer about the part that fit.
 *
 * That is a real gap and it is recorded rather than fixed here, because the fix is a composer
 * change and this file is only where the number lives. What the composer needs is already exported:
 * a file whose SIZE IN BYTES exceeds this constant may be truncated, and one at or under it cannot
 * be. That direction is exact rather than approximate — UTF-8 spends at least one byte per code
 * point, so a file of N bytes can never decode to more than N characters — which makes
 * `file.size > MAX_EXTRACTED_CHARACTERS` a warning that never fires on a file that will arrive
 * whole. It is deliberately a "may", since the same byte count is fewer characters in any script
 * that costs more than a byte each.
 */
export const MAX_EXTRACTED_CHARACTERS = 120_000;

/**
 * Whether a file of this many BYTES may reach the model truncated.
 *
 * Exact in the safe direction: UTF-8 decodes N bytes to at most N characters,
 * so `size <= MAX_EXTRACTED_CHARACTERS` cannot be cut, and `size > ...` may be
 * (fewer characters for multi-byte scripts, hence "may"). Callers show a
 * warning, never a refusal — the server still accepts and reads the prefix.
 */
export function mayBeTruncatedForModel(sizeBytes: number): boolean {
  return sizeBytes > MAX_EXTRACTED_CHARACTERS;
}

/**
 * `image/svg+xml` is deliberately absent.
 *
 * An SVG is an image and can also carry script. Served inline from this app's own origin, one
 * pasted into a channel is stored XSS against everybody in it. Nobody pastes an SVG expecting a
 * model to read it, so it is refused at the door rather than sanitised.
 */
export const ACCEPTED_IMAGE_MIME = [
  "image/png",
  "image/jpeg",
  "image/gif",
  "image/webp",
] as const;

export const ACCEPTED_TEXT_MIME = [
  "text/plain",
  "text/markdown",
  "text/csv",
  "application/json",
] as const;

/**
 * THE MEDIA TYPE ON ITS OWN: LOWER CASE, PARAMETERS GONE — AND THE ONLY FORM ANYTHING HERE COMPARES.
 *
 * A `File`'s type is whatever produced it. Bun's constructor appends `;charset=utf-8` to
 * `text/plain` and `application/json`, and a browser does the same for some clipboard entries; RFC
 * 2045 makes the type case-insensitive besides, and while `new File(...)` ASCII-lower-cases what it
 * is given, a `File` this app never built — one off a drop or a clipboard — carries whatever its
 * source wrote. `sniffMimeType` already normalises the same two ways before handing its answer to
 * `classifyAttachment`.
 *
 * Exported because the composer needs the identical normalisation for a different reason: the SDK's
 * own `accept` check is a case-sensitive `file.type === filter`, so anything handed to it has to
 * have been through here first or the two gates disagree about the same file.
 */
export function mediaTypeOf(mimeType: string): string {
  return mimeType.toLowerCase().split(";")[0].trim();
}

/**
 * CLAIMS THAT NAME NO FORMAT AT ALL, AND THE ONE COPY OF THAT LIST.
 *
 * A browser sends one of these for a file it has no mapping for — a `.txt` dragged out of an
 * editor, anything with an unfamiliar extension. Both sides have to treat them the same way and for
 * a while did not: `sniffMimeType` discards such a claim and reads the bytes, so the server accepts
 * the text file behind it, while the composer screened the claim alone and refused the same file at
 * pick time. That is the drift the note at the top of this file exists to prevent, so the list lives
 * here and `attachment-mime.ts` reads it from here rather than keeping its own.
 *
 * A blank claim, and anything not shaped like a MIME type, names nothing by the same reasoning.
 */
const MIME_NAMES_NOTHING: ReadonlySet<string> = new Set([
  "application/octet-stream",
  "binary/octet-stream",
  "application/unknown",
  "application/force-download",
]);

export function namesNoFormat(mimeType: string): boolean {
  const mediaType = mediaTypeOf(mimeType);
  return !mediaType.includes("/") || MIME_NAMES_NOTHING.has(mediaType);
}

export type AttachmentKind =
  | "image"
  | "text"
  | "unsupported-image"
  | "unsupported";

/**
 * `unsupported-image` is a separate answer from `unsupported` so the refusal can name the real
 * problem. "That is not a supported image type" tells somebody with a HEIC photo what to do;
 * "unsupported file" leaves them guessing whether images work at all.
 *
 * THE MIME TYPE AND NOTHING ELSE, WHICH IS WHY THE FILENAME IS NO LONGER ASKED FOR. This took
 * `{ name, mimeType }` and read only the second, and the dead argument was not merely untidy: it
 * read as a promise that a file called `notes.txt` would be given the benefit of the doubt, which
 * is exactly the doubt the composer's callers had. It must not be kept, either. The server calls
 * this with the type `sniffMimeType` earned from the BYTES, and letting an extension override that
 * answer is how a binary blob named `.txt` — or an empty file, which sniffs to
 * `application/octet-stream` on purpose — would be stored and served from this origin as text.
 * Where a name does deserve the benefit of the doubt the door is `namesNoFormat`, and it is the
 * client's alone: it is asked of a claim nobody has corroborated yet, not of an answer the bytes
 * have already given.
 */
export function classifyAttachment(mimeType: string): AttachmentKind {
  // The media type is what decides here — see `mediaTypeOf`. Without this the composer refused a
  // file the server then accepted, which is the drift the note at the top of this file exists to
  // prevent.
  const mediaType = mediaTypeOf(mimeType);
  if ((ACCEPTED_IMAGE_MIME as readonly string[]).includes(mediaType)) {
    return "image";
  }
  if (mediaType.startsWith("image/")) return "unsupported-image";
  if ((ACCEPTED_TEXT_MIME as readonly string[]).includes(mediaType)) {
    return "text";
  }
  return "unsupported";
}

/**
 * Whether a paste is an attachment or an ordinary text paste.
 *
 * Text wins whenever the clipboard carries any. A file is ours only when there is no text to
 * prefer.
 *
 * This is the one rule here that is NOT T3 Code's. Theirs claims an image even when text came with
 * it, on the reasoning that an image is unambiguously what was meant. It is not: copying a cell
 * from a spreadsheet, or a block from a word processor, puts an `image/png` rendering of the
 * selection on the clipboard ALONGSIDE the text. Under "an image always wins" that paste attached a
 * screenshot of the cell and typed nothing — verified in Chrome against the real app, not
 * theorised.
 *
 * A screenshot carries no text at all, so the case this rule exists for is untouched.
 *
 * DECLINING A PASTE IS NOT THE SAME AS THE TEXT BEING TYPED, AND FOR SOME SOURCES IT IS NOT WHAT
 * HAPPENS. This function only says "not ours". What the paste then does is PromptArea's rule, and
 * that rule is: if the clipboard carries an image AND the `text/html` is Microsoft Office markup —
 * it looks for `urn:schemas-microsoft-com:office`, a `ProgId` name, a `Mso` class, or an `mso-*`
 * property — the text is inserted and the image ignored. Anything else with an image on the
 * clipboard calls `onImagePaste` and RETURNS, having already called `preventDefault` and inserted
 * nothing.
 *
 * So the two halves of the spreadsheet case land differently, and it is worth being exact about
 * which is which:
 *
 *   - Word and Excel put Office markup in the `text/html`. Their text is typed. Handled.
 *   - Google Sheets, Numbers, and every other source that behaves like Office without being it,
 *     take the second branch. Their IMAGE is attached and THEIR TEXT IS NOT TYPED — the same
 *     outcome "an image always wins" gave, arrived at by a different route.
 *
 * That second bullet is a known defect and is being kept for now rather than fixed, so this comment
 * says so instead of implying the spreadsheet case is covered. Fixing it means teaching this rule
 * or PromptArea's the difference between an image that IS the selection and an image that merely
 * accompanies it, which no clipboard flag reports.
 *
 * The other trade-off, stated because it is also real: copying an image from a web page can put the
 * image's URL in `text/plain`, and that pastes the URL rather than attaching the image. Drag or the
 * `+` button still attach it, and a URL landing in the box is visible and undoable — where a
 * swallowed paste is neither.
 */
export function shouldClaimPaste(input: {
  kinds: readonly AttachmentKind[];
  plainText: string;
}): boolean {
  if (input.plainText.length > 0) return false;
  return input.kinds.length > 0;
}

/**
 * What a sent message carries instead of the bytes.
 *
 * This is an AG-UI `image`/`document` part with a URL source, NOT a part type of our own. AG-UI has
 * no reference member: `RunAgentInputSchema.parse` rejects `{type:"attachment"}` outright and the
 * runtime answers 400, verified against the installed 0.0.59 schema. The union is
 * `text | image | audio | video | document | binary`, and a source is `{type:"data"|"url"}` — there
 * is no `base64` literal and no `media_type` key anywhere.
 *
 * The URL is relative and is never fetched by a provider. `copilot.ts` swaps the source for
 * `{type:"data", value:<base64>}` as the run is built, which is what keeps the bytes out of the
 * stored thread while still putting the image in front of the model. `metadata` carries the id
 * because it is `z.unknown().optional()` and survives the parse; a sibling key would be silently
 * stripped.
 */
export type AttachmentSource =
  | { type: "data"; value: string; mimeType: string }
  | { type: "url"; value: string; mimeType?: string };

export type AttachmentPart = {
  type: "image" | "document";
  source: AttachmentSource;
  metadata?: { attachmentId: string; filename?: string };
};

/** The relative URL form stored in a sent message. Resolved server-side, never by a provider. */
export function attachmentUrl(attachmentId: string): string {
  return `/api/attachments/${attachmentId}`;
}