The Open Framework for Building Agents
fix(eve): keep tool-result media out of session history (#4224)
Andrew Barba0df9bbf66317
diff --git a/.changeset/tool-result-media-refs.md b/.changeset/tool-result-media-refs.mdnew file mode 100644index 000000000..acd26c143--- /dev/null+++ b/.changeset/tool-result-media-refs.md@@ -0,0 +1,5 @@+---+"eve": patch+---++Files that tools return to the model now live in the session sandbox, with only a reference in session history, so workflow state no longer re-stores image bytes on every step; each model call sends the same file, keeping it visible on later turns and the provider's prompt cache valid. Compaction now counts images at roughly what providers bill for them instead of their base64 length, and `read_file` shows PNG, JPEG, GIF, and WebP files to the model as images. Message attachments and tool files now stage under `/workspace/.eve/attachments` instead of `/workspace/attachments`, so they no longer collide with an agent's own files.diff --git a/docs/channels/slack.mdx b/docs/channels/slack.mdxindex 34156a1ec..b6de8b83c 100644--- a/docs/channels/slack.mdx+++ b/docs/channels/slack.mdx@@ -798,7 +798,7 @@ Private file downloads resolve credentials from the session's app installation wDownloads cover Slack's own file hosts plus URLs under a configured [`api.fileBaseUrl`](#point-the-channel-at-a-different-slack-api-host), and travel on `api.fetch` when one is configured.-Staged files live in the session sandbox under `/workspace/attachments`. Session history keeps a reference to that path, not a second copy of the Slack file. If the backend can no longer reattach the original sandbox, eve cannot restore the attachment bytes from history; later turns receive a missing-file result. Store files outside the sandbox when the agent must retain them independently of sandbox availability.+Staged files live in the session sandbox under `/workspace/.eve/attachments`. Session history keeps a reference to that path, not a second copy of the Slack file. If the backend can no longer reattach the original sandbox, eve cannot restore the attachment bytes from history; later turns receive a missing-file result. Store files outside the sandbox when the agent must retain them independently of sandbox availability.## What to read nextdiff --git a/docs/concepts/built-in-tools.md b/docs/concepts/built-in-tools.mdindex e24e429f2..2ae1c0083 100644--- a/docs/concepts/built-in-tools.md+++ b/docs/concepts/built-in-tools.md@@ -64,7 +64,7 @@ export default disableTool();### `read_file`-`read_file` reads text files from the sandbox with line-numbered output. It accepts absolute paths and paths beginning with `$HOME/`.+`read_file` reads text files from the sandbox with line-numbered output, and shows PNG, JPEG, GIF, and WebP images up to 3 MiB to the model as images. It accepts absolute paths and paths beginning with `$HOME/`.```sheve add tool/read_filediff --git a/docs/guides/dynamic-capabilities.md b/docs/guides/dynamic-capabilities.mdindex 4b7ec8b60..97b8e3edd 100644--- a/docs/guides/dynamic-capabilities.md+++ b/docs/guides/dynamic-capabilities.md@@ -55,7 +55,7 @@ export default defineAgent({});```-eve stages byte-backed `file` parts under `/workspace/attachments` before+eve stages byte-backed `file` parts under `/workspace/.eve/attachments` before`step.started`, but keeps their media type in `ctx.messages`. When an imagereaches the provider, vision models can process it and non-vision models rejectit. eve does not reroute automatically. See [Inbounddiff --git a/docs/sandbox/index.mdx b/docs/sandbox/index.mdxindex c5f8dfa46..62e8595ff 100644--- a/docs/sandbox/index.mdx+++ b/docs/sandbox/index.mdx@@ -149,6 +149,17 @@ interface SandboxProviderResourceTree {Docker and microsandbox mount the compiled trees read-only under `/eve/resources`, copy workspace seeds into writable `/workspace`, and expose skills read-only where possible. Vercel's snapshot environment writes workspace and skill contents before capturing its snapshot. Custom providers decide whether to upload, mount, or write each tree, but must expose the final target paths before authored preparation runs.+## Inbound attachments++eve writes byte-backed files that reach the agent into the sandbox under `/workspace/.eve/attachments/` and keeps only a reference in session history. This covers files attached to user messages and files that tools return to the model with `toolOutputPart.file` or as MCP image, audio, and blob content.++Before every model call, eve reads the referenced files back from the sandbox:++- An image up to 3 MiB or a PDF up to 20 MiB attached to a message reaches the model as a file. Larger files and other types reach the model as a text reference, such as `Attached file /workspace/.eve/attachments/<hash>/report.csv (text/csv)`, that the agent can open with `read_file` or `bash`.+- A file returned by a tool always reaches the model as a file.++Each call sends the same bytes for the same file, so the model still sees attachments from earlier turns and the provider's prompt cache stays valid. Compaction replaces files in older tool results with their text reference. The `read_file` tool shows PNG, JPEG, GIF, and WebP files up to 3 MiB to the model as images, so the agent can look at an image again after compaction.+## Custom providersDefine a provider with `defineSandboxProvider()` from `eve/sandbox/provider`. The five generic parameters describe environment options, live options, the prepared artifact, minimal serialized session state, and the provider-specific sandbox session type:diff --git a/docs/tools/overview.mdx b/docs/tools/overview.mdxindex 08a4882fe..dad58347b 100644--- a/docs/tools/overview.mdx+++ b/docs/tools/overview.mdx@@ -307,9 +307,11 @@ export default defineTool({The `toolOutput.text` and `toolOutput.json` builders construct the other two output shapes; hand-written literals remain valid everywhere.-File payloads must be base64 strings — raw bytes (`Uint8Array`, `Buffer`) are rejected because they do not survive eve's durable JSON boundary. Keep payloads small: a content-part image is persisted in session history and re-sent on every subsequent model call, and eve warns above 3 MiB. Sending image parts to a model without vision support fails with that provider's error, the same as image parts in user messages.+File payloads must be base64 strings — raw bytes (`Uint8Array`, `Buffer`) are rejected because they do not survive eve's durable JSON boundary. eve warns above 3 MiB. Sending image parts to a model without vision support fails with that provider's error, the same as image parts in user messages.-When older turns are compacted, file payloads are dropped from the summary and replaced with a text stub naming the file and media type — the model cannot re-see a compacted image. Content parts are for "look at this now"; if the agent may need an artifact again later, write it to the [sandbox](/docs/sandbox) and return its path.+eve writes each returned file to the session sandbox and keeps only a reference in session history, so the file's bytes are not stored again on every workflow step. Every later model call sends the same file, which keeps it visible on later turns and keeps the provider's prompt cache valid. See [Inbound attachments](/docs/sandbox#inbound-attachments). Compaction counts an image at roughly what providers bill for it rather than its base64 length.++When compaction drops older tool results, each file becomes a text reference such as `Attached file /workspace/.eve/attachments/<hash>/chart.png (image/png)`. The built-in `read_file` tool shows PNG, JPEG, GIF, and WebP files up to 3 MiB to the model as images, so the agent can open a compacted image again.Do not return secrets, credentials, unnecessary personal data, or unbounded sensitive content from tools. Filter, minimize, and redact tool outputs before returning them.diff --git a/e2e/fixtures/agent-compaction-regressions/agent/agent.ts b/e2e/fixtures/agent-compaction-regressions/agent/agent.tsindex 9042c53a0..b35dc6d3b 100644--- a/e2e/fixtures/agent-compaction-regressions/agent/agent.ts+++ b/e2e/fixtures/agent-compaction-regressions/agent/agent.ts@@ -17,9 +17,9 @@ import { HANDOFF_REFERENCE, REVIEW_REFERENCE } from "../release-reports";const TEST_CONTEXT_WINDOW_TOKENS = 32_000;// The compiled fixture's instructions, task system block, and 11 advertised-// tools occupy ~2,113 tokens. Reserve them in addition to the summarizer's-// history budget.-const TEST_REQUEST_ENVELOPE_TOKENS = 2_113;+// tools occupy ~2,163 tokens, including `read_file`'s image guidance. Reserve+// them in addition to the summarizer's history budget.+const TEST_REQUEST_ENVELOPE_TOKENS = 2_163;// Fit the capped file-output exchange, while forcing the larger review and// handoff reports into the assistant checkpoint consumed by the script.const TEST_HISTORY_BUDGET_TOKENS = 900;@@ -102,7 +102,9 @@ const taskModel = mockModel({const diagnostics = [toolText.includes(CONTENT_OUTPUT_TAIL_MARKER) ? "TAIL_PRESERVED" : "TAIL_LOST",toolText.includes(CONTENT_OUTPUT_LEAD_MARKER) ? "LEAD_PRESERVED" : "LEAD_LOST",- toolText.includes(`Attached file ${CONTENT_OUTPUT_FILENAME}`)+ // The stub names the file's sandbox path, which ends in its filename.+ toolText.includes("Attached file ") &&+ toolText.includes(`${CONTENT_OUTPUT_FILENAME} (application/octet-stream)`)? "FILE_STUB_RENDERED": "FILE_STUB_LOST",request.userMessages.some((text) => text.includes("[case: content-output-file-stub]"))diff --git a/e2e/fixtures/agent-compaction-regressions/evals/content-output-file-stub.eval.ts b/e2e/fixtures/agent-compaction-regressions/evals/content-output-file-stub.eval.tsindex 18662c5ec..90464be5d 100644--- a/e2e/fixtures/agent-compaction-regressions/evals/content-output-file-stub.eval.ts+++ b/e2e/fixtures/agent-compaction-regressions/evals/content-output-file-stub.eval.ts@@ -9,7 +9,7 @@ import { CONTENT_OUTPUT_COMPACTION_MARKER } from "../constants";// capped result honors the full contract:// - the raw payload is gone (a canary buried in the base64 detects the cap// and any leak);-// - the file rendered as its `Attached file <name> (<mediaType>)` stub;+// - the file rendered as its `Attached file <path> (<mediaType>)` stub;// - the text parts around the file survived in place (lead + tail markers);// - the surrounding conversation was untouched (the case's own user text).// On violation the model emits granular *_LOST diagnostics instead, so adiff --git a/e2e/fixtures/agent-prompt-cache/agent/tools/floor_plan.ts b/e2e/fixtures/agent-prompt-cache/agent/tools/floor_plan.tsnew file mode 100644index 000000000..85eb8d9f6--- /dev/null+++ b/e2e/fixtures/agent-prompt-cache/agent/tools/floor_plan.ts@@ -0,0 +1,67 @@+import { crc32, deflateSync } from "node:zlib";+import { defineTool, toolOutput, toolOutputPart } from "eve/tools";+import { z } from "zod";++const WIDTH = 768;+const HEIGHT = 512;+const STAGE_HEIGHT = 96;+const WHITE = [255, 255, 255] as const;+const BLUE = [0, 0, 255] as const;+const ORANGE = [255, 140, 0] as const;++// The stage is a blue band along the top; eight orange tables sit in a+// four-by-two grid below it. Large enough for providers to bill hundreds of+// image tokens, so the image is a real share of the cached prefix.+function pixel(x: number, y: number): readonly [number, number, number] {+ if (y < STAGE_HEIGHT) return BLUE;+ const cellX = x % (WIDTH / 4);+ const cellY = (y - STAGE_HEIGHT) % ((HEIGHT - STAGE_HEIGHT) / 2);+ return cellX >= 48 && cellX < 144 && cellY >= 64 && cellY < 144 ? ORANGE : WHITE;+}++function pngChunk(type: string, data: Buffer): Buffer {+ const length = Buffer.alloc(4);+ length.writeUInt32BE(data.length);+ const typeBuffer = Buffer.from(type, "latin1");+ const crc = Buffer.alloc(4);+ crc.writeUInt32BE(crc32(Buffer.concat([typeBuffer, data])) >>> 0);+ return Buffer.concat([length, typeBuffer, data, crc]);+}++function renderFloorPlanPng(): Buffer {+ const ihdr = Buffer.alloc(13);+ ihdr.writeUInt32BE(WIDTH, 0);+ ihdr.writeUInt32BE(HEIGHT, 4);+ ihdr[8] = 8; // bit depth+ ihdr[9] = 2; // truecolor+ const rows = Array.from({ length: HEIGHT }, (_, y) => {+ const row = Buffer.alloc(1 + WIDTH * 3); // leading scanline filter byte 0+ for (let x = 0; x < WIDTH; x += 1) row.set(pixel(x, y), 1 + x * 3);+ return row;+ });+ return Buffer.concat([+ Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),+ pngChunk("IHDR", ihdr),+ pngChunk("IDAT", deflateSync(Buffer.concat(rows), { level: 9 })),+ pngChunk("IEND", Buffer.alloc(0)),+ ]);+}++export default defineTool({+ description:+ "Show the main hall floor plan as an image. Only call when the user explicitly asks " ++ "to use `floor_plan`. The layout appears ONLY in the image; inspect it visually.",+ inputSchema: z.object({}),+ async execute() {+ return { room: "main hall" };+ },+ toModelOutput() {+ return toolOutput.content([+ toolOutputPart.text("Main hall floor plan:"),+ toolOutputPart.file(renderFloorPlanPng().toString("base64"), {+ filename: "main-hall-floor-plan.png",+ mediaType: "image/png",+ }),+ ]);+ },+});diff --git a/e2e/fixtures/agent-prompt-cache/cache-checks.ts b/e2e/fixtures/agent-prompt-cache/cache-checks.tsnew file mode 100644index 000000000..82f9b97a7--- /dev/null+++ b/e2e/fixtures/agent-prompt-cache/cache-checks.ts@@ -0,0 +1,47 @@+import assert from "node:assert/strict";+import type { EveEvalContext, EveEvalTurn } from "eve/evals";+import { satisfies } from "eve/evals/expect";++/** Asserts a turn ran on the matrix model without failures. */+export function expectHealthyTurn(turn: EveEvalTurn) {+ turn.expectOk();+ turn.noFailedActions();+ turn.notEvent("step.failed");+ turn.notEvent("step.completed", { data: { finishReason: "content-filter" } });+ const steps = turn.events.filter((event) => event.type === "step.started");+ assert(steps.length > 0, "the turn calls a model");+ assert(+ steps.every(({ data }) => data.modelId === process.env.EVE_E2E_MODEL),+ "each request uses the real matrix model",+ );+}++/**+ * Asserts that every parent request after the first reads at least 98% of the+ * preceding request's input from the provider cache, across the given turns.+ */+export function expectCacheReuse(t: EveEvalContext, turns: readonly EveEvalTurn[]) {+ const steps = turns.flatMap((turn) =>+ turn.events.filter((event) => event.type === "step.completed"),+ );+ assert(steps.length >= 2, "multiple parent requests exercise cache reuse");+ assert(+ (steps[0]?.data.usage?.inputTokens ?? 0) >= 4_096,+ "the first request is large enough for conversation caching",+ );+ for (let index = 1; index < steps.length; index += 1) {+ const previousInput = steps[index - 1]!.data.usage?.inputTokens;+ const usage = steps[index]!.data.usage;+ assert(previousInput !== undefined, "preceding input token usage is present");+ assert(usage?.cacheReadTokens !== undefined, "provider cache-read usage is present");+ t.log(`Parent request ${index + 1}: ${JSON.stringify({ previousInput, ...usage })}`);+ // Allow a small margin for differences in provider token counts.+ t.check(+ usage.cacheReadTokens / previousInput,+ satisfies(+ (ratio: number) => ratio >= 0.98,+ `request ${index + 1} reads at least 98% of the preceding input from the provider cache`,+ ),+ );+ }+}diff --git a/e2e/fixtures/agent-prompt-cache/evals/parallel-review.eval.ts b/e2e/fixtures/agent-prompt-cache/evals/parallel-review.eval.tsindex 60fd4a0fb..d37264aa5 100644--- a/e2e/fixtures/agent-prompt-cache/evals/parallel-review.eval.ts+++ b/e2e/fixtures/agent-prompt-cache/evals/parallel-review.eval.ts@@ -1,7 +1,8 @@import assert from "node:assert/strict";import { defineEval, type EveEvalContext, type EveEvalTurn } from "eve/evals";-import { satisfies } from "eve/evals/expect";import { z } from "zod";+import { expectCacheReuse, expectHealthyTurn } from "../cache-checks";+import { EVENT_OVERVIEW } from "../event-overview";import { purchasingSheets } from "../purchasing-sheets";const reviewSchema = z.object({@@ -34,7 +35,7 @@ export default ["first", "later"].map((launchTurn) =>expectFiveReviewers(started);expectReviewSummary(started);await expectParallelReviews(t, started);- expectCacheReuse(t, started);+ expectCacheReuse(t, [started]);},}),);@@ -77,61 +78,6 @@ async function expectParallelReviews(t: EveEvalContext, turn: EveEvalTurn) {assert(lastStarted < firstFinished, "all five reviews overlap");}-function expectHealthyTurn(turn: EveEvalTurn) {- turn.expectOk();- turn.noFailedActions();- turn.notEvent("step.failed");- turn.notEvent("step.completed", { data: { finishReason: "content-filter" } });- const steps = turn.events.filter((event) => event.type === "step.started");- assert(steps.length > 0, "the turn calls a model");- assert(- steps.every(({ data }) => data.modelId === process.env.EVE_E2E_MODEL),- "each request uses the real matrix model",- );-}--function expectCacheReuse(t: EveEvalContext, turn: EveEvalTurn) {- const steps = turn.events.filter((event) => event.type === "step.completed");- assert(steps.length >= 2, "multiple parent requests exercise cache reuse");- assert(- (steps[0]?.data.usage?.inputTokens ?? 0) >= 4_096,- "review packet is large enough for conversation caching",- );- for (let index = 1; index < steps.length; index += 1) {- const previousInput = steps[index - 1]!.data.usage?.inputTokens;- const usage = steps[index]!.data.usage;- assert(previousInput !== undefined, "preceding input token usage is present");- assert(usage?.cacheReadTokens !== undefined, "provider cache-read usage is present");- t.log(`Parent request ${index + 1}: ${JSON.stringify({ previousInput, ...usage })}`);- // Allow a small margin for differences in provider token counts.- t.check(- usage.cacheReadTokens / previousInput,- satisfies(- (ratio: number) => ratio >= 0.98,- `request ${index + 1} reads at least 98% of the preceding input from the provider cache`,- ),- );- }-}--// Keeps the first request above the provider's cache minimum without relying-// on the size of the default tool descriptions.-const EVENT_OVERVIEW = `Event overview: The weekend programme runs on Saturday and Sunday in the ground-floor hall, the studio, and the two meeting rooms beside the garden. Alice coordinates the tutors and volunteers, while Bob manages purchasing, supplier contact, and the shared budget sheet kept in the office. The centre's caretaker opens the building each morning and holds the keys for the storeroom, the side entrance, and the equipment cupboard.--Visitors book workshop places through the front desk or the centre's newsletter. Most sessions are for adults, but the Sunday afternoon drawing workshop welcomes families, so the tutors keep a small box of simpler materials for younger visitors. Volunteers greet arrivals at the main entrance, direct them to the right room, and collect short feedback cards at the end of each session. Tea and water are available in the garden room throughout the day.--The budget for the weekend was agreed at the last committee meeting. It covers materials, equipment hire, printed signs, and a modest amount for refreshments. Bob records each purchase against the budget sheet as soon as an order is confirmed, and Alice reviews the running total every Wednesday. Any change above the agreed amounts needs a short note explaining why it is needed and what it replaces.--Accessibility is part of every room plan. The hall and the studio have step-free access from the side entrance, and the meeting rooms share a ramp from the garden path. Tutors keep one table in each room clear of stools for visitors who use wheelchairs, and printed instructions come in a larger type size on request. The front desk keeps a short list of quiet spaces for anyone who needs a break from the busier sessions.--Supplies arrive in two deliveries. The first comes on Thursday afternoon with paper, paint, clay, and the printed signs, and the caretaker signs for it at the side entrance. The second comes on Friday morning with the hired easels, the portable kiln, and the extra folding tables. Bob checks each delivery against the order confirmation, notes any missing items on the budget sheet, and tells Alice before the end of the day so the tutors can adjust their plans if something is late.--Each workshop has a named tutor and one volunteer helper. The tutor sets up the room before the first session, explains the safety notes for any tools or materials, and keeps an eye on the time so each group finishes promptly. The helper hands out materials, keeps the tables tidy between sessions, and fetches anything the tutor needs from the storeroom. At the end of each day the tutors leave a short note in the office about what went well, what ran short, and anything that should be ordered differently next time.--Planning before the event happens through a shared noticeboard in the office and a weekly email from Alice. The email lists the confirmed workshops, the number of places booked so far, and any changes to rooms or times. Tutors reply with their final materials lists by the Monday before the event, which gives Bob enough time to place the last orders and confirm delivery slots with the suppliers.--After the event, volunteers pack away the reusable materials, count what remains, and label the storeroom shelves so the next programme can start from an accurate inventory. The committee would like each review to be practical and brief, focused on whether the current plan works as written rather than on redesigning the weekend.`;-function reviewPacket(): string {const sheets = purchasingSheets.map(diff --git a/e2e/fixtures/agent-prompt-cache/evals/tool-image-cache.eval.ts b/e2e/fixtures/agent-prompt-cache/evals/tool-image-cache.eval.tsnew file mode 100644index 000000000..2aec12874--- /dev/null+++ b/e2e/fixtures/agent-prompt-cache/evals/tool-image-cache.eval.ts@@ -0,0 +1,52 @@+import { defineEval } from "eve/evals";+import { expectCacheReuse, expectHealthyTurn } from "../cache-checks";+import { EVENT_OVERVIEW } from "../event-overview";+import { purchasingSheets } from "../purchasing-sheets";++// A tool image must stay byte-identical in every later prompt: session history+// keeps a sandbox ref and each model call loads the same bytes. Rewriting an+// earlier image (for example, stubbing it when a turn ends) breaks the+// provider's cached prefix, and Anthropic invalidates its whole message cache+// when images appear or disappear. The follow-up turns therefore gate on+// reading the preceding prompt, image included, from the provider cache.+export default defineEval({+ tags: ["real-model"],+ description: "A tool image stays in the cached prompt prefix on later turns.",+ async test(t) {+ const session = await t.session();++ const shown = await session.send(+ "Alice is laying out the main hall for the community centre event, and Bob will print " ++ "the room signs. Please call `floor_plan` exactly once, look at the image, and tell Bob " ++ `in one short sentence which colour marks the stage.\n\n${eventBackground()}`,+ );+ expectHealthyTurn(shown);+ shown.calledTool("floor_plan", { count: 1 });++ const tables = await session.send(+ "Bob is preparing table numbers. Without calling any tool, look at the floor plan again " ++ "and tell him how many tables it shows.",+ );+ const stage = await session.send(+ "Alice wants one last check before the signs are printed. Without calling any tool, " ++ "which colour marks the stage on the floor plan?",+ );+ for (const turn of [tables, stage]) {+ expectHealthyTurn(turn);+ turn.usedNoTools();+ }+ for (const turn of [shown, tables, stage]) turn.notEvent("compaction.completed");++ // Live vision answers vary, so they are tracked rather than gated.+ tables.messageIncludes(/\b(8|eight)\b/iu).soft();+ stage.messageIncludes(/blue/iu).soft();++ expectCacheReuse(t, [shown, tables, stage]);+ },+});++// Keeps the first request above every provider's cache minimum.+function eventBackground(): string {+ const sheets = purchasingSheets.map((sheet) => `${sheet.title}\n\n${sheet.notes}`).join("\n\n");+ return `${EVENT_OVERVIEW}\n\n${sheets}`;+}diff --git a/e2e/fixtures/agent-prompt-cache/event-overview.ts b/e2e/fixtures/agent-prompt-cache/event-overview.tsnew file mode 100644index 000000000..f38965103--- /dev/null+++ b/e2e/fixtures/agent-prompt-cache/event-overview.ts@@ -0,0 +1,17 @@+// Keeps the first request above the provider's cache minimum without relying+// on the size of the default tool descriptions.+export const EVENT_OVERVIEW = `Event overview: The weekend programme runs on Saturday and Sunday in the ground-floor hall, the studio, and the two meeting rooms beside the garden. Alice coordinates the tutors and volunteers, while Bob manages purchasing, supplier contact, and the shared budget sheet kept in the office. The centre's caretaker opens the building each morning and holds the keys for the storeroom, the side entrance, and the equipment cupboard.++Visitors book workshop places through the front desk or the centre's newsletter. Most sessions are for adults, but the Sunday afternoon drawing workshop welcomes families, so the tutors keep a small box of simpler materials for younger visitors. Volunteers greet arrivals at the main entrance, direct them to the right room, and collect short feedback cards at the end of each session. Tea and water are available in the garden room throughout the day.++The budget for the weekend was agreed at the last committee meeting. It covers materials, equipment hire, printed signs, and a modest amount for refreshments. Bob records each purchase against the budget sheet as soon as an order is confirmed, and Alice reviews the running total every Wednesday. Any change above the agreed amounts needs a short note explaining why it is needed and what it replaces.++Accessibility is part of every room plan. The hall and the studio have step-free access from the side entrance, and the meeting rooms share a ramp from the garden path. Tutors keep one table in each room clear of stools for visitors who use wheelchairs, and printed instructions come in a larger type size on request. The front desk keeps a short list of quiet spaces for anyone who needs a break from the busier sessions.++Supplies arrive in two deliveries. The first comes on Thursday afternoon with paper, paint, clay, and the printed signs, and the caretaker signs for it at the side entrance. The second comes on Friday morning with the hired easels, the portable kiln, and the extra folding tables. Bob checks each delivery against the order confirmation, notes any missing items on the budget sheet, and tells Alice before the end of the day so the tutors can adjust their plans if something is late.++Each workshop has a named tutor and one volunteer helper. The tutor sets up the room before the first session, explains the safety notes for any tools or materials, and keeps an eye on the time so each group finishes promptly. The helper hands out materials, keeps the tables tidy between sessions, and fetches anything the tutor needs from the storeroom. At the end of each day the tutors leave a short note in the office about what went well, what ran short, and anything that should be ordered differently next time.++Planning before the event happens through a shared noticeboard in the office and a weekly email from Alice. The email lists the confirmed workshops, the number of places booked so far, and any changes to rooms or times. Tutors reply with their final materials lists by the Monday before the event, which gives Bob enough time to place the last orders and confirm delivery slots with the suppliers.++After the event, volunteers pack away the reusable materials, count what remains, and label the storeroom shelves so the next programme can start from an accurate inventory. The committee would like each review to be practical and brief, focused on whether the current plan works as written rather than on redesigning the weekend.`;diff --git a/e2e/fixtures/agent-tools-hitl/agent/agent.ts b/e2e/fixtures/agent-tools-hitl/agent/agent.tsindex bb8bcd9d3..331976ece 100644--- a/e2e/fixtures/agent-tools-hitl/agent/agent.ts+++ b/e2e/fixtures/agent-tools-hitl/agent/agent.ts@@ -9,7 +9,7 @@ const SCOPED_APPROVAL_DIRECTIVE =/call the dynamic_scoped_approval tool exactly once with scope "([^"]+)"/iu;const REPLY_DIRECTIVE = /reply with exactly ([A-Z0-9-]+)/iu;const APPROVAL_FOLLOWUP_DIRECTIVE =- /call the (gate|read-status) tool exactly once with marker "([^"]+)"/iu;+ /call the (gate|read-draft-status) tool exactly once with marker "([^"]+)"/iu;const ASK_QUESTION_DIRECTIVE = /call the ask_question tool exactly once with question "([^"]+)"/iu;/**diff --git a/e2e/fixtures/agent-tools-hitl/agent/tools/read-status.ts b/e2e/fixtures/agent-tools-hitl/agent/tools/read-draft-status.tssimilarity index 56%rename from e2e/fixtures/agent-tools-hitl/agent/tools/read-status.tsrename to e2e/fixtures/agent-tools-hitl/agent/tools/read-draft-status.tsindex 2369030b6..8b7a02a42 100644--- a/e2e/fixtures/agent-tools-hitl/agent/tools/read-status.ts+++ b/e2e/fixtures/agent-tools-hitl/agent/tools/read-draft-status.ts@@ -2,7 +2,9 @@ import { defineTool } from "eve/tools";import { z } from "zod";export default defineTool({- description: "Read the current draft status without requiring approval.",+ description:+ "Read the review status of Alice's draft document without requiring approval. " ++ "Only call when the user explicitly asks to read the draft status.",inputSchema: z.object({ marker: z.string() }),async execute({ marker }) {return { marker, status: "ready" };diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/pending-approval-followup-dedup.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/pending-approval-followup-dedup.eval.tsindex e0b481945..ab43802cd 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/pending-approval-followup-dedup.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/pending-approval-followup-dedup.eval.ts@@ -4,7 +4,7 @@ import { equals } from "eve/evals/expect";const MARKER = "followup-dedup-H4K8";const TOOL_NAME = "gate";const FOLLOW_UP_QUESTIONS = [- "What is the current status of my request?",+ "Is the gate call still waiting for my approval?","Has the requested action executed yet?","What are you waiting for before continuing?","Can you summarize what remains blocked?",diff --git a/e2e/fixtures/agent-tools-hitl/evals/hitl/pending-approval-tool-followup.eval.ts b/e2e/fixtures/agent-tools-hitl/evals/hitl/pending-approval-tool-followup.eval.tsindex 5f506ac52..cdbf321af 100644--- a/e2e/fixtures/agent-tools-hitl/evals/hitl/pending-approval-tool-followup.eval.ts+++ b/e2e/fixtures/agent-tools-hitl/evals/hitl/pending-approval-tool-followup.eval.ts@@ -4,7 +4,7 @@ import { requestFrom } from "./continuation/helpers.ts";const MARKER = "draft-status-3494";const READ_STATUS =- `Alice is checking her draft. Call the read-status tool exactly once with marker "${MARKER}". ` ++ `Alice is checking her draft. Call the read-draft-status tool exactly once with marker "${MARKER}". ` +"After the tool returns, tell Alice the status and marker from its result.";export default [@@ -23,7 +23,7 @@ export default [// Then the tool executes once and the completed reply includes its status and marker.turn.expectOk();- turn.calledTool("read-status", { status: "completed", count: 1 });+ turn.calledTool("read-draft-status", { status: "completed", count: 1 });turn.event("turn.completed", { count: 1 });turn.messageIncludes(MARKER);turn.messageIncludes("ready");@@ -34,7 +34,7 @@ export default [data: {turnId: received.data.turnId,status: "completed",- result: { toolName: "read-status" },+ result: { toolName: "read-draft-status" },},count: 1,},@@ -74,14 +74,14 @@ export default [);// Then the tool result reaches a completed reply without executing the account change.const result = await live.waitForEvent("action.result", {- data: { status: "completed", result: { toolName: "read-status" } },+ data: { status: "completed", result: { toolName: "read-draft-status" } },});t.log(`Follow-up tool completed before waiting for its reply: ${JSON.stringify(result)}`);const received = await live.waitForEvent("message.received");await live.waitForEvent("turn.completed", { data: { turnId: received.data.turnId } });const followup = await live.result();followup.expectOk();- followup.calledTool("read-status", { status: "completed", count: 1 });+ followup.calledTool("read-draft-status", { status: "completed", count: 1 });followup.messageIncludes(MARKER);followup.messageIncludes("ready");followup.eventOrder([@@ -96,7 +96,7 @@ export default [data: {turnId: received.data.turnId,status: "completed",- result: { toolName: "read-status" },+ result: { toolName: "read-draft-status" },},count: 1,},diff --git a/e2e/fixtures/agent-tools/agent/agent.ts b/e2e/fixtures/agent-tools/agent/agent.tsindex ec92ec7da..9ac25076c 100644--- a/e2e/fixtures/agent-tools/agent/agent.ts+++ b/e2e/fixtures/agent-tools/agent/agent.ts@@ -1,8 +1,44 @@import { e2eAgentConfig } from "@eve-e2e/config";import { defineAgent } from "eve";+import type { MockModelRequest } from "eve/evals";++import { readStripeColors, STRIPE_COUNT } from "./lib/stripes";++const STAGED_IMAGE_PATH = /\/workspace\/\.eve\/attachments\/\S+\.png/u;++/**+ * Stripe colors in the latest tool-result image the prompt carries, whether+ * `render-stripes` produced it or `read_file` reopened it.+ */+function promptStripeColors(request: MockModelRequest): string[] | undefined {+ for (const result of [...request.toolResults].reverse()) {+ if (!Array.isArray(result.output)) continue;+ for (const part of result.output as { type?: string; data?: { data?: unknown } }[]) {+ if (part.type === "file" && typeof part.data?.data === "string") {+ return readStripeColors(Buffer.from(part.data.data, "base64"), STRIPE_COUNT);+ }+ }+ }+ return undefined;+}const base = e2eAgentConfig({mock(request) {+ if (request.userMessages.some((message) => message.includes("`render-stripes`"))) {+ // The compaction summarizer runs without tools and needs a text summary.+ if (request.tools.length === 0) return "Alice asked for the colors of the rendered stripes.";+ const colors = promptStripeColors(request);+ if (colors !== undefined) return colors.join(", ");+ if (request.lastUserMessage?.includes("`render-stripes` exactly once")) {+ return { toolCalls: [{ name: "render-stripes", input: {} }] };+ }+ const stagedPath = request.lastUserMessage?.match(STAGED_IMAGE_PATH)?.[0];+ if (stagedPath !== undefined) {+ return { toolCalls: [{ name: "read_file", input: { filePath: stagedPath } }] };+ }+ // Every later turn answers from the image replayed out of history.+ throw new Error("The render-stripes image is missing from the replayed prompt.");+ }if (request.lastUserMessage?.includes("`callback_identity`")) {const roles = request.messages.map((message) => message.role);if (roles.lastIndexOf("tool") <= roles.lastIndexOf("user")) {diff --git a/e2e/fixtures/agent-tools/agent/lib/stripes.ts b/e2e/fixtures/agent-tools/agent/lib/stripes.tsnew file mode 100644index 000000000..13e4afa18--- /dev/null+++ b/e2e/fixtures/agent-tools/agent/lib/stripes.ts@@ -0,0 +1,83 @@+import { crc32, deflateSync, inflateSync } from "node:zlib";++// Single-token names no model paraphrases (unlike cyan/teal or purple/violet),+// so the eval can match the reply against the rendered sequence verbatim.+export const PALETTE = {+ black: [0, 0, 0],+ blue: [0, 0, 255],+ green: [0, 160, 0],+ orange: [255, 140, 0],+ red: [255, 0, 0],+ yellow: [255, 220, 0],+} as const;++export type ColorName = keyof typeof PALETTE;++export const STRIPE_COUNT = 3;++const WIDTH = 240;+const HEIGHT = 120;+const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);++function pngChunk(type: string, data: Buffer): Buffer {+ const length = Buffer.alloc(4);+ length.writeUInt32BE(data.length);+ const typeBuffer = Buffer.from(type, "latin1");+ const crc = Buffer.alloc(4);+ crc.writeUInt32BE(crc32(Buffer.concat([typeBuffer, data])) >>> 0);+ return Buffer.concat([length, typeBuffer, data, crc]);+}++/** Renders equal-width vertical stripes as an 8-bit truecolor PNG. */+export function renderStripesPng(stripes: readonly ColorName[]): Buffer {+ const ihdr = Buffer.alloc(13);+ ihdr.writeUInt32BE(WIDTH, 0);+ ihdr.writeUInt32BE(HEIGHT, 4);+ ihdr[8] = 8; // bit depth+ ihdr[9] = 2; // truecolor++ const stripeWidth = WIDTH / stripes.length;+ const row = Buffer.alloc(1 + WIDTH * 3); // leading scanline filter byte 0+ for (let x = 0; x < WIDTH; x += 1) {+ const stripe = Math.min(Math.floor(x / stripeWidth), stripes.length - 1);+ const color = PALETTE[stripes[stripe]!];+ row.set(color, 1 + x * 3);+ }+ const idat = deflateSync(Buffer.concat(Array.from({ length: HEIGHT }, () => row)), { level: 9 });++ return Buffer.concat([+ PNG_SIGNATURE,+ pngChunk("IHDR", ihdr),+ pngChunk("IDAT", idat),+ pngChunk("IEND", Buffer.alloc(0)),+ ]);+}++/**+ * Reads the stripe colors back out of a {@link renderStripesPng} image, so a+ * scripted model can answer only when the pixels actually reached it.+ */+export function readStripeColors(png: Buffer, stripeCount: number): ColorName[] | undefined {+ if (!png.subarray(0, 8).equals(PNG_SIGNATURE)) return undefined;+ const width = png.readUInt32BE(16);+ const idat: Buffer[] = [];+ for (let offset = 8; offset + 8 <= png.length;) {+ const length = png.readUInt32BE(offset);+ if (png.toString("latin1", offset + 4, offset + 8) === "IDAT") {+ idat.push(png.subarray(offset + 8, offset + 8 + length));+ }+ offset += 12 + length;+ }+ const firstRow = inflateSync(Buffer.concat(idat)).subarray(1, 1 + width * 3);+ const colors: ColorName[] = [];+ for (let stripe = 0; stripe < stripeCount; stripe += 1) {+ const x = Math.floor((width / stripeCount) * (stripe + 0.5));+ const pixel = [...firstRow.subarray(x * 3, x * 3 + 3)];+ const name = (Object.keys(PALETTE) as ColorName[]).find((candidate) =>+ PALETTE[candidate].every((channel, index) => channel === pixel[index]),+ );+ if (name === undefined) return undefined;+ colors.push(name);+ }+ return colors;+}diff --git a/e2e/fixtures/agent-tools/agent/tools/render-stripes.ts b/e2e/fixtures/agent-tools/agent/tools/render-stripes.tsindex 7f564b9fc..6a3c0d047 100644--- a/e2e/fixtures/agent-tools/agent/tools/render-stripes.ts+++ b/e2e/fixtures/agent-tools/agent/tools/render-stripes.ts@@ -1,56 +1,7 @@-import { crc32, deflateSync } from "node:zlib";import { defineTool, toolOutput, toolOutputPart } from "eve/tools";import { z } from "zod";-// Single-token names no model paraphrases (unlike cyan/teal or purple/violet),-// so the eval can match the reply against the rendered sequence verbatim.-const PALETTE = {- black: [0, 0, 0],- blue: [0, 0, 255],- green: [0, 160, 0],- orange: [255, 140, 0],- red: [255, 0, 0],- yellow: [255, 220, 0],-} as const;--type ColorName = keyof typeof PALETTE;--const WIDTH = 240;-const HEIGHT = 120;-const STRIPE_COUNT = 3;--function pngChunk(type: string, data: Buffer): Buffer {- const length = Buffer.alloc(4);- length.writeUInt32BE(data.length);- const typeBuffer = Buffer.from(type, "latin1");- const crc = Buffer.alloc(4);- crc.writeUInt32BE(crc32(Buffer.concat([typeBuffer, data])) >>> 0);- return Buffer.concat([length, typeBuffer, data, crc]);-}--function renderStripesPng(stripes: readonly ColorName[]): Buffer {- const ihdr = Buffer.alloc(13);- ihdr.writeUInt32BE(WIDTH, 0);- ihdr.writeUInt32BE(HEIGHT, 4);- ihdr[8] = 8; // bit depth- ihdr[9] = 2; // truecolor-- const stripeWidth = WIDTH / stripes.length;- const row = Buffer.alloc(1 + WIDTH * 3); // leading scanline filter byte 0- for (let x = 0; x < WIDTH; x += 1) {- const stripe = Math.min(Math.floor(x / stripeWidth), stripes.length - 1);- const color = PALETTE[stripes[stripe]!];- row.set(color, 1 + x * 3);- }- const idat = deflateSync(Buffer.concat(Array.from({ length: HEIGHT }, () => row)), { level: 9 });-- return Buffer.concat([- Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),- pngChunk("IHDR", ihdr),- pngChunk("IDAT", idat),- pngChunk("IEND", Buffer.alloc(0)),- ]);-}+import { type ColorName, PALETTE, renderStripesPng, STRIPE_COUNT } from "../lib/stripes";export default defineTool({description:diff --git a/e2e/fixtures/agent-tools/evals/static-tools/read-file-reopens-compacted-image.eval.ts b/e2e/fixtures/agent-tools/evals/static-tools/read-file-reopens-compacted-image.eval.tsnew file mode 100644index 000000000..3440f8679--- /dev/null+++ b/e2e/fixtures/agent-tools/evals/static-tools/read-file-reopens-compacted-image.eval.ts@@ -0,0 +1,66 @@+import { createHash } from "node:crypto";+import { defineEval } from "eve/evals";++import {+ assistantAnswers,+ isRenderStripesOutput,+ namesColorsInOrder,+ TOOL_NAME,+} from "./render-stripes-shared";++// Compaction summarizes the tool result away, but the image stays staged in+// the sandbox under a content-addressed path. `read_file` must show it to the+// model again. The eval derives the staged path from the tool's answer key, so+// the follow-up does not depend on the summary keeping the path. The+// fixture's mock model decodes the pixels and fails the turn when the+// reopened image is missing; live color recognition is tracked, not gated.+export default defineEval({+ description: "read_file reopens a tool image after compaction removes it from the prompt.",+ async test(t) {+ const shown = await t.send(+ `Call \`${TOOL_NAME}\` exactly once, look at the rendered image, and reply with only ` ++ "the stripe colors left to right, comma-separated.",+ );+ shown.expectOk();+ shown.calledTool(TOOL_NAME, { count: 1, output: isRenderStripesOutput });+ const output = shown.requireToolCall(TOOL_NAME).output as { readonly imageBase64: string };+ const session = shown.session;++ const liveCompaction = t.target.watchTurn(session.sessionId, {+ startIndex: session.events.length,+ });+ const response = await t.target.fetch(+ `/eve/v1/session/${encodeURIComponent(session.sessionId)}/compact`,+ { body: "{}", headers: { "content-type": "application/json" }, method: "POST" },+ );+ if (!response.ok) {+ throw new Error(+ `Compacting the session failed (${response.status}): ${await response.text()}`,+ );+ }+ const compacted = await liveCompaction.result();+ compacted.event("compaction.completed", { count: 1 });+ compacted.notEvent("session.failed");++ const reopened = await session.send(+ `Earlier context was compacted. The \`${TOOL_NAME}\` image is saved at ` ++ `${stagedImagePath(output.imageBase64)}. Open that file with \`read_file\` before ` ++ "answering, even if you remember the colors, then reply with only the stripe colors " ++ "left to right, comma-separated.",+ );+ reopened.expectOk();+ reopened.noFailedActions();+ reopened.calledTool("read_file", { count: 1 });+ reopened.notCalledTool(TOOL_NAME);+ t.eventsSatisfy("the reopened image answers the follow-up", (events) => {+ const answer = assistantAnswers(reopened.events).at(-1);+ return answer !== undefined && namesColorsInOrder(events, answer);+ }).soft();+ },+});++/** Where eve stages a tool-result file: its content hash and filename. */+function stagedImagePath(base64: string): string {+ const sha = createHash("sha256").update(Buffer.from(base64, "base64")).digest("hex");+ return `/workspace/.eve/attachments/${sha.slice(0, 16)}/stripes.png`;+}diff --git a/e2e/fixtures/agent-tools/evals/static-tools/render-stripes-shared.ts b/e2e/fixtures/agent-tools/evals/static-tools/render-stripes-shared.tsnew file mode 100644index 000000000..f2d2fb9e7--- /dev/null+++ b/e2e/fixtures/agent-tools/evals/static-tools/render-stripes-shared.ts@@ -0,0 +1,45 @@+import type { MessageStreamEvent } from "eve/client";++export const TOOL_NAME = "render-stripes";++export function isRenderStripesOutput(value: unknown): boolean {+ if (typeof value !== "object" || value === null) return false;+ const output = value as { readonly colors?: unknown; readonly imageBase64?: unknown };+ return (+ Array.isArray(output.colors) &&+ output.colors.length > 0 &&+ output.colors.every((color) => typeof color === "string") &&+ typeof output.imageBase64 === "string" &&+ output.imageBase64.startsWith("iVBOR") // PNG magic bytes, base64-encoded+ );+}++export function renderedColors(events: readonly MessageStreamEvent[]): readonly string[] {+ for (const event of events) {+ if (event.type !== "action.result" || event.data.result.kind !== "tool-result") continue;+ if (event.data.result.toolName !== TOOL_NAME) continue;+ const output = event.data.result.output as { readonly colors?: unknown };+ if (Array.isArray(output?.colors) && output.colors.every((c) => typeof c === "string")) {+ return output.colors as string[];+ }+ }+ return [];+}++/** Final (non-tool-call) assistant messages, in turn order. */+export function assistantAnswers(events: readonly MessageStreamEvent[]): readonly string[] {+ return events.flatMap((event) =>+ event.type === "message.completed" &&+ event.data.finishReason !== "tool-calls" &&+ event.data.message.trim().length > 0+ ? [event.data.message]+ : [],+ );+}++export function namesColorsInOrder(events: readonly MessageStreamEvent[], answer: string): boolean {+ const colors = renderedColors(events);+ if (colors.length === 0) return false;+ const pattern = new RegExp(colors.map((color) => `\\b${color}\\b`).join("[\\s\\S]*"), "iu");+ return pattern.test(answer);+}diff --git a/e2e/fixtures/agent-tools/evals/static-tools/to-model-output-content-parts.eval.ts b/e2e/fixtures/agent-tools/evals/static-tools/to-model-output-content-parts.eval.tsindex 8d05dcd9e..d72fb1eed 100644--- a/e2e/fixtures/agent-tools/evals/static-tools/to-model-output-content-parts.eval.ts+++ b/e2e/fixtures/agent-tools/evals/static-tools/to-model-output-content-parts.eval.ts@@ -1,15 +1,20 @@-import type { MessageStreamEvent } from "eve/client";import { defineEval } from "eve/evals";-const TOOL_NAME = "render-stripes";+import {+ assistantAnswers,+ isRenderStripesOutput,+ namesColorsInOrder,+ TOOL_NAME,+} from "./render-stripes-shared";// The stripe colors are randomized per run, so a blind model cannot pass by// guessing; the eval stays self-contained by validating the reply against// the answer key the tool records on action.result. The pixels reach the// model exclusively through `toModelOutput` content parts. Color recognition-// remains tracked rather than gated because live vision quality varies.+// remains tracked rather than gated because live vision quality varies; the+// fixture's mock model decodes the pixels and fails the turn when they are+// missing, so world suites gate on the image reaching every call.export default defineEval({- tags: ["real-model"],description: "Static tools smoke: toModelOutput content parts deliver an image to the model.",async test(t) {const { session } = await t.send(@@ -25,8 +30,8 @@ export default defineEval({return answer !== undefined && namesColorsInOrder(events, answer);}).soft();- // The content part is baked into persisted history, so a follow-up turn- // must answer from replay without re-running the tool.+ // History keeps a sandbox ref to the image and every model call hydrates+ // it, so a follow-up turn answers from replay without re-running the tool.await session.send("Without calling any tool, repeat the stripe colors left to right, comma-separated.",);@@ -39,45 +44,3 @@ export default defineEval({}).soft();},});--function isRenderStripesOutput(value: unknown): boolean {- if (typeof value !== "object" || value === null) return false;- const output = value as { readonly colors?: unknown; readonly imageBase64?: unknown };- return (- Array.isArray(output.colors) &&- output.colors.length > 0 &&- output.colors.every((color) => typeof color === "string") &&- typeof output.imageBase64 === "string" &&- output.imageBase64.startsWith("iVBOR") // PNG magic bytes, base64-encoded- );-}--function renderedColors(events: readonly MessageStreamEvent[]): readonly string[] {- for (const event of events) {- if (event.type !== "action.result" || event.data.result.kind !== "tool-result") continue;- if (event.data.result.toolName !== TOOL_NAME) continue;- const output = event.data.result.output as { readonly colors?: unknown };- if (Array.isArray(output?.colors) && output.colors.every((c) => typeof c === "string")) {- return output.colors as string[];- }- }- return [];-}--/** Final (non-tool-call) assistant messages, in turn order. */-function assistantAnswers(events: readonly MessageStreamEvent[]): readonly string[] {- return events.flatMap((event) =>- event.type === "message.completed" &&- event.data.finishReason !== "tool-calls" &&- event.data.message.trim().length > 0- ? [event.data.message]- : [],- );-}--function namesColorsInOrder(events: readonly MessageStreamEvent[], answer: string): boolean {- const colors = renderedColors(events);- if (colors.length === 0) return false;- const pattern = new RegExp(colors.map((color) => `\\b${color}\\b`).join("[\\s\\S]*"), "iu");- return pattern.test(answer);-}diff --git a/packages/eve/src/execution/sandbox/read-file.test.ts b/packages/eve/src/execution/sandbox/read-file.test.tsnew file mode 100644index 000000000..b2f071973--- /dev/null+++ b/packages/eve/src/execution/sandbox/read-file.test.ts@@ -0,0 +1,110 @@+import type { ModelMessage } from "ai";+import { describe, expect, it } from "vitest";++import { ContextContainer, contextStorage } from "#context/container.js";+import { SandboxKey } from "#context/keys.js";+import { executeReadFileOnSandbox } from "#execution/sandbox/read-file.js";+import { stageToolResultMedia } from "#harness/attachment-staging.js";+import { decodeSandboxRef } from "#internal/attachments/sandbox-refs.js";+import { pngBytes } from "#internal/testing/media-fixtures.js";+import { mockSandbox } from "#internal/testing/mocks/mock-sandbox.js";+import { readFile } from "#tools/provided/read-file.js";++function sandboxContext(sandbox: ReturnType<typeof mockSandbox>): ContextContainer {+ const ctx = new ContextContainer();+ ctx.set(SandboxKey, sandbox.access);+ return ctx;+}++function toolResultMessage(file: Record<string, unknown>): ModelMessage {+ return {+ content: [+ {+ output: { type: "content", value: [file] },+ toolCallId: "call-1",+ toolName: "browser_screenshot",+ type: "tool-result",+ },+ ],+ role: "tool",+ } as ModelMessage;+}++function stagedPath(message: ModelMessage | undefined): string | undefined {+ const part = message?.role === "tool" ? message.content[0] : undefined;+ const file =+ part?.type === "tool-result" && part.output.type === "content"+ ? part.output.value[0]+ : undefined;+ return file?.type === "file" && file.data.type === "url"+ ? decodeSandboxRef(file.data.url).path+ : undefined;+}++describe("read_file images", () => {+ it("keeps image bytes out of the raw output and shows them to the model", async () => {+ const png = pngBytes(32, 32, 64);+ const sandbox = mockSandbox();+ await sandbox.session.writeBinaryFile({ content: png, path: "/workspace/chart.png" });++ const output = await executeReadFileOnSandbox(sandbox.session, {+ filePath: "/workspace/chart.png",+ });+ const modelOutput = await contextStorage.run(sandboxContext(sandbox), async () =>+ readFile.toModelOutput?.(output),+ );++ // The raw output feeds action.result, so it carries metadata only.+ expect(output.image).toEqual({+ height: 32,+ mediaType: "image/png",+ size: png.byteLength,+ width: 32,+ });+ expect(modelOutput).toEqual({+ type: "content",+ value: [+ { text: `Image /workspace/chart.png (image/png, ${png.byteLength} bytes).`, type: "text" },+ {+ data: { data: png.toString("base64"), type: "data" },+ filename: "chart.png",+ mediaType: "image/png",+ type: "file",+ },+ ],+ });+ });++ it("reopens a nameless tool image from the path its compaction stub names", async () => {+ const png = pngBytes(16, 16);+ const sandbox = mockSandbox();++ // MCP image blocks reach eve as file parts without a filename.+ const [message] = await contextStorage.run(sandboxContext(sandbox), () =>+ stageToolResultMedia([+ toolResultMessage({+ data: { data: png.toString("base64"), type: "data" },+ mediaType: "image/png",+ type: "file",+ }),+ ]),+ );+ const path = stagedPath(message) ?? "";++ expect(path).toMatch(/^\/workspace\/\.eve\/attachments\/[0-9a-f]{16}\/file-[0-9a-f]{16}\.png$/);+ const output = await executeReadFileOnSandbox(sandbox.session, { filePath: path });+ expect(output.image).toMatchObject({ mediaType: "image/png" });+ });++ it("rejects images too large to show inline", async () => {+ const sandbox = mockSandbox();+ await sandbox.session.writeBinaryFile({+ content: pngBytes(4000, 4000, 3 * 1024 * 1024),+ path: "/workspace/photo.png",+ });++ await expect(+ executeReadFileOnSandbox(sandbox.session, { filePath: "/workspace/photo.png" }),+ ).rejects.toThrow("read_file shows images up to 3 MiB");+ });+});diff --git a/packages/eve/src/execution/sandbox/read-file.ts b/packages/eve/src/execution/sandbox/read-file.tsindex 92bcac4c0..26f06d49b 100644--- a/packages/eve/src/execution/sandbox/read-file.ts+++ b/packages/eve/src/execution/sandbox/read-file.ts@@ -6,6 +6,7 @@ import {setReadFileStamp,} from "#execution/tools/file-state.js";import { resolveAbsoluteFilePath } from "#execution/sandbox/require-sandbox.js";+import { readMediaMetadata, type MediaMetadata } from "#internal/attachments/media-metadata.js";import type { SandboxSession } from "#shared/sandbox-session.js";import { capLineLength, MAX_OUTPUT_BYTES } from "#execution/sandbox/truncate-output.js";@@ -16,6 +17,16 @@ import { capLineLength, MAX_OUTPUT_BYTES } from "#execution/sandbox/truncate-outconst DEFAULT_OFFSET = 1;const DEFAULT_LIMIT = 2000;+// Matches the inline cap for inbound image attachments.+const MAX_IMAGE_BYTES = 3 * 1024 * 1024;+const IMAGE_MEDIA_TYPES: Readonly<Record<string, string>> = {+ gif: "image/gif",+ jpeg: "image/jpeg",+ jpg: "image/jpeg",+ png: "image/png",+ webp: "image/webp",+};+// ---------------------------------------------------------------------------// Input / result shapes// ---------------------------------------------------------------------------@@ -34,6 +45,11 @@ export interface ReadFileInput {*/export interface ReadFileResult {readonly content: string;+ /**+ * Set when the file is a PNG, JPEG, GIF, or WebP image the model sees as+ * an image. Carries no bytes, so `action.result` stays small.+ */+ readonly image?: MediaMetadata;readonly nextOffset?: number;readonly path: string;readonly totalLines: number;@@ -47,7 +63,8 @@ export interface ReadFileResult {/*** Reads one text file from the sandbox, applies output shaping* (offset, limit, line numbering, truncation), and persists a full-file- * stamp into durable read-file state for stale-write detection.+ * stamp into durable read-file state for stale-write detection. Image+ * files return their bytes for the model to view instead.** Used by the framework `read_file` tool and authored wrappers around its* exported definition.@@ -61,6 +78,11 @@ export async function executeReadFileOnSandbox(const resolvedPath = await resolveAbsoluteFilePath(sandbox, filePath);const normalizedPath = normalizeModelPath(resolvedPath);+ const imageMediaType = IMAGE_MEDIA_TYPES[normalizedPath.split(".").pop()?.toLowerCase() ?? ""];+ if (imageMediaType !== undefined) {+ return await readImageFile(sandbox, resolvedPath, normalizedPath, imageMediaType);+ }+// ── Validate offset / limit ─────────────────────────────────────────const effectiveOffset = offset ?? DEFAULT_OFFSET;const effectiveLimit = limit ?? DEFAULT_LIMIT;@@ -179,3 +201,30 @@ export async function executeReadFileOnSandbox(truncated: false,};}++async function readImageFile(+ sandbox: SandboxSession,+ resolvedPath: string,+ path: string,+ mediaType: string,+): Promise<ReadFileResult> {+ const bytes = await sandbox.readBinaryFile({ path: resolvedPath });+ if (bytes === null) {+ throw new Error(+ `File not found: ${path}. Verify the path exists and is accessible in the sandbox.`,+ );+ }+ if (bytes.byteLength > MAX_IMAGE_BYTES) {+ throw new Error(+ `Image "${path}" is ${bytes.byteLength} bytes; read_file shows images up to 3 MiB. ` ++ "Resize or crop it in the sandbox first.",+ );+ }+ return {+ content: `Image ${path} (${mediaType}, ${bytes.byteLength} bytes).`,+ image: readMediaMetadata(bytes, mediaType),+ path,+ totalLines: 0,+ truncated: false,+ };+}diff --git a/packages/eve/src/harness/attachment-staging.integration.test.ts b/packages/eve/src/harness/attachment-staging.integration.test.tsindex 369a32144..7562901dc 100644--- a/packages/eve/src/harness/attachment-staging.integration.test.ts+++ b/packages/eve/src/harness/attachment-staging.integration.test.ts@@ -151,7 +151,7 @@ describe("stageAttachmentsToSandbox (integration)", () => {expect(staged).toHaveLength(2);const filePart = staged[1] as FilePart;expect(filePart.filename).toMatch(- new RegExp(`^${ATTACHMENTS_ROOT}/[0-9a-f]{16}/report\\.csv$`),+ new RegExp(`^${ATTACHMENTS_ROOT.replaceAll(".", "\\.")}/[0-9a-f]{16}/report\\.csv$`),);// `data` is replaced with an eve-sandbox: ref — the bytes live in// the sandbox and are rehydrated at the model call site. The@@ -453,7 +453,7 @@ describe("hydrateSandboxAttachments (integration)", () => {expect((hydratedFilePart.data as Buffer).equals(smallImageBytes)).toBe(true);expect(hydratedFilePart.mediaType).toBe("image/png");expect(hydratedFilePart.filename).toMatch(- /^\/workspace\/attachments\/[0-9a-f]{16}\/logo\.png$/,+ /^\/workspace\/\.eve\/attachments\/[0-9a-f]{16}\/logo\.png$/,);});diff --git a/packages/eve/src/harness/attachment-staging.test.ts b/packages/eve/src/harness/attachment-staging.test.tsindex f0bbb77d6..645a75e17 100644--- a/packages/eve/src/harness/attachment-staging.test.ts+++ b/packages/eve/src/harness/attachment-staging.test.ts@@ -1,14 +1,18 @@-import type { FilePart, UserContent } from "ai";+import type { FilePart, ModelMessage, UserContent } from "ai";import { describe, expect, it } from "vitest";import type { ChannelAdapterContext } from "#channel/adapter.js";import { ContextContainer } from "#context/container.js";import { decodeSandboxRef, isSandboxRefUrl } from "#internal/attachments/sandbox-refs.js";import { mockSandbox } from "#internal/testing/mocks/mock-sandbox.js";-import { ATTACHMENTS_ROOT, stageAttachmentsForAdapter } from "#harness/attachment-staging.js";+import {+ ATTACHMENTS_ROOT,+ moveToolResultFilesToUserMessages,+ stageAttachmentsForAdapter,+} from "#harness/attachment-staging.js";const UTF8 = new TextEncoder();-const ATTACHMENTS_PATH_PATTERN = /^\/workspace\/attachments\/[0-9a-f]{16}\//;+const ATTACHMENTS_PATH_PATTERN = /^\/workspace\/\.eve\/attachments\/[0-9a-f]{16}\//;/*** Minimal {@link ChannelAdapterContext} for tests that never encounter@@ -75,12 +79,14 @@ describe("stageAttachmentsForAdapter", () => {const ref = decodeSandboxRef(filePart.data as URL);expect(ref.mediaType).toBe("text/csv");expect(ref.size).toBe(data.byteLength);- expect(ref.path).toMatch(/^\/workspace\/attachments\/[0-9a-f]{16}\/quarterly\.csv$/);+ expect(ref.path).toMatch(/^\/workspace\/\.eve\/attachments\/[0-9a-f]{16}\/quarterly\.csv$/);expect(filePart.filename).toBe(ref.path);expect(sandbox.writes).toHaveLength(1);const write = sandbox.writes[0];- expect(write?.path).toMatch(new RegExp(`^${ATTACHMENTS_ROOT}/[0-9a-f]{16}/quarterly\\.csv$`));+ expect(write?.path).toMatch(+ new RegExp(`^${ATTACHMENTS_ROOT.replaceAll(".", "\\.")}/[0-9a-f]{16}/quarterly\\.csv$`),+ );expect(write?.content).toEqual(data);});@@ -364,3 +370,59 @@ describe("stageAttachmentsForAdapter", () => {expect(filePart.filename).toMatch(ATTACHMENTS_PATH_PATTERN);});});++describe("moveToolResultFilesToUserMessages", () => {+ it("moves tool-result files into a user message after the tool results", () => {+ const file = {+ data: { data: "iVBORw0KGgo=", type: "data" as const },+ filename: "chart.png",+ mediaType: "image/png",+ type: "file" as const,+ };+ const messages: ModelMessage[] = [+ {+ content: [{ input: {}, toolCallId: "call-1", toolName: "render", type: "tool-call" }],+ role: "assistant",+ },+ {+ content: [+ {+ output: { type: "content", value: [{ text: "Rendered.", type: "text" }, file] },+ toolCallId: "call-1",+ toolName: "render",+ type: "tool-result",+ },+ ],+ role: "tool",+ },+ { content: "It is a chart.", role: "assistant" },+ ];++ const moved = moveToolResultFilesToUserMessages(messages);++ expect(moved.map((message) => message.role)).toEqual([+ "assistant",+ "tool",+ "user",+ "assistant",+ ]);+ expect(moved[1]?.content).toEqual([+ {+ output: {+ type: "content",+ value: [+ { text: "Rendered.", type: "text" },+ { text: "Attached file chart.png (image/png) follows this tool result.", type: "text" },+ ],+ },+ toolCallId: "call-1",+ toolName: "render",+ type: "tool-result",+ },+ ]);+ expect(moved[2]?.content).toEqual([+ { text: "Files returned by the preceding tool results:", type: "text" },+ file,+ ]);+ });+});diff --git a/packages/eve/src/harness/attachment-staging.ts b/packages/eve/src/harness/attachment-staging.tsindex b41db691f..3e01e83ff 100644--- a/packages/eve/src/harness/attachment-staging.ts+++ b/packages/eve/src/harness/attachment-staging.ts@@ -1,32 +1,36 @@import { createHash } from "node:crypto";-import { basename } from "node:path";-import type { FilePart, ModelMessage, TextPart, UserContent } from "ai";+import { basename, extname } from "node:path";+import type { FilePart, ModelMessage, TextPart, ToolResultPart, UserContent } from "ai";import { buildAdapterContext } from "#channel/adapter-context.js";import type { ChannelAdapterContext, FetchFileResult } from "#channel/adapter.js";import { getAdapterKind } from "#channel/adapter.js";import { buildSessionHandle } from "#channel/session.js";-import { loadContext } from "#context/container.js";+import { contextStorage, loadContext } from "#context/container.js";import { SandboxKey } from "#context/keys.js";+import { createFrameworkUserMessage } from "#harness/messages.js";import { ChannelKey } from "#runtime/sessions/runtime-context-keys.js";import { fileDataToBytes } from "#internal/attachments/data.js";import { EveAttachmentError } from "#internal/attachments/errors.js";import { createLogger } from "#internal/logging.js";+import { readMediaMetadata } from "#internal/attachments/media-metadata.js";import { deserializeUrlFilePart, isSerializedUrlFilePart } from "#internal/attachments/url-refs.js";import {decodeSandboxRef,encodeSandboxRef,+ inlinesSandboxRefAsBytes,isSandboxRefUrl,type SandboxRef,} from "#internal/attachments/sandbox-refs.js";import type { SandboxSession } from "#public/definitions/sandbox.js";/**- * Sandbox directory where inbound file attachments are staged before the- * model call. Authored canonical path — {@link SandboxSession.writeFile}- * translates to the backend-native location.+ * Sandbox directory where eve stages message attachments and tool-result+ * files. It sits under eve's own dot-directory so it never collides with an+ * agent's `/workspace/attachments`. Authored canonical path —+ * {@link SandboxSession.writeFile} translates to the backend-native location.*/-export const ATTACHMENTS_ROOT = "/workspace/attachments";+export const ATTACHMENTS_ROOT = "/workspace/.eve/attachments";const log = createLogger("harness.attachment-staging");@@ -35,22 +39,19 @@ const SHA_PREFIX_LENGTH = 16;const DEFAULT_MEDIA_TYPE = "application/octet-stream";-/**- * Upper bound, in bytes, on image payloads that hydrate as inline bytes- * on the model call. Larger images are substituted for a text reference- * pointing at the staged sandbox path so the agent can read the file- * through its normal filesystem tools (`read_file`, `bash`, etc.).- */-const HYDRATE_IMAGE_INLINE_MAX_BYTES = 3 * 1024 * 1024;--/**- * Upper bound, in bytes, on PDF payloads that hydrate as inline bytes.- * Matches provider-side caps for native document understanding.- */-const HYDRATE_PDF_INLINE_MAX_BYTES = 20 * 1024 * 1024;+// A staged name keeps or gains the extension its media type implies, so a+// nameless file (an MCP image, say) can still be opened by path later.+const MEDIA_TYPE_EXTENSIONS: Readonly<Record<string, string>> = {+ "application/pdf": ".pdf",+ "image/gif": ".gif",+ "image/jpeg": ".jpg",+ "image/png": ".png",+ "image/webp": ".webp",+};-const PDF_MEDIA_TYPE = "application/pdf";-const IMAGE_MEDIA_TYPE_PREFIX = "image/";+type ToolResultOutput = ToolResultPart["output"];+type ToolOutputContentPart = Extract<ToolResultOutput, { type: "content" }>["value"][number];+type ToolOutputFilePart = Extract<ToolOutputContentPart, { type: "file" }>;/*** Writes inbound `FilePart` bytes into the sandbox and rewrites each@@ -126,12 +127,49 @@ export async function stageAttachmentsToSandbox(return stageAttachmentsForAdapter(message, sandbox, adapterCtx);}+/**+ * Moves inline file payloads in `content` tool outputs into the sandbox and+ * leaves `eve-sandbox:` refs in their place, so durable history never+ * carries tool-result bytes. {@link hydrateSandboxAttachments} restores the+ * bytes on every model call. Without an active sandbox, messages pass+ * through unchanged.+ */+export async function stageToolResultMedia<T extends ModelMessage>(+ messages: readonly T[],+): Promise<T[]> {+ if (!messages.some(hasInlineToolResultFile)) {+ return [...messages];+ }+ const sandbox = await contextStorage.getStore()?.get(SandboxKey)?.get();+ if (sandbox === undefined || sandbox === null) {+ return [...messages];+ }+ return Promise.all(+ messages.map(async (message) => {+ if (message.role !== "tool" || !hasInlineToolResultFile(message)) {+ return message;+ }+ const content = await Promise.all(+ message.content.map(async (part) =>+ part.type === "tool-result"+ ? { ...part, output: await stageToolOutputFiles(part.output, sandbox) }+ : part,+ ),+ );+ return { ...message, content };+ }),+ );+}+/*** Hydrates `eve-sandbox:` file refs for a single model call.*- * Small images and PDFs are inlined as bytes; larger or unsupported files- * become text references to their sandbox path. The returned messages must- * not be written back to session history, which stays ref-only across steps.+ * Tool-result refs always hydrate as bytes: the tool chose to show them to+ * the model. Inbound attachments inline small images and PDFs; larger or+ * unsupported files become text references to their sandbox path. Every+ * decision is pure in the ref, so each message renders identically on every+ * call and the provider's prompt cache stays valid. The returned messages+ * must not be written back to session history, which stays ref-only.*/export async function hydrateSandboxAttachments(messages: readonly ModelMessage[],@@ -193,10 +231,54 @@ function messageContainsSandboxRef(message: ModelMessage): boolean {if (isSandboxRefFilePart(part)) {return true;}+ if (part.type === "tool-result" && contentOutputParts(part.output).some(isToolOutputRefFile)) {+ return true;+ }}return false;}+function contentOutputParts(output: ToolResultOutput): readonly ToolOutputContentPart[] {+ return output.type === "content" ? output.value : [];+}++function hasInlineToolResultFile(message: ModelMessage): boolean {+ return (+ message.role === "tool" &&+ message.content.some(+ (part) =>+ part.type === "tool-result" && contentOutputParts(part.output).some(isInlineToolOutputFile),+ )+ );+}++function isInlineToolOutputFile(part: ToolOutputContentPart): part is ToolOutputFilePart {+ return part.type === "file" && part.data.type === "data";+}++function isToolOutputRefFile(part: ToolOutputContentPart): part is ToolOutputFilePart {+ return part.type === "file" && part.data.type === "url" && isSandboxRefUrl(part.data.url);+}++async function stageToolOutputFiles(+ output: ToolResultOutput,+ sandbox: SandboxSession,+): Promise<ToolResultOutput> {+ if (output.type !== "content" || !output.value.some(isInlineToolOutputFile)) {+ return output;+ }+ const value = await Promise.all(+ output.value.map(async (part) => {+ if (!isInlineToolOutputFile(part) || part.data.type !== "data") return part;+ const bytes = await fileDataToBytes(part.data.data);+ if (bytes === null) return part;+ const ref = await writeSandboxRef(bytes, part.mediaType, part.filename, sandbox);+ return { ...part, data: { type: "url" as const, url: encodeSandboxRef(ref) } };+ }),+ );+ return { ...output, value };+}+/*** Narrows a `UserContent` / `ModelMessage.content` element to a* {@link FilePart} whose `data` is an `eve-sandbox:` ref URL.@@ -221,55 +303,117 @@ async function hydrateMessageContent(content: unknown, sandbox: SandboxSession):}return Promise.all(content.map(async (part) => {- if (!isSandboxRefFilePart(part)) {- return part;+ if (isSandboxRefFilePart(part)) {+ const ref = decodeSandboxRef(part.data as URL);+ if (!inlinesSandboxRefAsBytes(ref)) {+ return renderSandboxRefAsTextPart(ref);+ }+ const bytes = await readSandboxRefBytes(ref, sandbox);+ return bytes === null+ ? renderMissingSandboxRef(ref)+ : { ...part, data: bytes, mediaType: ref.mediaType };}- const filePart = part;- const ref = decodeSandboxRef(filePart.data as URL);- if (!shouldInlineSandboxRefAsBytes(ref)) {- return renderSandboxRefAsTextPart(ref);+ if ((part as { type?: unknown }).type === "tool-result") {+ const toolResult = part as ToolResultPart;+ return { ...toolResult, output: await hydrateToolOutput(toolResult.output, sandbox) };}- const bytes = await sandbox.readBinaryFile({ path: ref.path });- if (bytes === null) {- // #325: sandbox snapshots can change during a session lifecycle- // as they are deployment bounded.- log.warn(- "sandbox-ref attachment bytes missing on hydration — degrading to text reference",- {- mediaType: ref.mediaType,- path: ref.path,- size: ref.size,- },- );- return {- text: `FileNotFound: Current snapshot may be newer and does not contain ${ref.path}.`,- type: "text",- } satisfies TextPart;- }- return { ...filePart, data: bytes, mediaType: ref.mediaType };+ return part;+ }),+ );+}++async function hydrateToolOutput(+ output: ToolResultOutput,+ sandbox: SandboxSession,+): Promise<ToolResultOutput> {+ if (output.type !== "content" || !output.value.some(isToolOutputRefFile)) {+ return output;+ }+ const value = await Promise.all(+ output.value.map(async (part): Promise<ToolOutputContentPart> => {+ if (!isToolOutputRefFile(part) || part.data.type !== "url") return part;+ const ref = decodeSandboxRef(part.data.url);+ const bytes = await readSandboxRefBytes(ref, sandbox);+ if (bytes === null) return renderMissingSandboxRef(ref);+ return {+ ...part,+ data: { data: Buffer.from(bytes).toString("base64"), type: "data" },+ };}),);+ return { ...output, value };+}++async function readSandboxRefBytes(+ ref: SandboxRef,+ sandbox: SandboxSession,+): Promise<Uint8Array | null> {+ const bytes = await sandbox.readBinaryFile({ path: ref.path });+ if (bytes === null) {+ // #325: sandbox snapshots can change during a session lifecycle+ // as they are deployment bounded.+ log.warn("sandbox-ref attachment bytes missing on hydration — degrading to text reference", {+ mediaType: ref.mediaType,+ path: ref.path,+ size: ref.size,+ });+ }+ return bytes;+}++function renderMissingSandboxRef(ref: SandboxRef): TextPart {+ return {+ text: `FileNotFound: Current snapshot may be newer and does not contain ${ref.path}.`,+ type: "text",+ };}/**- * Decides whether a sandbox-resident attachment should flow to the- * model as inline bytes at the hydration step, or be substituted for a- * text reference pointing at the staged sandbox path.- *- * Keep this decision narrow: only the shapes every major provider- * supports natively qualify for byte inlining. Everything else — raw- * documents, archives, source code, oversized images/PDFs — reaches- * the model as a text reference so the agent's filesystem tools do- * the reading.+ * Chat Completions-style providers (`*.chat`) serialize `content` tool+ * outputs as JSON text, so a file would reach the model as base64+ * characters. Moves the files of each run of tool messages into one user+ * message right after it, leaving a text stub in the tool result. The move+ * is deterministic, so the prompt prefix stays cache-stable.*/-function shouldInlineSandboxRefAsBytes(ref: SandboxRef): boolean {- if (ref.mediaType.startsWith(IMAGE_MEDIA_TYPE_PREFIX)) {- return ref.size <= HYDRATE_IMAGE_INLINE_MAX_BYTES;- }- if (ref.mediaType === PDF_MEDIA_TYPE) {- return ref.size <= HYDRATE_PDF_INLINE_MAX_BYTES;+export function moveToolResultFilesToUserMessages(+ messages: readonly ModelMessage[],+): ModelMessage[] {+ const moved: ModelMessage[] = [];+ let files: FilePart[] = [];+ for (const message of messages) {+ if (message.role !== "tool" && files.length > 0) {+ moved.push(createReturnedFilesMessage(files));+ files = [];+ }+ if (message.role !== "tool") {+ moved.push(message);+ continue;+ }+ const content = message.content.map((part) => {+ if (part.type !== "tool-result" || part.output.type !== "content") return part;+ const value = part.output.value.map((entry): ToolOutputContentPart => {+ if (entry.type !== "file") return entry;+ const file = { data: entry.data, mediaType: entry.mediaType, type: "file" } as FilePart;+ if (entry.filename !== undefined) file.filename = entry.filename;+ files.push(file);+ return {+ text: `Attached file ${entry.filename ?? "file"} (${entry.mediaType}) follows this tool result.`,+ type: "text",+ };+ });+ return { ...part, output: { ...part.output, value } };+ });+ moved.push({ ...message, content });}- return false;+ if (files.length > 0) moved.push(createReturnedFilesMessage(files));+ return moved;+}++function createReturnedFilesMessage(files: readonly FilePart[]): ModelMessage {+ return createFrameworkUserMessage("context.state", [+ { text: "Files returned by the preceding tool results:", type: "text" },+ ...files,+ ]);}/**@@ -332,20 +476,27 @@ async function stageResolvedBytes(resolved: FetchFileResult,sandbox: SandboxSession,): Promise<FilePart> {- const bytes = resolved.bytes;- const sha = sha256Prefix(bytes);const mediaType = resolved.mediaType ?? part.mediaType ?? DEFAULT_MEDIA_TYPE;- const name = safeFilename(resolved.filename ?? part.filename, sha);- const authored = `${ATTACHMENTS_ROOT}/${sha}/${name}`;+ const ref = await writeSandboxRef(+ resolved.bytes,+ mediaType,+ resolved.filename ?? part.filename,+ sandbox,+ );+ return { ...part, data: encodeSandboxRef(ref), filename: ref.path, mediaType };+}+/** Writes content-addressed bytes under {@link ATTACHMENTS_ROOT} and describes them as a ref. */+async function writeSandboxRef(+ bytes: Buffer,+ mediaType: string,+ filename: string | undefined,+ sandbox: SandboxSession,+): Promise<SandboxRef> {+ const sha = sha256Prefix(bytes);+ const authored = `${ATTACHMENTS_ROOT}/${sha}/${safeFilename(filename, sha, mediaType)}`;await sandbox.writeBinaryFile({ content: bytes, path: authored });- const resolvedPath = sandbox.resolvePath(authored);- return {- ...part,- data: encodeSandboxRef({ mediaType, path: resolvedPath, size: bytes.byteLength }),- filename: resolvedPath,- mediaType,- };+ return { ...readMediaMetadata(bytes, mediaType), path: sandbox.resolvePath(authored) };}async function tryFetchFile(@@ -402,10 +553,9 @@ function sha256Prefix(bytes: Buffer): string {return createHash("sha256").update(bytes).digest("hex").slice(0, SHA_PREFIX_LENGTH);}-function safeFilename(provided: string | undefined, sha: string): string {- if (provided === undefined) {- return `file-${sha}`;- }- const base = basename(provided).replace(UNSAFE_FILENAME_CHARS, "_");- return base.length > 0 ? base : `file-${sha}`;+function safeFilename(provided: string | undefined, sha: string, mediaType: string): string {+ const base = provided === undefined ? "" : basename(provided).replace(UNSAFE_FILENAME_CHARS, "_");+ const name = base.length > 0 ? base : `file-${sha}`;+ const extension = MEDIA_TYPE_EXTENSIONS[mediaType];+ return extension === undefined || extname(name) !== "" ? name : `${name}${extension}`;}diff --git a/packages/eve/src/harness/compaction-prompt.ts b/packages/eve/src/harness/compaction-prompt.tsindex 1bd0339ef..1ea46fc88 100644--- a/packages/eve/src/harness/compaction-prompt.ts+++ b/packages/eve/src/harness/compaction-prompt.ts@@ -1,6 +1,7 @@import type { ModelMessage } from "ai";import { estimateTokens } from "#harness/token-estimate.js";+import { decodeSandboxRef, isSandboxRefUrl } from "#internal/attachments/sandbox-refs.js";export const COMPACTION_CHECKPOINT_MARKER = "Summary of our conversation so far:";@@ -245,14 +246,13 @@ export function stubContentOutputFileParts(output: unknown): unknown {if (part === null || typeof part !== "object") return part;const candidate = part as {readonly type?: unknown;- readonly filename?: unknown;readonly mediaType?: unknown;};if (candidate.type !== "file" && candidate.type !== "media") return part;changed = true;return {text: renderAttachedFileStub(- typeof candidate.filename === "string" ? candidate.filename : undefined,+ contentFileName(part),typeof candidate.mediaType === "string" ? candidate.mediaType : "unknown",),type: "text",@@ -276,13 +276,26 @@ function renderContentToolOutputPart(part: unknown, limit: number): string {// an inline payload the summarizer cannot read.if (candidate.type === "file" || candidate.type === "media") {return renderAttachedFileStub(- typeof candidate.filename === "string" ? candidate.filename : undefined,+ contentFileName(part),typeof candidate.mediaType === "string" ? candidate.mediaType : "unknown",);}return renderPayload(part, limit);}+/**+ * Names a content file part in its stub. A staged file names its sandbox+ * path, so the agent can open it again after compaction drops the bytes.+ */+function contentFileName(part: object): string | undefined {+ const { data, filename } = part as { readonly data?: unknown; readonly filename?: unknown };+ const tagged = data as { readonly type?: unknown; readonly url?: unknown } | undefined;+ if (tagged?.type === "url" && isSandboxRefUrl(tagged.url)) {+ return decodeSandboxRef(tagged.url).path;+ }+ return typeof filename === "string" ? filename : undefined;+}+function renderToolCall(part: { toolName: string; input?: unknown }, limit: number): string {const input = renderPayload(part.input, limit);return input ? `Called ${part.toolName} with ${input}` : `Called ${part.toolName}`;diff --git a/packages/eve/src/harness/compaction.test.ts b/packages/eve/src/harness/compaction.test.tsindex f1f803fc7..8e4759280 100644--- a/packages/eve/src/harness/compaction.test.ts+++ b/packages/eve/src/harness/compaction.test.ts@@ -11,6 +11,8 @@ import {import { createFrameworkUserMessage } from "#harness/messages.js";import { estimateTokens } from "#harness/token-estimate.js";import type { CompactionConfig } from "#harness/types.js";+import { encodeSandboxRef } from "#internal/attachments/sandbox-refs.js";+import { pngBytes } from "#internal/testing/media-fixtures.js";vi.mock("ai", () => ({generateText: vi.fn(),@@ -25,6 +27,23 @@ const config: CompactionConfig = {threshold: 100,};+function toolFileMessage(data: unknown, mediaType = "image/png"): ModelMessage {+ return {+ content: [+ {+ output: {+ type: "content",+ value: [{ data, mediaType, type: "file" }],+ } as never,+ toolCallId: "call-image",+ toolName: "render",+ type: "tool-result",+ },+ ],+ role: "tool",+ };+}+describe("estimateTokens", () => {it("estimates based on serialized character length", () => {const messages: ModelMessage[] = [{ content: "a".repeat(400), role: "user" }];@@ -121,9 +140,72 @@ describe("estimateTokens", () => {expect(estimateTokens(base)).toBeGreaterThan(0);expect(estimateTokens(withReasoning)).toBeGreaterThan(estimateTokens(base));});++ it("counts an inline image by its pixel patches instead of its base64 length", () => {+ const base64 = pngBytes(1000, 1000, 300_000).toString("base64");+ const estimate = estimateTokens([toolFileMessage({ data: base64, type: "data" })]);++ expect(estimate).toBeGreaterThan(1296);+ expect(estimate).toBeLessThan(1296 + 100);+ });++ it("counts a staged image from its ref metadata without the bytes", () => {+ const ref = encodeSandboxRef({+ height: 1080,+ mediaType: "image/png",+ path: "/workspace/.eve/attachments/abc/screen.png",+ size: 2_000_000,+ width: 1920,+ });+ const estimate = estimateTokens([toolFileMessage({ type: "url", url: ref })]);++ expect(estimate).toBeGreaterThan(2691);+ expect(estimate).toBeLessThan(2691 + 100);+ });++ it("counts an inbound attachment that renders as a path reference as text", () => {+ const message: ModelMessage = {+ content: [+ {+ data: encodeSandboxRef({+ mediaType: "text/csv",+ path: "/workspace/.eve/attachments/abc/report.csv",+ size: 3_000_000,+ }),+ mediaType: "text/csv",+ type: "file",+ },+ ],+ role: "user",+ };++ expect(estimateTokens([message])).toBeLessThan(100);+ });});describe("getInputTokenCount", () => {+ // #3799: two page scans returned by a vision tool counted as ~160k tokens+ // of base64 text and forced compaction against a 43,895-token prompt.+ it("does not count tool-result image bytes as text before provider usage arrives", () => {+ const page = pngBytes(1000, 1000, 240_000).toString("base64");+ const prompt: ModelMessage[] = [{ content: "Read both scanned pages.", role: "user" }];+ const messages = [+ ...prompt,+ toolFileMessage({ data: page, type: "data" }),+ toolFileMessage({ data: page, type: "data" }),+ ];++ const result = getInputTokenCount(messages, {+ lastKnownInputTokens: 43_895,+ lastKnownPromptMessageCount: prompt.length,+ recentWindowSize: 2,+ threshold: 200_000,+ });++ expect(result).toBeGreaterThan(43_895 + 2 * 1296);+ expect(result).toBeLessThan(43_895 + 2 * 1296 + 200);+ });+it("prefers the last known exact token count when available", () => {const messages: ModelMessage[] = [{ content: "a".repeat(400), role: "user" }];@@ -625,6 +707,52 @@ describe("compactMessages: tool-result cap heuristic", () => {});});+ it("stubs a staged tool file with its sandbox path so the agent can reopen it", async () => {+ const path = "/workspace/.eve/attachments/0123456789abcdef/chart.png";+ const messages: ModelMessage[] = [+ user("render the chart"),+ {+ content: [{ input: {}, toolCallId: "call-0", toolName: "render_chart", type: "tool-call" }],+ role: "assistant",+ },+ {+ content: [+ {+ output: {+ type: "content",+ value: [+ {+ data: {+ type: "url",+ url: encodeSandboxRef({ mediaType: "image/png", path, size: 4096 }),+ },+ filename: "chart.png",+ mediaType: "image/png",+ type: "file",+ },+ { text: "x".repeat(800), type: "text" },+ ],+ },+ toolCallId: "call-0",+ toolName: "render_chart",+ type: "tool-result",+ },+ ],+ role: "tool",+ },+ user("what does it show?"),+ ];++ const { result } = await compact(messages, { recentWindowSize: 1 });++ const cappedPart = Array.isArray(result[2]?.content) ? result[2].content[0] : undefined;+ const output = cappedPart?.type === "tool-result" ? cappedPart.output : undefined;+ expect(output).toMatchObject({+ type: "content",+ value: [{ text: `Attached file ${path} (image/png)`, type: "text" }, { type: "text" }],+ });+ });+it("falls back to the annotated text cap when a content output is oversized after stubbing", async () => {const longText = "finding line ".repeat(500);const messages: ModelMessage[] = [diff --git a/packages/eve/src/harness/token-estimate.ts b/packages/eve/src/harness/token-estimate.tsindex 94ab84b21..f37cd0bee 100644--- a/packages/eve/src/harness/token-estimate.ts+++ b/packages/eve/src/harness/token-estimate.ts@@ -1,11 +1,129 @@+import {+ estimateMediaTokens,+ readMediaMetadata,+ type MediaMetadata,+} from "#internal/attachments/media-metadata.js";+import {+ decodeSandboxRef,+ inlinesSandboxRefAsBytes,+ isSandboxRefUrl,+} from "#internal/attachments/sandbox-refs.js";++// Enough base64 to cover image headers, including a JPEG's EXIF block.+const HEADER_BASE64_CHARS = 96 * 1024;+const REMOTE_URL = /^https?:\/\//i;+/**- * Rough token estimate: serialized JSON length / 4. Good enough for- * deciding whether compaction is needed; the real token count comes back- * from the model each step via `CompactionConfig.lastKnownInputTokens`.+ * Rough token estimate: serialized JSON length / 4, with file parts counted+ * at what providers bill for media instead of their base64 length. Good+ * enough for deciding whether compaction is needed; the real token count+ * comes back from the model each step via+ * `CompactionConfig.lastKnownInputTokens`.** Accepts any JSON-serializable value so callers can apply the same heuristic* to whole message arrays or individual content parts on one consistent ruler.*/export function estimateTokens(value: unknown): number {- return JSON.stringify(value).length / 4;+ let mediaTokens = 0;+ const serialized = JSON.stringify(value, (_key, candidate: unknown) => {+ const media = estimateFilePartTokens(candidate);+ if (media === undefined) return candidate;+ mediaTokens += media;+ const { mediaType, type } = candidate as {+ readonly mediaType?: unknown;+ readonly type: unknown;+ };+ return { mediaType, type };+ });+ return (serialized?.length ?? 0) / 4 + mediaTokens;+}++/**+ * Media tokens for a message file part or a `content` tool-output file part,+ * or `undefined` when the value is not one or renders as plain text.+ */+function estimateFilePartTokens(value: unknown): number | undefined {+ if (value === null || typeof value !== "object") return undefined;+ const part = value as {+ readonly data?: unknown;+ readonly image?: unknown;+ readonly mediaType?: unknown;+ readonly type?: unknown;+ };+ if (part.type === "image" && part.image !== undefined) {+ return estimateFileDataTokens(+ part.image,+ typeof part.mediaType === "string" ? part.mediaType : "image/*",+ );+ }+ if ((part.type !== "file" && part.type !== "media") || typeof part.mediaType !== "string") {+ return undefined;+ }+ return estimateFileDataTokens(part.data, part.mediaType);+}++function estimateFileDataTokens(data: unknown, mediaType: string): number | undefined {+ if (isSandboxRefUrl(data)) {+ // Inbound attachments that hydrate as text references stay text.+ const ref = decodeSandboxRef(data);+ return inlinesSandboxRefAsBytes(ref) ? estimateMediaTokens(ref) : undefined;+ }+ if (data instanceof URL) return estimateRemoteTokens(mediaType);+ if (isTaggedFileData(data)) {+ switch (data.type) {+ case "data":+ return estimateInlineTokens(data.data, mediaType);+ case "url":+ return isSandboxRefUrl(data.url)+ ? estimateMediaTokens(decodeSandboxRef(data.url))+ : estimateRemoteTokens(mediaType);+ case "reference":+ return estimateRemoteTokens(mediaType);+ default:+ return undefined;+ }+ }+ return estimateInlineTokens(data, mediaType);+}++function isTaggedFileData(+ value: unknown,+): value is { readonly type: string; readonly data?: unknown; readonly url?: unknown } {+ return (+ value !== null &&+ typeof value === "object" &&+ !(value instanceof Uint8Array) &&+ !(value instanceof ArrayBuffer) &&+ typeof (value as { readonly type?: unknown }).type === "string"+ );+}++/** Providers fetch remote files themselves; only images have a known cost. */+function estimateRemoteTokens(mediaType: string): number | undefined {+ return mediaType.startsWith("image/") ? estimateMediaTokens({ mediaType, size: 0 }) : undefined;+}++function estimateInlineTokens(data: unknown, mediaType: string): number | undefined {+ if (typeof data === "string") {+ if (REMOTE_URL.test(data)) return estimateRemoteTokens(mediaType);+ const base64 = data.startsWith("data:") ? data.slice(data.indexOf(",") + 1) : data;+ const size = Math.floor((base64.length * 3) / 4);+ return estimateMediaTokens(withHeaderMetadata(base64Header(base64), mediaType, size));+ }+ if (data instanceof Uint8Array || data instanceof ArrayBuffer) {+ const bytes = data instanceof Uint8Array ? data : new Uint8Array(data);+ return estimateMediaTokens(withHeaderMetadata(bytes, mediaType, bytes.byteLength));+ }+ return undefined;+}++function base64Header(base64: string): Uint8Array {+ return Buffer.from(base64.slice(0, HEADER_BASE64_CHARS), "base64");+}++// Only image dimensions come from a header; page counts need the whole+// payload, so inline PDFs fall back to their size.+function withHeaderMetadata(header: Uint8Array, mediaType: string, size: number): MediaMetadata {+ if (!mediaType.startsWith("image/")) return { mediaType, size };+ return { ...readMediaMetadata(header, mediaType), size };}diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.tsindex 15838c793..2a78becfd 100644--- a/packages/eve/src/harness/tool-loop.test.ts+++ b/packages/eve/src/harness/tool-loop.test.ts@@ -31,6 +31,7 @@ import {import { invocationOwnerKey } from "#internal/invocation/metadata.js";import { decodeSandboxRef, isSandboxRefUrl } from "#internal/attachments/sandbox-refs.js";import { attachClientContext } from "#internal/client-context.js";+import { pngBytes } from "#internal/testing/media-fixtures.js";import { mockSandbox } from "#internal/testing/mocks/mock-sandbox.js";import type { UnstampedMessageStreamEvent } from "#protocol/message.js";import type { InstrumentationStepStartedEventInput } from "#public/instrumentation/index.js";@@ -10747,7 +10748,7 @@ describe("createToolLoopHarness", () => {const historyRef = decodeSandboxRef(historyFilePart.data as URL);expect(historyRef.mediaType).toBe("image/png");expect(historyRef.size).toBe(imageBytes.byteLength);- expect(historyRef.path).toMatch(/^\/workspace\/attachments\/[0-9a-f]{16}\/logo\.png$/);+ expect(historyRef.path).toMatch(/^\/workspace\/\.eve\/attachments\/[0-9a-f]{16}\/logo\.png$/);// --- Invariant 3: the mocked ToolLoopAgent.generate saw hydrated bytes.//@@ -10775,6 +10776,160 @@ describe("createToolLoopHarness", () => {expect(streamFilePart!.mediaType).toBe("image/png");});+ it("keeps tool-result files out of history and hydrates identical bytes on every later call", async () => {+ const base64 = pngBytes(64, 64, 2048).toString("base64");+ const file = {+ data: { data: base64, type: "data" as const },+ filename: "chart.png",+ mediaType: "image/png",+ type: "file" as const,+ };+ const toolCall = {+ input: {},+ toolCallId: "render-1",+ toolName: "render_chart",+ type: "tool-call" as const,+ };+ const toolResult = {+ output: { type: "content" as const, value: [file] },+ toolCallId: "render-1",+ toolName: "render_chart",+ type: "tool-result" as const,+ };+ const reply = (text: string) => ({+ finishReason: "stop",+ response: { messages: [{ content: text, role: "assistant" }] },+ text,+ toolCalls: [],+ toolResults: [],+ });+ setupMockAgentSequence([+ {+ finishReason: "tool-calls",+ response: {+ messages: [+ { content: [toolCall], role: "assistant" },+ { content: [toolResult], role: "tool" },+ ],+ },+ text: "",+ toolCalls: [toolCall],+ toolResults: [toolResult],+ },+ reply("A bar chart."),+ reply("Still a bar chart."),+ ]);+ const sandbox = mockSandbox({ id: "sbx_tool_media" });+ const ctx = new ContextContainer();+ ctx.set(SandboxKey, sandbox.access);+ const runStep = createToolLoopHarness(createTestConfig());++ const first = await contextStorage.run(ctx, () =>+ runStep(createTestSession(), { message: "Render the chart." }),+ );+ if (typeof first.next !== "function") throw new Error("Expected a tool continuation.");+ const next = first.next;+ const second = await contextStorage.run(ctx, () => next(first.session));+ const third = await contextStorage.run(ctx, () =>+ runStep(second.session, { message: "Describe it again." }),+ );++ expect(sandbox.writes).toHaveLength(1);+ expect(JSON.stringify(third.session.history)).not.toContain(base64);+ // The next step and the next turn both render the file exactly as the+ // tool returned it, so the provider sees a stable prompt prefix.+ const laterCalls = vi+ .mocked(ToolLoopAgent)+ .mock.results.slice(1)+ .map(+ (result) =>+ (result.value as { generate: ReturnType<typeof vi.fn> }).generate.mock+ .calls[0]?.[0] as {+ messages: ModelMessage[];+ },+ );+ expect(laterCalls).toHaveLength(2);+ for (const call of laterCalls) {+ expect(call.messages.find((message) => message.role === "tool")?.content).toEqual([+ toolResult,+ ]);+ }+ });++ it("keeps files a workflow tool projects for the model out of history", async () => {+ const { setPendingCoordinationBatch } = await import("#harness/coordination.js");+ const { toolOutput, toolOutputPart } = await import("#tools/model-output.js");+ setupMockAgent({+ finishReason: "stop",+ response: { messages: [{ content: "Captured.", role: "assistant" }] },+ text: "Captured.",+ toolCalls: [],+ toolResults: [],+ });+ const base64 = pngBytes(32, 32, 512).toString("base64");+ const tools: ToolLoopHarnessConfig["tools"] = new Map([+ [+ "screenshot",+ {+ description: "Capture a screenshot.",+ inputSchema: jsonSchema({ type: "object" }),+ name: "screenshot",+ toModelOutput: (output: unknown) =>+ toolOutput.content([+ toolOutputPart.file((output as { png: string }).png, { mediaType: "image/png" }),+ ]),+ workflowId: "workflow//./agent/tools/screenshot//execute",+ },+ ],+ ]);+ const parked = setPendingCoordinationBatch({+ event: { sequence: 0, stepIndex: 0, turnId: "turn_0" },+ responseMessages: [+ {+ content: [+ { input: {}, toolCallId: "shot-1", toolName: "screenshot", type: "tool-call" },+ ],+ role: "assistant",+ },+ ],+ session: createTestSession(),+ tasks: [+ {+ callId: "shot-1",+ entry: { entryPoint: "execute" },+ input: {},+ kind: "workflow-task",+ toolName: "screenshot",+ workflowId: "workflow//./agent/tools/screenshot//execute",+ },+ ],+ });+ const sandbox = mockSandbox({ id: "sbx_workflow_media" });+ const ctx = new ContextContainer();+ ctx.set(SandboxKey, sandbox.access);++ const result = await contextStorage.run(ctx, () =>+ createToolLoopHarness(createTestConfig(undefined, { tools }))(parked, {+ runtimeActionResults: [+ {+ callId: "shot-1",+ kind: "tool-result",+ output: { png: base64 },+ toolName: "screenshot",+ },+ ],+ }),+ );++ expect(sandbox.writes).toHaveLength(1);+ expect(JSON.stringify(result.session.history)).not.toContain(base64);+ const agent = vi.mocked(ToolLoopAgent).mock.results[0]?.value as+ | { generate: ReturnType<typeof vi.fn> }+ | undefined;+ const modelCall = agent?.generate.mock.calls[0]?.[0] as { messages: ModelMessage[] };+ expect(JSON.stringify(modelCall.messages)).toContain(base64);+ });+it("stages non-inlinable FilePart bytes into the sandbox and hands the model a text reference instead of bytes", async () => {setupMockAgent({finishReason: "stop",diff --git a/packages/eve/src/harness/tool-loop.ts b/packages/eve/src/harness/tool-loop.tsindex 8ae076a42..a92c37357 100644--- a/packages/eve/src/harness/tool-loop.ts+++ b/packages/eve/src/harness/tool-loop.ts@@ -75,7 +75,9 @@ import type { HarnessToolDefinition } from "#harness/execute-tool.js";import type { InputRequest } from "#shared/input.js";import {hydrateSandboxAttachments,+ moveToolResultFilesToUserMessages,stageAttachmentsToSandbox,+ stageToolResultMedia,} from "#harness/attachment-staging.js";import {compactMessages,@@ -650,12 +652,17 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {? consumeDeferredStepInput({ input, session }): { input, session };session = stepInput.session;- const resolvedCoordination = await resolvePendingCoordination({+ const coordination = await resolvePendingCoordination({emit,session,stepInput: stepInput.input,tools: config.tools,});+ // Workflow tool results join history here, so their files leave as refs too.+ const resolvedCoordination =+ coordination.outcome === "resolved"+ ? { ...coordination, messages: await stageToolResultMedia(coordination.messages) }+ : coordination;if (resolvedCoordination.outcome === "unresolved") {return { next: null, session: resolvedCoordination.session };}@@ -1525,6 +1532,9 @@ export function createToolLoopHarness(config: ToolLoopHarnessConfig): StepFn {);generation.begin();modelMessages = await hydrateSandboxAttachments(currentMessages.nonSystemMessages);+ if (sendsToolResultFilesAsText(model)) {+ modelMessages = moveToolResultFilesToUserMessages(modelMessages);+ }const { instructions, telemetryRuntimeContext = {} } = prepareModelCallInput(opts.extraSystemNote,);@@ -2602,7 +2612,7 @@ async function handleStepResult(input: {messages: rawResponseMessages,providerExecutedOutcomeIds,});- const responseMessages = normalizedProviderHistory.messages;+ const responseMessages = await stageToolResultMedia(normalizedProviderHistory.messages);const baseSession = setRequestEnvelopeTokens({@@ -3128,6 +3138,15 @@ async function finishTurn(input: {return { next: null, session, settledTurn };}+/** Chat Completions models receive `content` tool outputs as JSON text. */+function sendsToolResultFilesAsText(model: LanguageModel): boolean {+ return (+ typeof model !== "string" &&+ typeof model.provider === "string" &&+ model.provider.endsWith(".chat")+ );+}+function createNextCompactionConfig(current: CompactionConfig,durablePromptMessageCount: number | undefined,diff --git a/packages/eve/src/internal/attachments/media-metadata.test.ts b/packages/eve/src/internal/attachments/media-metadata.test.tsnew file mode 100644index 000000000..4e129b885--- /dev/null+++ b/packages/eve/src/internal/attachments/media-metadata.test.ts@@ -0,0 +1,89 @@+import { describe, expect, it } from "vitest";++import { estimateMediaTokens, readMediaMetadata } from "#internal/attachments/media-metadata.js";+import { pngBytes } from "#internal/testing/media-fixtures.js";++function gifBytes(width: number, height: number): Buffer {+ const bytes = Buffer.alloc(13);+ bytes.write("GIF89a", 0, "latin1");+ bytes.writeUInt16LE(width, 6);+ bytes.writeUInt16LE(height, 8);+ return bytes;+}++function jpegBytes(width: number, height: number): Buffer {+ // SOI, an APP0 segment the reader must skip, then SOF0 with the frame size.+ const app0 = Buffer.from([0xff, 0xe0, 0x00, 0x10, ...Buffer.alloc(14)]);+ const sof0 = Buffer.alloc(19);+ sof0.writeUInt16BE(0xffc0, 0);+ sof0.writeUInt16BE(17, 2);+ sof0.writeUInt8(8, 4);+ sof0.writeUInt16BE(height, 5);+ sof0.writeUInt16BE(width, 7);+ return Buffer.concat([Buffer.from([0xff, 0xd8]), app0, sof0]);+}++function webpBytes(chunk: "VP8 " | "VP8L" | "VP8X", width: number, height: number): Buffer {+ const bytes = Buffer.alloc(32);+ bytes.write("RIFF", 0, "latin1");+ bytes.write("WEBP", 8, "latin1");+ bytes.write(chunk, 12, "latin1");+ if (chunk === "VP8 ") {+ bytes.set([0x9d, 0x01, 0x2a], 23);+ bytes.writeUInt16LE(width, 26);+ bytes.writeUInt16LE(height, 28);+ } else if (chunk === "VP8L") {+ bytes.writeUInt8(0x2f, 20);+ bytes.writeUInt32LE(((width - 1) | ((height - 1) << 14)) >>> 0, 21);+ } else {+ bytes.writeUIntLE(width - 1, 24, 3);+ bytes.writeUIntLE(height - 1, 27, 3);+ }+ return bytes;+}++describe("readMediaMetadata", () => {+ it.each([+ ["PNG", pngBytes(640, 480), "image/png"],+ ["GIF", gifBytes(640, 480), "image/gif"],+ ["JPEG", jpegBytes(640, 480), "image/jpeg"],+ ["lossy WebP", webpBytes("VP8 ", 640, 480), "image/webp"],+ ["lossless WebP", webpBytes("VP8L", 640, 480), "image/webp"],+ ["extended WebP", webpBytes("VP8X", 640, 480), "image/webp"],+ ])("reads %s dimensions from the header", (_format, bytes, mediaType) => {+ expect(readMediaMetadata(bytes, mediaType)).toMatchObject({ height: 480, width: 640 });+ });++ it("counts PDF page objects without counting the page tree", () => {+ const pdf = Buffer.from(+ "%PDF-1.4\n1 0 obj << /Type /Pages /Count 2 >>\n2 0 obj << /Type /Page >>\n3 0 obj << /Type/Page >>",+ "latin1",+ );++ expect(readMediaMetadata(pdf, "application/pdf")).toMatchObject({ pages: 2 });+ });++ it("leaves dimensions out when the header is not a known image format", () => {+ expect(readMediaMetadata(Buffer.from("not an image"), "image/heic")).toEqual({+ mediaType: "image/heic",+ size: 12,+ });+ });+});++describe("estimateMediaTokens", () => {+ it("counts image patches up to the largest provider cap", () => {+ expect(+ estimateMediaTokens({ height: 1000, mediaType: "image/png", size: 1, width: 1000 }),+ ).toBe(1296);+ expect(+ estimateMediaTokens({ height: 3000, mediaType: "image/png", size: 1, width: 4000 }),+ ).toBe(4784);+ expect(estimateMediaTokens({ mediaType: "image/png", size: 1 })).toBe(4784);+ });++ it("counts PDFs per page and other files by their base64 size", () => {+ expect(estimateMediaTokens({ mediaType: "application/pdf", pages: 3, size: 9 })).toBe(9000);+ expect(estimateMediaTokens({ mediaType: "application/octet-stream", size: 9000 })).toBe(3000);+ });+});diff --git a/packages/eve/src/internal/attachments/media-metadata.ts b/packages/eve/src/internal/attachments/media-metadata.tsnew file mode 100644index 000000000..627028df9--- /dev/null+++ b/packages/eve/src/internal/attachments/media-metadata.ts@@ -0,0 +1,129 @@+/**+ * Byte-level media facts read once when a file enters the session, so token+ * estimates never need the payload again.+ */++/** Facts about one file payload that drive its token estimate. */+export interface MediaMetadata {+ readonly mediaType: string;+ readonly size: number;+ readonly width?: number;+ readonly height?: number;+ readonly pages?: number;+}++// Anthropic bills one visual token per 28px patch and caps each image at+// 4,784; OpenAI's 32px patches times its 1.2 multiplier land just under the+// 28px count, and Gemini caps images at 2,240. One patch rule under the+// largest common cap stays within the right magnitude for every provider.+const IMAGE_PATCH_PX = 28;+const MAX_IMAGE_TOKENS = 4_784;++// Providers render each PDF page as text plus a page image; ~3,000 tokens per+// page is the upper end of that range.+const PDF_PAGE_TOKENS = 3_000;++/** Reads the dimensions or page count of a payload, as far as its header allows. */+export function readMediaMetadata(bytes: Uint8Array, mediaType: string): MediaMetadata {+ const metadata = { mediaType, size: bytes.byteLength };+ if (mediaType.startsWith("image/")) {+ const dimensions = readImageDimensions(bytes);+ return dimensions === undefined ? metadata : { ...metadata, ...dimensions };+ }+ if (mediaType === "application/pdf") {+ const pages = countPdfPages(bytes);+ return pages === undefined ? metadata : { ...metadata, pages };+ }+ return metadata;+}++/**+ * Estimates the input tokens a provider bills for one file. Images and PDFs+ * use provider-style costs; anything else falls back to the size of its+ * base64 text, which is how text-only providers receive it.+ */+export function estimateMediaTokens(metadata: MediaMetadata): number {+ if (metadata.mediaType.startsWith("image/")) {+ if (metadata.width === undefined || metadata.height === undefined) return MAX_IMAGE_TOKENS;+ const patches =+ Math.ceil(metadata.width / IMAGE_PATCH_PX) * Math.ceil(metadata.height / IMAGE_PATCH_PX);+ return Math.min(patches, MAX_IMAGE_TOKENS);+ }+ if (metadata.mediaType === "application/pdf" && metadata.pages !== undefined) {+ return metadata.pages * PDF_PAGE_TOKENS;+ }+ return Math.ceil(metadata.size / 3);+}++/** Reads PNG, GIF, JPEG, or WebP dimensions from the leading bytes. */+export function readImageDimensions(+ bytes: Uint8Array,+): { readonly width: number; readonly height: number } | undefined {+ const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);+ const ascii = (start: number, end: number) =>+ String.fromCharCode(...bytes.subarray(start, Math.min(end, bytes.byteLength)));++ if (bytes.byteLength >= 24 && view.getUint32(0) === 0x89504e47) {+ return dimensions(view.getUint32(16), view.getUint32(20));+ }+ if (bytes.byteLength >= 10 && ascii(0, 4) === "GIF8") {+ return dimensions(view.getUint16(6, true), view.getUint16(8, true));+ }+ if (bytes.byteLength >= 4 && view.getUint16(0) === 0xffd8) {+ return readJpegDimensions(bytes, view);+ }+ if (bytes.byteLength >= 30 && ascii(0, 4) === "RIFF" && ascii(8, 12) === "WEBP") {+ const chunk = ascii(12, 16);+ if (chunk === "VP8 ") {+ return dimensions(view.getUint16(26, true) & 0x3fff, view.getUint16(28, true) & 0x3fff);+ }+ if (chunk === "VP8L") {+ const bits = view.getUint32(21, true);+ return dimensions((bits & 0x3fff) + 1, ((bits >>> 14) & 0x3fff) + 1);+ }+ if (chunk === "VP8X") {+ return dimensions(readUint24(bytes, 24) + 1, readUint24(bytes, 27) + 1);+ }+ }+ return undefined;+}++function readJpegDimensions(+ bytes: Uint8Array,+ view: DataView,+): { readonly width: number; readonly height: number } | undefined {+ let offset = 2;+ while (offset + 9 < bytes.byteLength) {+ if (bytes[offset] !== 0xff) return undefined;+ const marker = bytes[offset + 1]!;+ if (marker === 0xff) {+ offset += 1;+ continue;+ }+ // SOF0–SOF15 carry the frame size, except DHT (C4), JPG (C8), and DAC (CC).+ if (marker >= 0xc0 && marker <= 0xcf && marker !== 0xc4 && marker !== 0xc8 && marker !== 0xcc) {+ return dimensions(view.getUint16(offset + 7), view.getUint16(offset + 5));+ }+ offset += 2 + view.getUint16(offset + 2);+ }+ return undefined;+}++function countPdfPages(bytes: Uint8Array): number | undefined {+ const text = Buffer.from(bytes.buffer, bytes.byteOffset, bytes.byteLength).toString("latin1");+ // Page objects in compressed object streams are invisible here; the+ // estimate then falls back to the payload size.+ const pages = text.match(/\/Type\s*\/Page(?![A-Za-z])/g)?.length ?? 0;+ return pages > 0 ? pages : undefined;+}++function readUint24(bytes: Uint8Array, offset: number): number {+ return bytes[offset]! | (bytes[offset + 1]! << 8) | (bytes[offset + 2]! << 16);+}++function dimensions(+ width: number,+ height: number,+): { readonly width: number; readonly height: number } | undefined {+ return width > 0 && height > 0 ? { height, width } : undefined;+}diff --git a/packages/eve/src/internal/attachments/sandbox-refs.test.ts b/packages/eve/src/internal/attachments/sandbox-refs.test.tsindex d9dbcf4c6..53eba01c3 100644--- a/packages/eve/src/internal/attachments/sandbox-refs.test.ts+++ b/packages/eve/src/internal/attachments/sandbox-refs.test.ts@@ -71,9 +71,11 @@ describe("encodeSandboxRef", () => {describe("decodeSandboxRef", () => {it("round-trips every field end-to-end via a URL instance", () => {const original: SandboxRef = {+ height: 1080,mediaType: "image/png",path: "/vercel/sandbox/workspace/attachments/abc123/diagram.png",size: 524288,+ width: 1920,};const decoded = decodeSandboxRef(encodeSandboxRef(original));expect(decoded).toEqual(original);diff --git a/packages/eve/src/internal/attachments/sandbox-refs.ts b/packages/eve/src/internal/attachments/sandbox-refs.tsindex b0f49648d..fcf33aae2 100644--- a/packages/eve/src/internal/attachments/sandbox-refs.ts+++ b/packages/eve/src/internal/attachments/sandbox-refs.ts@@ -5,10 +5,12 @@* have been written into the sandbox:** ```- * eve-sandbox:?path=<urlencoded-resolved-path>&size=<bytes>&type=<mediaType>+ * eve-sandbox:?path=<urlencoded-resolved-path>&size=<bytes>&type=<mediaType>[&width=&height=][&pages=]* ```*/+import type { MediaMetadata } from "#internal/attachments/media-metadata.js";+/*** Custom URL scheme used by every sandbox-resident attachment ref. The* trailing colon is part of the scheme per WHATWG URL semantics.@@ -18,19 +20,26 @@ export const SANDBOX_URL_SCHEME = "eve-sandbox:";const PATH_QUERY_KEY = "path";const SIZE_QUERY_KEY = "size";const TYPE_QUERY_KEY = "type";+const METADATA_QUERY_KEYS = ["width", "height", "pages"] as const;++/**+ * Upper bound, in bytes, on inbound images that hydrate as inline bytes.+ * Larger images reach the model as a text reference to their sandbox path.+ */+const INLINE_IMAGE_MAX_BYTES = 3 * 1024 * 1024;++/** Upper bound, in bytes, on inbound PDFs that hydrate as inline bytes. */+const INLINE_PDF_MAX_BYTES = 20 * 1024 * 1024;/*** Serializable description of one sandbox-resident file attachment.** `path` is the backend-native absolute path returned by- * {@link SandboxSession.resolvePath}; `size` and `mediaType` are- * snapshotted so hydration can decide whether to inline without- * re-reading the file.+ * {@link SandboxSession.resolvePath}; the {@link MediaMetadata} fields are+ * snapshotted so hydration and token estimates never re-read the file.*/-export interface SandboxRef {+export interface SandboxRef extends MediaMetadata {readonly path: string;- readonly size: number;- readonly mediaType: string;}function isValidSize(value: number): boolean {@@ -58,6 +67,10 @@ export function encodeSandboxRef(ref: SandboxRef): URL {url.searchParams.set(PATH_QUERY_KEY, ref.path);url.searchParams.set(SIZE_QUERY_KEY, String(ref.size));url.searchParams.set(TYPE_QUERY_KEY, ref.mediaType);+ for (const key of METADATA_QUERY_KEYS) {+ const value = ref[key];+ if (value !== undefined && isValidSize(value)) url.searchParams.set(key, String(value));+ }return url;}@@ -94,7 +107,24 @@ export function decodeSandboxRef(value: URL | string): SandboxRef {throw new Error('SandboxRef URL is missing the required "type" query param.');}- return { mediaType, path, size };+ const ref: { -readonly [K in keyof SandboxRef]: SandboxRef[K] } = { mediaType, path, size };+ for (const key of METADATA_QUERY_KEYS) {+ const value = Number(url.searchParams.get(key) ?? Number.NaN);+ if (isValidSize(value) && value > 0) ref[key] = value;+ }+ return ref;+}++/**+ * Whether an inbound attachment ref reaches the model as bytes rather than a+ * text reference. Only shapes every major provider reads natively qualify:+ * images up to 3 MiB and PDFs up to 20 MiB. Pure in the ref, so a message+ * renders the same way on every model call.+ */+export function inlinesSandboxRefAsBytes(ref: SandboxRef): boolean {+ if (ref.mediaType.startsWith("image/")) return ref.size <= INLINE_IMAGE_MAX_BYTES;+ if (ref.mediaType === "application/pdf") return ref.size <= INLINE_PDF_MAX_BYTES;+ return false;}/**diff --git a/packages/eve/src/internal/testing/media-fixtures.ts b/packages/eve/src/internal/testing/media-fixtures.tsnew file mode 100644index 000000000..75e0b7b82--- /dev/null+++ b/packages/eve/src/internal/testing/media-fixtures.ts@@ -0,0 +1,13 @@+/**+ * A PNG signature and IHDR chunk for the given dimensions, padded with+ * `paddingBytes` of zeros to model a realistic payload size.+ */+export function pngBytes(width: number, height: number, paddingBytes = 0): Buffer {+ const header = Buffer.alloc(24 + paddingBytes);+ Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]).copy(header, 0);+ header.writeUInt32BE(13, 8);+ header.write("IHDR", 12, "latin1");+ header.writeUInt32BE(width, 16);+ header.writeUInt32BE(height, 20);+ return header;+}diff --git a/packages/eve/src/tools/provided/read-file.ts b/packages/eve/src/tools/provided/read-file.tsindex c12769eaf..680b0104a 100644--- a/packages/eve/src/tools/provided/read-file.ts+++ b/packages/eve/src/tools/provided/read-file.ts@@ -1,6 +1,12 @@import { type ReadFileInput, executeReadFileOnSandbox } from "#execution/sandbox/read-file.js";import { toolLabel } from "#tools/tool-label.js";import { defineTool, type ToolDefinition } from "#tools/definition.js";+import { basename } from "node:path";++import { contextStorage } from "#context/container.js";+import { SandboxKey } from "#context/keys.js";+import { resolveAbsoluteFilePath } from "#execution/sandbox/require-sandbox.js";+import { toolOutput, toolOutputPart } from "#tools/model-output.js";import { defineJsonSchema } from "#tools/schema.js";export interface ReadFileToolInput {@@ -11,6 +17,13 @@ export interface ReadFileToolInput {export interface ReadFileToolOutput {content: string;+ /** Set when the file is a PNG, JPEG, GIF, or WebP image the model sees as an image. */+ image?: {+ height?: number;+ mediaType: string;+ size: number;+ width?: number;+ };nextOffset?: number;path: string;totalLines: number;@@ -49,6 +62,17 @@ export const READ_FILE_OUTPUT_SCHEMA = defineJsonSchema<ReadFileToolOutput>({type: "object",properties: {content: { type: "string" },+ image: {+ type: "object",+ properties: {+ height: { type: "integer", minimum: 1 },+ mediaType: { type: "string" },+ size: { type: "integer", minimum: 0 },+ width: { type: "integer", minimum: 1 },+ },+ required: ["mediaType", "size"],+ additionalProperties: false,+ },nextOffset: { type: "integer", minimum: 1 },path: { type: "string" },totalLines: { type: "integer", minimum: 0 },@@ -73,6 +97,7 @@ export const readFile: ToolDefinition<ReadFileToolInput, ReadFileToolOutput> = d"- To read later sections, call this tool again with a larger offset.",'- Contents are returned with each line prefixed by its line number as `<line>: <content>`. For example, if a file has contents "foo\\n", you will receive "1: foo\\n".',"- Any line longer than 2000 characters is truncated.",+ "- PNG, JPEG, GIF, and WebP files up to 3 MiB are shown to you as images, including files named in `Attached file <path>` references.","- Call this tool in parallel when you know there are multiple files you want to read.","- Avoid tiny repeated slices (30 line chunks). If you need more context, read a larger window.",].join("\n"),@@ -81,6 +106,27 @@ export const readFile: ToolDefinition<ReadFileToolInput, ReadFileToolOutput> = d},inputSchema: READ_FILE_INPUT_SCHEMA,outputSchema: READ_FILE_OUTPUT_SCHEMA,+ async toModelOutput(output) {+ if (output.image === undefined) return toolOutput.json(output);+ // The bytes load here rather than in `execute`, so `action.result` never+ // carries them. The harness then stages this file part under its own name+ // as a sandbox ref before it enters history.+ const sandbox = await contextStorage.getStore()?.get(SandboxKey)?.get();+ const bytes =+ sandbox === undefined || sandbox === null+ ? null+ : await sandbox.readBinaryFile({+ path: await resolveAbsoluteFilePath(sandbox, output.path),+ });+ if (bytes === null) return toolOutput.text(`${output.content} The image could not be loaded.`);+ return toolOutput.content([+ toolOutputPart.text(output.content),+ toolOutputPart.file(Buffer.from(bytes).toString("base64"), {+ filename: basename(output.path),+ mediaType: output.image.mediaType,+ }),+ ]);+ },});export default readFile;diff --git a/research/tool-result-media.md b/research/tool-result-media.mdnew file mode 100644index 000000000..31d41b876--- /dev/null+++ b/research/tool-result-media.md@@ -0,0 +1,243 @@+---+issue: https://github.com/vercel/eve/issues/3833+status: implemented+last_updated: "2026-10-02"+---++# Media in session history: storage, prompt cache, and token estimates++Covers [#3833](https://github.com/vercel/eve/issues/3833) (tool-result media+retained in persisted history) and+[#3799](https://github.com/vercel/eve/issues/3799) (compaction counts image+bytes as text). Supersedes #4016/#4017 (stub at turn settle) and #3802 (stub-size+estimate).++## Summary++Durable session history stores **media refs, never bytes**. Every model call+hydrates every ref **deterministically**, so the provider sees byte-identical+prompts from call to call. Media leaves the model view only during+**compaction**, which already resets the prompt cache. A **media-aware+estimator** replaces `JSON.stringify(...).length / 4` for media, so compaction+fires at the right time.++Together these fix storage, which the refs remove, and keep the prompt cache+intact, since no rewrite happens between compactions. Images stay visible+until compaction, like any other context. Old images can still be reopened.++## What the prompt cache requires++Provider facts that constrain the design:++| | Anthropic (direct, Bedrock, Vertex) | OpenAI (Responses) | Gemini |+| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------- |+| Matching | Prefix hash at breakpoints. Reads look back ≤ 20 positions for earlier writes. | Longest matching prefix across breakpoints. | Implicit prefix caching (2.5+). |+| Media in cached prefix | Yes | Yes, including `detail` | Yes |+| Special media rule | **Adding or removing images anywhere invalidates the messages cache** ([docs](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#what-invalidates-the-cache)) | — | — |+| Pricing | Write 1.25×, read 0.1×, 5-minute TTL | Discounted reads | Discounted reads |++eve already designs for cache stability. `prompt-cache.ts` adds Anthropic+breakpoints on tools, system, the last message, and the previous assistant+message, and sets `gateway.caching = "auto"`. Announcements and connection+tools are deduped to keep the prefix stable. The rules that follow for media:++1. **Rendering is append-only.** Once a message has been sent, it renders+ byte-identically on later calls: same bytes, `mediaType`, part order,+ filename, and provider options. Hydration must be a pure function of the+ ref, never of turn age, media count, or wall-clock time.+2. **Image presence must not toggle** between calls. On Anthropic, going from+ "some image present" to "no image present", or back, invalidates the whole+ messages cache.+3. **Evict rarely and in batches**, at moments that already reset the cache.+ Anthropic's own tools follow this rule: context editing has `clear_at_least`,+ and the computer-use reference has `min_removal_threshold`.+4. **Hydration failures must not silently change the prompt.** A transient+ store error retries the step. Degrading to text is for media that is gone+ for good, as in #325's stale-snapshot case.++## Why #4016 has the wrong shape++#4016 stubs tool-result files when a turn settles. That rewrites earlier+messages between ordinary calls, which breaks rules 1–3:++- **Anthropic, images in every turn** (the #3833 workload): call 1 of each turn+ has no images because the earlier ones were stubbed, so presence toggles and+ the whole messages cache is rewritten. The tool then returns images, presence+ toggles back, and the cache is rewritten again. Worked example: 50k-token+ history, 16 retained images at 1.5k tokens each, 2 calls per turn, warm+ cache. Keeping images costs about 18k input-token-equivalents per turn.+ #4016 costs about 128k, roughly 7× more. With a cold cache (turns more than+ 5 minutes apart) the second rewrite still happens.+- **OpenAI and Gemini**: the divergence starts at the previous turn's first+ image. Only that turn's (now small) tail is re-processed, so token cost is+ about neutral. The model still permanently loses the image.+- **Semantics:** a later turn can't look at an earlier result again (`read_file`+ is text-only). Inbound attachments, which stay hydrated forever, and tool+ media now follow different rules.+- **Coverage:** within-turn persistence is not addressed. Each `turnStep`+ input (`step_created.input`) still carries every byte returned so far in the+ turn. That is O(steps²) for screenshot-loop agents.++The presence-toggle numbers rely on Anthropic's documented rule. Verification+below measures them directly.++## Design++### Invariant++> History that leaves a durable step never contains inline media bytes.+> Media appears as bytes only in the per-call model request.++`hydrateSandboxAttachments` already enforces this for inbound attachments. The+design generalizes it to tool results (authored `toolOutputPart.file`, MCP+`image` / `audio` / `resource.blob` via `connectionToolModelOutput`, and+workflow-tool projections).++### Media refs++```text+tool result ──► handleStepResult / coordination: put(bytes) ──► ref in history+ (before any provider has seen the part) │ durable+model call ◄── hydrate every ref (pure, content-addressed) ◄───────┘+compaction ──► batched eviction: ref ──► "Attached file <path> (<type>)"+```++- **Ref pass.** Runs where new messages enter history: `handleStepResult`+ response messages and coordination tool results. Because tool results reach+ a provider only on the next call, the first provider request already sees+ the hydrated, canonical form, so there is no one-time cache divergence.+- **Ref shape.** Today's `eve-sandbox:` ref, extended with optional+ `width`, `height`, and `pages`. A tool-result file part keeps its filename+ and media type and swaps `data: { type: "data" }` for+ `data: { type: "url", url: <eve-sandbox: ref> }`. Dimensions and page counts+ are parsed once at write time from image headers (PNG, JPEG, GIF, WebP) and+ the PDF page tree, with no dependency. They exist for the estimator.+- **Store contract.** Write bytes to `/workspace/.eve/attachments/<sha>/<name>` and+ read them back by ref path. Content-addressed and immutable. Refs are+ minted only by eve: the HTTP channel already rejects internal ref schemes,+ and `normalizeToolModelOutput` rejects non-`data` file tags. The sandbox is+ the interim backend, the same one inbound attachments use. Its costs: a+ vision tool forces the lazy sandbox to boot, hydration needs a running+ sandbox, and snapshot staleness (#325). **The backend belongs to the+ first-class file work** (see the boundary below).+- **Hydration.** `hydrateSandboxAttachments` descends into `tool-result`+ `content` outputs. It still decides inline-vs-text from the ref alone (size+ and media type) and hydrates **all** refs on every call.++### Eviction only at compaction++Media stays in the model view until compaction. Compaction's existing first+rung, the tool-result cap heuristic, already replaces files in the older region+with stubs in one pass, before any summarization. Its stub now names the+staged sandbox path. The sandbox keeps the bytes, and an image-aware+`read_file` (returning a content image part) lets the agent reopen an evicted+image. Reopening adds a new ref, and content addressing dedupes the bytes.++### Token estimation++Provider usage stays authoritative. The estimate only bridges the gap until+the next usage report (`getInputTokenCount`) and drives compaction's own+rulers (`evaluateThreshold`, `selectRecentWindowSize`, `withResumptionGuard`).+It needs to be within the right order of magnitude, not exact. Provider image+costs vary too much for a constant:++| Provider | Image tokens | Max per image |+| -------------- | -------------------------------------------------- | ---------------------------------- |+| Anthropic | `⌈w/28⌉·⌈h/28⌉` after downscale | 4,784 (Claude 4.7+), 1,568 earlier |+| OpenAI GPT-5.x | `⌈w/32⌉·⌈h/32⌉ × 1.2`; `auto` = `original` on 5.5+ | ~36k (30k patches) |+| Gemini 3 | Fixed by `media_resolution` | 1,120 default, 2,240 `ultra_high` |++One estimator in `token-estimate.ts` walks parts structurally:++- **Images:** `⌈w/28⌉·⌈h/28⌉`, capped at 4,784, the largest Anthropic cost+ and above Gemini's 2,240. A 28 px patch overestimates OpenAI's 32 px × 1.2+ slightly. Unknown dimensions count as 4,784. One cap keeps the estimator free+ of model identity; the cost is underestimating very large images on OpenAI's+ `original` detail until the next usage report.+- **PDFs:** pages × 3,000. Unknown page count falls back to the character+ heuristic.+- **Other media and all non-media content:** the existing character heuristic,+ with media payloads removed.++This applies to inline parts and to refs, including inbound sandbox refs,+which today count as a few bytes but hydrate to full images. Every+`estimateTokens` caller shares it. #3799's case becomes about 44k + two page+images (~9.5k on Claude 4.7+), against a measured `204152.25` before.++### Providers that stringify tool-result media++`@ai-sdk/openai` `.chat()` serializes `content` tool outputs with+`JSON.stringify`, so base64 reaches the model as text. That costs about 100k++tokens per image and the model can't see it. OpenAI-compatible chat providers+(LiteLLM-style gateways, like the one in #3799) likely do the same. eve's own+OpenAI factory uses `.responses()`, and the Anthropic and Google adapters send+native blocks.++For `*.chat` providers, the model-view projection moves tool-result media into+a user message that immediately follows the tool message. The move is+deterministic, so it is cache-stable. Gateway server-side conversion is+unverified and needs a live test.++## Boundary with first-class file support++| This design (harness) | First-class files |+| ------------------------------------------------- | ---------------------------------------------------------------------- |+| No-bytes-in-history invariant and ref pass | Store backend (sandbox, Blob, world-native), retention, deletion, auth |+| Deterministic hydration and media cache rules | Public authoring API for files and handles |+| Media-aware estimator and media metadata | Client and channel rendering of files |+| Compaction eviction rung, image-aware `read_file` | Provider Files API uploads (`file_id`) as an optimization |++The interface between them is the store contract and ref shape above. This+design ships on the sandbox backend and takes the new backend when that work+lands.++## Follow-ups++- A compaction trigger on provider media limits. Anthropic allows 100 or 600+ images per request and rejects any image over 2000 px once a request has more+ than 20 images+ ([vision docs](https://platform.claude.com/docs/en/build-with-claude/vision)).+ Ref metadata already carries the dimensions.+- Per-provider image caps, if the single cap proves too coarse.+- A per-process read cache for hydration, keyed by content hash.+- A development-time guard that history leaving a step has no inline bytes.++## Out of scope++- Gemini `thoughtSignature` retention (#3833). Needs provider confirmation.+- Raw `execute` output in `action.result` events. It is persisted once, not+ per step. Document the pattern of returning a reference from `execute` and+ resolving bytes in `toModelOutput`.+- Accurate audio and video costing.+- Timing-based eviction (evicting only when the cache is already cold). It+ saves tokens but makes visibility depend on timing.++## Verification++- **Unit:**+ - The tool loop writes a tool-result file to the sandbox once, keeps only the+ ref in history, and both later calls (next step, next turn) receive the+ exact file the tool returned.+ - A workflow tool's projected file is staged the same way.+ - The estimator counts images by patches from header or ref metadata, keeps+ path-reference attachments as text, and the #3799 case stays near the+ measured baseline (red on the old estimator).+ - Compaction stubs a staged file with its sandbox path.+ - `*.chat` relocation moves files into a following user message.+ - `read_file` returns PNGs as image parts and rejects images over 3 MiB.+- **E2E:** `agent-tools` `to-model-output-content-parts` drops its+ `real-model` tag. The fixture's mock model decodes the stripe colors from the+ image it receives and fails the turn when the replay turn lacks it, so world+ suites gate on hydration across durable steps. `agent-compaction-regressions`+ accepts the path-named stub.+- **Live cache experiment** (gateway key required; not run): same session shape+ under keep-with-refs and #4016-style stubbing. Record `cacheReadTokens` and+ `cacheWriteTokens` from `TokenUsage`, plus Anthropic cache diagnostics+ (`messages_changed`). Run on Claude, GPT-5.x, and Gemini, directly and+ through the gateway.++## Disposition of open PRs++- **#4016/#4017:** superseded by refs; close.+- **#3801/#3802:** superseded by the media-aware estimator and its regression+ test; close.