Skip to content

armgit / eve

Public Imported from GitHub
0

The Open Framework for Building Agents

feat(eve): add `modelOptions.promptCache.anthropic.ttl` to set Anthropic cache breakpoints (#4584)

Ben Pankow91f4b68359f3
diff --git a/.changeset/anthropic-prompt-cache-options.md b/.changeset/anthropic-prompt-cache-options.md
new file mode 100644
index 000000000..0203129db
--- /dev/null
+++ b/.changeset/anthropic-prompt-cache-options.md
@@ -0,0 +1,5 @@
+---
+"eve": patch
+---
+
+Add `modelOptions.promptCache` for models eve calls directly. `promptCache: { anthropic: { ttl: "1h" } }` switches eve's Anthropic cache breakpoints to a 1-hour lifetime. `promptCache: { anthropic: {} }` turns breakpoints on for an Anthropic model eve can't recognize from its id, such as a Bedrock application inference profile.
diff --git a/docs/agent-config.md b/docs/agent-config.md
index 7c11eea60..be095607e 100644
--- a/docs/agent-config.md
+++ b/docs/agent-config.md
@@ -152,6 +152,29 @@ the agent-level setting for that selection. Omitting it inherits the agent setti
Run `eve set model --reasoning high` to update this field from the command line.
+## Prompt caching
+
+eve caches prompts automatically for AI Gateway models and for Anthropic models
+it recognizes by provider or model id. Set `modelOptions.promptCache` on a
+directly called model to mark an unrecognized model as Anthropic (such as a
+Bedrock application inference profile) or to use a 1-hour cache:
+
+```ts title="agent/agent.ts"
+import { bedrock } from "@ai-sdk/amazon-bedrock";
+import { defineAgent } from "eve";
+
+export default defineAgent({
+ model: bedrock(process.env.BEDROCK_INFERENCE_PROFILE_ARN!),
+ modelContextWindowTokens: 200_000,
+ modelOptions: {
+ promptCache: { anthropic: { ttl: "1h" } }, // or { anthropic: {} } for the default 5m
+ },
+});
+```
+
+Not every Claude model supports a 1-hour cache. eve rejects `promptCache` on
+Gateway models; use `providerOptions.gateway.caching` instead.
+
## Compaction
Compaction summarizes older turns as you approach the context window. It's on by default, so you only tune when it kicks in. eve adds the estimated fixed checkpoint-prompt envelope to the trigger count, so compaction starts sooner than the conversation-only estimate. Lower `thresholdPercent` to compact sooner:
@@ -369,7 +392,7 @@ it falls back to the World's default retention period.
| Field | Type | Default | Description |
| -------------- | ------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `reasoning` | `AgentReasoningDefinition` | provider default | Provider-agnostic reasoning effort forwarded to the agent's turn model calls. |
-| `modelOptions` | `AgentModelOptionsDefinition` | none | Provider option overrides forwarded to the model call. |
+| `modelOptions` | `AgentModelOptionsDefinition` | none | Provider option overrides forwarded to the model call, and [`promptCache`](#prompt-caching) settings. |
| `limits` | `AgentLimitsDefinition` | field-specific | Framework-owned runtime limits. Sessions complete after 30 days by default; usage-limit defaults and inheritance are described above. Set a limit to `false` to disable it. |
| `experimental` | `AgentExperimentalDefinition` | unset | Unstable opt-ins. `workflow.world` selects the Workflow world package on the root agent; `workflow.modelCallsPerStep` batches sequential model calls into a wider replay unit; `workflow.retention` controls how long the durable runtime keeps run data. |
| `build` | `{ externalDependencies?: string[] }` | none | Hosted-build packaging controls. `externalDependencies` keeps listed packages external while eve compiles authored modules such as tools and channels, and traces those packages into the hosted output. |
diff --git a/packages/eve/extension-contracts/compatibility/dynamicTool/v78.ts b/packages/eve/extension-contracts/compatibility/dynamicTool/v78.ts
new file mode 100644
index 000000000..99d4faac8
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/dynamicTool/v78.ts
@@ -0,0 +1,16 @@
+import { defineDynamic, defineTool } from "#public/tools/index.js";
+
+// Epoch 79 adds an optional `promptCache` to the `modelOptions` a dynamic model
+// selection can return, which reaches this capability through `auto`'s result
+// type. Dynamic tools don't return model selections.
+export default defineDynamic({
+ events: {
+ "turn.started": (_event, ctx) => ({
+ model_id: defineTool({
+ description: "Return the active model identifier.",
+ inputSchema: { type: "object", properties: {} },
+ execute: () => ({ modelId: ctx.model?.id ?? null }),
+ }),
+ }),
+ },
+});
diff --git a/packages/eve/extension-contracts/compatibility/subagent/v34.ts b/packages/eve/extension-contracts/compatibility/subagent/v34.ts
new file mode 100644
index 000000000..2208acc03
--- /dev/null
+++ b/packages/eve/extension-contracts/compatibility/subagent/v34.ts
@@ -0,0 +1,9 @@
+import { defineAgent } from "#public/index.js";
+
+// Epoch 35 adds an optional `modelOptions.promptCache`. A subagent that sets
+// only provider options keeps compiling.
+export default defineAgent({
+ description: "Summarize a customer's open support tickets.",
+ model: "anthropic/claude-sonnet-5",
+ modelOptions: { providerOptions: { gateway: { order: ["anthropic"] } } },
+});
diff --git a/packages/eve/extension-contracts/reports/dynamicTool/v79.json b/packages/eve/extension-contracts/reports/dynamicTool/v79.json
new file mode 100644
index 000000000..c30b0dc36
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/dynamicTool/v79.json
@@ -0,0 +1,16 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "dynamicTool",
+ "epoch": 79,
+ "sha256": "41b22408ac58cc95c4b94d1600422f47a0d1204d7702a717bf16708ad323fd34",
+ "exports": [
+ "DynamicToolEntry",
+ "DynamicToolEvents",
+ "DynamicToolResult",
+ "DynamicToolSet",
+ "auto",
+ "defineDurableCallback",
+ "defineDurableSchema",
+ "defineDynamic"
+ ]
+}
diff --git a/packages/eve/extension-contracts/reports/subagent/v35.json b/packages/eve/extension-contracts/reports/subagent/v35.json
new file mode 100644
index 000000000..44b11ff2a
--- /dev/null
+++ b/packages/eve/extension-contracts/reports/subagent/v35.json
@@ -0,0 +1,23 @@
+{
+ "kind": "eve-extension-capability-contract",
+ "capability": "subagent",
+ "epoch": 35,
+ "sha256": "80344ccec496d2e20d5d0a8e02852eecac6863a20fec493b1f2c9d1e7bbb9773",
+ "exports": [
+ "AgentCompactionDefinition",
+ "AgentDefinition",
+ "AgentModelDefinition",
+ "AgentStaticModelDefinition",
+ "DefinedAgent",
+ "DynamicLocalSubagentDefinition",
+ "DynamicSubagentDefinition",
+ "RemoteAgentDefinition",
+ "RemoteAgentDefinitionInput",
+ "WorkspaceAgentDefinition",
+ "WorkspaceAgentTransport",
+ "defineAgent",
+ "defineDynamic",
+ "defineRemoteAgent",
+ "defineWorkspaceAgent"
+ ]
+}
diff --git a/packages/eve/src/compiler/extension-compatibility.ts b/packages/eve/src/compiler/extension-compatibility.ts
index 772287745..3ef086040 100644
--- a/packages/eve/src/compiler/extension-compatibility.ts
+++ b/packages/eve/src/compiler/extension-compatibility.ts
@@ -89,10 +89,10 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
},
dynamicTool: {
- current: 78,
+ current: 79,
supported: [
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22, 31, 32, 33, 52, 54,
- 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 72, 73, 75, 76, 77, 78,
+ 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 72, 73, 75, 76, 77, 78, 79,
],
dropped: {
21: "Message and reasoning append events now expose deltas instead of cumulative snapshots.",
@@ -171,10 +171,10 @@ const EXTENSION_CAPABILITY_CONTRACTS = {
},
},
subagent: {
- current: 34,
+ current: 35,
supported: [
3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
- 34,
+ 34, 35,
],
dropped: {
1: "Persistent subagent sessions are now the default and the experimental opt-in was removed",
diff --git a/packages/eve/src/compiler/manifest.ts b/packages/eve/src/compiler/manifest.ts
index 2e0e398f3..d901e818a 100644
--- a/packages/eve/src/compiler/manifest.ts
+++ b/packages/eve/src/compiler/manifest.ts
@@ -25,6 +25,7 @@ import type { NamedSkillDefinition } from "#shared/skill-definition.js";
import { WEB_SEARCH_FALLBACK_PROVIDERS, WEB_SEARCH_PROVIDERS } from "#shared/web-search.js";
import {
AGENT_WORKFLOW_RETENTION_VALUES,
+ ANTHROPIC_PROMPT_CACHE_TTLS,
type InternalAgentDefinition,
type InternalAgentModelDefinition,
type InternalAgentCompactionDefinition,
@@ -577,6 +578,15 @@ const compiledRuntimeModelReferenceSchema: z.ZodType<CompiledRuntimeModelReferen
id: z.string(),
maxOutputTokens: z.number().int().positive().optional(),
source: moduleSourceRefSchema.optional(),
+ promptCache: z
+ .object({
+ anthropic: z
+ .object({ ttl: z.literal(ANTHROPIC_PROMPT_CACHE_TTLS).optional() })
+ .strict()
+ .optional(),
+ })
+ .strict()
+ .optional(),
providerOptions: z.record(z.string(), jsonObjectSchema).optional(),
routing: modelRoutingSchema,
})
@@ -1357,6 +1367,9 @@ function cloneCompiledRuntimeModelReference(
if (model.providerOptions !== undefined) {
clone.providerOptions = { ...model.providerOptions };
}
+ if (model.promptCache !== undefined) {
+ clone.promptCache = structuredClone(model.promptCache);
+ }
if (model.source !== undefined) {
clone.source = { ...model.source };
}
diff --git a/packages/eve/src/compiler/normalize-agent-config.ts b/packages/eve/src/compiler/normalize-agent-config.ts
index 134ab0766..f6d304867 100644
--- a/packages/eve/src/compiler/normalize-agent-config.ts
+++ b/packages/eve/src/compiler/normalize-agent-config.ts
@@ -11,8 +11,10 @@ import { parseJsonObject, type JsonObject } from "#shared/json.js";
import type { ModuleSourceRef } from "#shared/source-ref.js";
import {
isDynamicModelDefinition,
- type PublicAgentStaticModelDefinition,
+ type AgentPromptCacheDefinition,
type AgentToolExposure,
+ type ModelRouting,
+ type PublicAgentStaticModelDefinition,
} from "#shared/agent-definition.js";
import type { DynamicToolEventName } from "#dynamic/definition.js";
import type { CompiledAgentDefinition, CompiledRuntimeModelReference } from "#compiler/manifest.js";
@@ -66,6 +68,7 @@ export async function compileAgentConfig(
modelCatalog: context.modelCatalog,
purpose: "the primary compaction trigger model",
contextWindowTokens: definition.modelContextWindowTokens,
+ promptCache: definition.modelOptions?.promptCache,
providerOptions: definition.modelOptions?.providerOptions,
source: configModule,
sourcePath: configModulePath,
@@ -198,17 +201,20 @@ async function normalizeAuthoredModelReference(input: {
readonly modelCatalog: ManifestCompileContext["modelCatalog"];
readonly purpose: string;
readonly contextWindowTokens?: number;
+ readonly promptCache?: AgentPromptCacheDefinition;
readonly providerOptions?: Record<string, JsonObject>;
readonly source?: ModuleSourceRef;
readonly sourcePath?: string;
readonly value: PublicAgentStaticModelDefinition;
}): Promise<CompiledRuntimeModelReference> {
if (typeof input.value === "string") {
+ const routing = classifyModelRouting(input.value, input.providerOptions);
+ rejectGatewayPromptCache(input, routing);
return await withCompiledRuntimeModelLimits(
{
id: formatLanguageModelGatewayId(input.value),
providerOptions: parseProviderOptionsRecord(input.providerOptions),
- routing: classifyModelRouting(input.value, input.providerOptions),
+ routing,
},
input,
);
@@ -247,7 +253,9 @@ async function normalizeAuthoredModelReference(input: {
);
}
- const sourceBackedModel = {
+ const routing = classifyModelRouting(languageModel, input.providerOptions);
+ rejectGatewayPromptCache(input, routing);
+ const sourceBackedModel: CompiledRuntimeModelReference = {
id: formatLanguageModelGatewayId(languageModel),
source: {
exportName: source.exportName,
@@ -256,8 +264,9 @@ async function normalizeAuthoredModelReference(input: {
sourceId: source.sourceId,
},
providerOptions: parseProviderOptionsRecord(input.providerOptions),
- routing: classifyModelRouting(languageModel, input.providerOptions),
+ routing,
};
+ if (input.promptCache !== undefined) sourceBackedModel.promptCache = input.promptCache;
if (input.contextWindowTokens === undefined) {
// Codex models have no Gateway catalog entry, so use eve's known context limit.
@@ -290,6 +299,20 @@ async function normalizeAuthoredModelReference(input: {
return await withCompiledRuntimeModelLimits(sourceBackedModel, input);
}
+function rejectGatewayPromptCache(
+ input: {
+ readonly promptCache?: AgentPromptCacheDefinition;
+ readonly sourcePath?: string;
+ readonly value: PublicAgentStaticModelDefinition;
+ },
+ routing: ModelRouting,
+): void {
+ if (input.promptCache === undefined || routing.kind !== "gateway") return;
+ throw new Error(
+ `${input.sourcePath ?? "agent.ts"}: modelOptions.promptCache applies only to models eve calls directly, but "${formatLanguageModelGatewayId(input.value)}" routes through AI Gateway, which manages its own prompt caching. Remove promptCache, or configure Gateway caching with providerOptions.gateway.caching.`,
+ );
+}
+
function formatAgentConfigModulePath(
manifest: AgentSourceManifest,
configModule: ModuleSourceRef,
diff --git a/packages/eve/src/compiler/normalize-manifest.test.ts b/packages/eve/src/compiler/normalize-manifest.test.ts
index 0fd8e592b..6b4b19088 100644
--- a/packages/eve/src/compiler/normalize-manifest.test.ts
+++ b/packages/eve/src/compiler/normalize-manifest.test.ts
@@ -704,6 +704,62 @@ describe("compileAgentManifest source graph", () => {
});
});
+ it("carries a direct model's prompt cache options to the resolved agent", async () => {
+ const bedrockProfile = {
+ doGenerate: async () => ({}),
+ doStream: async () => ({}),
+ modelId: "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc",
+ provider: "amazon-bedrock",
+ specificationVersion: "v3",
+ } as never;
+ const sourceRegistry = registry([
+ {
+ logicalPath: "agent.ts",
+ loadNamespace: async () => ({
+ default: defineAgent({
+ model: bedrockProfile,
+ modelContextWindowTokens: 200_000,
+ modelOptions: { promptCache: { anthropic: { ttl: "1h" } } },
+ }),
+ }),
+ },
+ ]);
+ const compiled = await compileAgentManifest(manifest(), {
+ sourceRegistries: [sourceRegistry],
+ });
+ const moduleMap = await createProgrammaticCompiledModuleMap(compiled, [
+ frameworkAgentSourceRegistry,
+ sourceRegistry,
+ ]);
+
+ const resolved = await resolveAgent({
+ manifest: compiledAgentManifestSchema.parse(compiled),
+ moduleMap,
+ });
+
+ expect(resolved.config?.model?.promptCache).toEqual({ anthropic: { ttl: "1h" } });
+ });
+
+ it("rejects prompt cache options on an AI Gateway model", async () => {
+ const sourceRegistry = registry([
+ {
+ logicalPath: "agent.ts",
+ loadNamespace: async () => ({
+ default: defineAgent({
+ model: "anthropic/claude-opus-5",
+ modelOptions: { promptCache: { anthropic: { ttl: "1h" } } },
+ }),
+ }),
+ },
+ ]);
+
+ await expect(
+ compileAgentManifest(manifest(), { sourceRegistries: [sourceRegistry] }),
+ ).rejects.toThrow(
+ 'modelOptions.promptCache applies only to models eve calls directly, but "anthropic/claude-opus-5" routes through AI Gateway',
+ );
+ });
+
it("classifies extension mount initialization as runtime-only", async () => {
const discovered = manifest();
const extensionManifest = createAgentSourceManifest({
diff --git a/packages/eve/src/evals/types.ts b/packages/eve/src/evals/types.ts
index ec27f98e5..b0ebd8afb 100644
--- a/packages/eve/src/evals/types.ts
+++ b/packages/eve/src/evals/types.ts
@@ -413,7 +413,8 @@ export interface EveEvalTurn extends EveEvalAssertions, EveEvalOutputAssertions
export interface EveEvalJudgeConfig {
/** Decision model ID or instance. Defaults to the model used by `eve/ai` decide. */
readonly model?: DecisionModel;
- readonly modelOptions?: AgentModelOptionsDefinition;
+ /** Judge calls carry provider options only; eve places no prompt-cache breakpoints on them. */
+ readonly modelOptions?: Pick<AgentModelOptionsDefinition, "providerOptions">;
}
/** JSON content accepted as decision state, instructions, or rubric descriptions. */
diff --git a/packages/eve/src/harness/model-call/call.ts b/packages/eve/src/harness/model-call/call.ts
index d33aa2d10..2933a2fe8 100644
--- a/packages/eve/src/harness/model-call/call.ts
+++ b/packages/eve/src/harness/model-call/call.ts
@@ -48,7 +48,7 @@ import { estimateTokens } from "#harness/token-estimate.js";
import { buildToolApproval } from "#harness/tools.js";
import { throwIfTurnAborted } from "#harness/turn-cancellation.js";
import { addTurnUsage, type TokenUsageDelta } from "#harness/turn-tag-state.js";
-import type { StepResult } from "#harness/types.js";
+import { requireSessionModelReference, type StepResult } from "#harness/types.js";
import type { InstrumentationAttempt } from "#instrumentation/runtime.js";
import { createLogger, logError } from "#internal/logging.js";
import { maybeCompact } from "#harness/compaction/step.js";
@@ -128,7 +128,10 @@ export class ModelCaller {
this.prompt = prompt;
this.input = input;
- this.profile = resolveModelProfile(input.model);
+ this.profile = resolveModelProfile(
+ input.model,
+ requireSessionModelReference(step.session).promptCache,
+ );
this.attributionHeaders = buildGatewayAttributionHeaders(
this.profile,
step.config.runtimeIdentity,
diff --git a/packages/eve/src/harness/model-call/request.ts b/packages/eve/src/harness/model-call/request.ts
index 111d1d26b..fab2eeb07 100644
--- a/packages/eve/src/harness/model-call/request.ts
+++ b/packages/eve/src/harness/model-call/request.ts
@@ -12,6 +12,7 @@ import {
type HarnessModelMessage,
type UserModelMessage,
} from "#harness/messages.js";
+import type { ModelProfile } from "#harness/model-profile.js";
import { applySystemCacheBreakpoint } from "#harness/prompt-cache.js";
import type { Step } from "#harness/step/context.js";
import type { HarnessSession } from "#harness/types.js";
@@ -59,7 +60,7 @@ export function requestMessages(
export function modelInstructions(input: {
readonly session: HarnessSession;
readonly systemMessages: readonly SystemModelMessage[];
- readonly anthropicCache: boolean;
+ readonly anthropicCache: ModelProfile["anthropicCache"];
readonly extraSystemNote?: string;
}): SystemModelMessage | string | undefined {
const { session } = input;
@@ -74,7 +75,9 @@ export function modelInstructions(input: {
: [];
const instructions = [...extra, ...base, ...input.systemMessages];
return mergeSystemInstructions(
- input.anthropicCache ? applySystemCacheBreakpoint(instructions) : instructions,
+ input.anthropicCache
+ ? applySystemCacheBreakpoint(instructions, input.anthropicCache)
+ : instructions,
);
}
diff --git a/packages/eve/src/harness/model-call/tools.ts b/packages/eve/src/harness/model-call/tools.ts
index 0f2e714ea..99dac66f7 100644
--- a/packages/eve/src/harness/model-call/tools.ts
+++ b/packages/eve/src/harness/model-call/tools.ts
@@ -44,7 +44,7 @@ export async function prepareModelTools(
}
const effectiveTools = profile.anthropicCache
- ? applyLastToolCacheBreakpoint(modelTools)
+ ? applyLastToolCacheBreakpoint(modelTools, profile.anthropicCache)
: modelTools;
for (const [name, tool] of Object.entries(effectiveTools)) {
// Whatever produced this tool, the AI SDK must only receive its own
diff --git a/packages/eve/src/harness/model-profile.test.ts b/packages/eve/src/harness/model-profile.test.ts
index 932141076..940e9c332 100644
--- a/packages/eve/src/harness/model-profile.test.ts
+++ b/packages/eve/src/harness/model-profile.test.ts
@@ -3,20 +3,21 @@ import { MockLanguageModelV3 } from "ai/test";
import { describe, expect, it } from "vitest";
import { type ModelProfile, resolveModelProfile } from "#harness/model-profile.js";
+import type { AgentPromptCacheDefinition } from "#shared/agent-definition.js";
function model(provider: string, modelId = "test-model"): LanguageModel {
return new MockLanguageModelV3({ modelId, provider });
}
const direct = {
- anthropicCache: false,
+ anthropicCache: undefined,
filesOutsideToolResults: false,
gateway: false,
googleSearchDropsTools: false,
};
describe("resolveModelProfile", () => {
- it.each<[string, LanguageModel, ModelProfile]>([
+ it.each<[string, LanguageModel, ModelProfile, AgentPromptCacheDefinition?]>([
["a bare Gateway id", "openai/gpt-5", { ...direct, gateway: true, provider: "openai" }],
[
"a Gateway instance used by local auth",
@@ -31,17 +32,17 @@ describe("resolveModelProfile", () => {
[
"direct Anthropic",
model("anthropic.messages"),
- { ...direct, anthropicCache: true, provider: "anthropic" },
+ { ...direct, anthropicCache: { ttl: "5m" }, provider: "anthropic" },
],
[
"Vertex Anthropic",
model("vertex.anthropic"),
- { ...direct, anthropicCache: true, provider: "vertex" },
+ { ...direct, anthropicCache: { ttl: "5m" }, provider: "vertex" },
],
[
"Bedrock Converse with an Anthropic model id",
model("amazon-bedrock", "ANTHROPIC.claude-3-5-sonnet-20241022-v2:0"),
- { ...direct, anthropicCache: true, provider: "amazon-bedrock" },
+ { ...direct, anthropicCache: { ttl: "5m" }, provider: "amazon-bedrock" },
],
[
"Bedrock Converse with another model id",
@@ -71,7 +72,25 @@ describe("resolveModelProfile", () => {
{ ...direct, gateway: true, googleSearchDropsTools: true, provider: "google" },
],
["a provider-less test double", {} as LanguageModel, { ...direct, provider: "" }],
- ])("describes %s", (_, input, expected) => {
- expect(resolveModelProfile(input)).toEqual(expected);
+ [
+ "an opaque Bedrock inference profile declared as Anthropic",
+ model("amazon-bedrock", "arn:aws:bedrock:us-east-1:123:application-inference-profile/abc"),
+ { ...direct, anthropicCache: { ttl: "5m" }, provider: "amazon-bedrock" },
+ { anthropic: {} },
+ ],
+ [
+ "direct Anthropic with a 1-hour cache",
+ model("anthropic.messages"),
+ { ...direct, anthropicCache: { ttl: "1h" }, provider: "anthropic" },
+ { anthropic: { ttl: "1h" } },
+ ],
+ [
+ "a Gateway model, which caches on its own",
+ "anthropic/claude-opus-5",
+ { ...direct, gateway: true, provider: "anthropic" },
+ { anthropic: { ttl: "1h" } },
+ ],
+ ])("describes %s", (_, input, expected, promptCache) => {
+ expect(resolveModelProfile(input, promptCache)).toEqual(expected);
});
});
diff --git a/packages/eve/src/harness/model-profile.ts b/packages/eve/src/harness/model-profile.ts
index fc33dd5d9..11fec65af 100644
--- a/packages/eve/src/harness/model-profile.ts
+++ b/packages/eve/src/harness/model-profile.ts
@@ -1,5 +1,10 @@
import type { LanguageModel } from "ai";
+import type {
+ AgentPromptCacheDefinition,
+ AnthropicPromptCacheTtl,
+} from "#shared/agent-definition.js";
+
/**
* What eve assumes about the model serving one call. Every decision that shapes a request for
* its provider reads this record, so a step and its compaction agree about the model.
@@ -16,8 +21,8 @@ export interface ModelProfile {
* `anthropicCache`.
*/
readonly provider: string;
- /** Takes Anthropic prompt-cache breakpoints. */
- readonly anthropicCache: boolean;
+ /** Takes Anthropic prompt-cache breakpoints, which live for `ttl`. */
+ readonly anthropicCache: { readonly ttl: AnthropicPromptCacheTtl } | undefined;
/** Chat Completions APIs can't carry files in tool results, so they move to user messages. */
readonly filesOutsideToolResults: boolean;
/**
@@ -27,25 +32,34 @@ export interface ModelProfile {
readonly googleSearchDropsTools: boolean;
}
-export function resolveModelProfile(model: LanguageModel): ModelProfile {
+/**
+ * `promptCache` is the model's authored `modelOptions.promptCache`. It names the cache protocol
+ * of a model whose provider and id don't, such as a Bedrock application inference profile.
+ */
+export function resolveModelProfile(
+ model: LanguageModel,
+ promptCache?: AgentPromptCacheDefinition,
+): ModelProfile {
const provider = typeof model === "string" ? "gateway" : lowerCaseOrEmpty(model.provider);
const modelId = typeof model === "string" ? model : lowerCaseOrEmpty(model.modelId);
const topLevelProvider = provider.split(".")[0]!;
if (topLevelProvider === "gateway") {
return {
- anthropicCache: false,
+ anthropicCache: undefined,
filesOutsideToolResults: false,
gateway: true,
googleSearchDropsTools: PRE_GEMINI_3_MODEL.test(modelId),
provider: modelId.split("/")[0]!,
};
}
+ // The Bedrock Converse provider reports `amazon-bedrock` and carries the Anthropic identity in
+ // the model id (`anthropic.claude-…`).
+ const anthropic =
+ promptCache?.anthropic !== undefined ||
+ provider.includes("anthropic") ||
+ (provider.includes("bedrock") && modelId.includes("anthropic"));
return {
- // The Bedrock Converse provider reports `amazon-bedrock` and carries the Anthropic identity
- // in the model id (`anthropic.claude-…`).
- anthropicCache:
- provider.includes("anthropic") ||
- (provider.includes("bedrock") && modelId.includes("anthropic")),
+ anthropicCache: anthropic ? { ttl: promptCache?.anthropic?.ttl ?? "5m" } : undefined,
filesOutsideToolResults: provider.endsWith(".chat"),
gateway: false,
googleSearchDropsTools: PRE_GEMINI_3_MODEL.test(modelId),
diff --git a/packages/eve/src/harness/prompt-cache.test.ts b/packages/eve/src/harness/prompt-cache.test.ts
index 35c58e9b1..09d13215f 100644
--- a/packages/eve/src/harness/prompt-cache.test.ts
+++ b/packages/eve/src/harness/prompt-cache.test.ts
@@ -4,14 +4,47 @@ import { describe, expect, it } from "vitest";
import {
applyConversationCacheControl,
applyLastToolCacheBreakpoint,
+ applySystemCacheBreakpoint,
mergeGatewayAutoCaching,
} from "#harness/prompt-cache.js";
+const fiveMinutes = { ttl: "5m" } as const;
const marker = {
anthropic: { cacheControl: { type: "ephemeral" } },
bedrock: { cachePoint: { type: "default" } },
};
+describe("Anthropic cache breakpoints with a 1-hour TTL", () => {
+ it("carry the TTL on every breakpoint in both provider namespaces", () => {
+ const oneHour = { ttl: "1h" } as const;
+ const hourMarker = {
+ anthropic: { cacheControl: { type: "ephemeral", ttl: "1h" } },
+ bedrock: { cachePoint: { type: "default", ttl: "1h" } },
+ };
+ const tools = applyLastToolCacheBreakpoint(
+ { only: { description: "one" } } as unknown as ToolSet,
+ oneHour,
+ ) as Record<string, { providerOptions?: unknown }>;
+ const system = applySystemCacheBreakpoint([{ role: "system", content: "rules" }], oneHour);
+ const messages = applyConversationCacheControl(
+ [
+ { role: "user", content: "hi" },
+ { role: "assistant", content: "yo" },
+ { role: "user", content: "hi2" },
+ ],
+ oneHour,
+ );
+
+ expect(tools.only?.providerOptions).toEqual(hourMarker);
+ expect(system[0]?.providerOptions).toEqual(hourMarker);
+ expect(messages.map((message) => message.providerOptions)).toEqual([
+ undefined,
+ hourMarker,
+ hourMarker,
+ ]);
+ });
+});
+
describe("mergeGatewayAutoCaching", () => {
it("creates a fresh gateway block when base is undefined", () => {
expect(mergeGatewayAutoCaching(undefined)).toEqual({
@@ -52,7 +85,7 @@ describe("mergeGatewayAutoCaching", () => {
describe("applyLastToolCacheBreakpoint", () => {
it("is a no-op for an empty tool set", () => {
const tools = {} as ToolSet;
- expect(applyLastToolCacheBreakpoint(tools)).toEqual({});
+ expect(applyLastToolCacheBreakpoint(tools, fiveMinutes)).toEqual({});
});
it("attaches the marker only to the last tool", () => {
@@ -62,7 +95,7 @@ describe("applyLastToolCacheBreakpoint", () => {
gamma: { description: "third" },
} as unknown as ToolSet;
- const result = applyLastToolCacheBreakpoint(tools) as Record<
+ const result = applyLastToolCacheBreakpoint(tools, fiveMinutes) as Record<
string,
{ description: string; providerOptions?: unknown }
>;
@@ -83,7 +116,7 @@ describe("applyLastToolCacheBreakpoint", () => {
},
} as unknown as ToolSet;
- const result = applyLastToolCacheBreakpoint(tools) as Record<
+ const result = applyLastToolCacheBreakpoint(tools, fiveMinutes) as Record<
string,
{ providerOptions: Record<string, unknown> } | undefined
>;
@@ -102,7 +135,7 @@ describe("applyLastToolCacheBreakpoint", () => {
two: { description: "second" },
} as unknown as ToolSet;
const snapshot = JSON.parse(JSON.stringify(tools));
- applyLastToolCacheBreakpoint(tools);
+ applyLastToolCacheBreakpoint(tools, fiveMinutes);
expect(tools).toEqual(snapshot);
});
});
@@ -110,14 +143,14 @@ describe("applyLastToolCacheBreakpoint", () => {
describe("applyConversationCacheControl", () => {
it("returns a fresh empty array for empty input", () => {
const input: readonly ModelMessage[] = [];
- const out = applyConversationCacheControl(input);
+ const out = applyConversationCacheControl(input, fiveMinutes);
expect(out).toEqual([]);
expect(out).not.toBe(input);
});
it("marks a sole user message", () => {
const messages: ModelMessage[] = [{ role: "user", content: "hi" }];
- const out = applyConversationCacheControl(messages);
+ const out = applyConversationCacheControl(messages, fiveMinutes);
expect(out[0]).toEqual({
role: "user",
content: "hi",
@@ -132,7 +165,7 @@ describe("applyConversationCacheControl", () => {
{ role: "user", content: "hi2" },
{ role: "assistant", content: "yo2" },
];
- const out = applyConversationCacheControl(messages);
+ const out = applyConversationCacheControl(messages, fiveMinutes);
expect(out[0]).toEqual({ role: "user", content: "hi" });
expect(out[1]).toEqual({
@@ -167,7 +200,7 @@ describe("applyConversationCacheControl", () => {
],
},
];
- const out = applyConversationCacheControl(messages);
+ const out = applyConversationCacheControl(messages, fiveMinutes);
// The trailing tool-result message carries the final breakpoint.
expect((out[2] as { providerOptions?: unknown }).providerOptions).toEqual({ ...marker });
@@ -187,7 +220,7 @@ describe("applyConversationCacheControl", () => {
providerOptions: { openai: { someKey: "someValue" } },
},
];
- const out = applyConversationCacheControl(messages);
+ const out = applyConversationCacheControl(messages, fiveMinutes);
expect((out[0] as { providerOptions: Record<string, unknown> }).providerOptions).toEqual({
openai: { someKey: "someValue" },
...marker,
@@ -203,7 +236,7 @@ describe("applyConversationCacheControl", () => {
const originalRef0 = messages[0];
const originalRef1 = messages[1];
- const out = applyConversationCacheControl(messages);
+ const out = applyConversationCacheControl(messages, fiveMinutes);
expect(messages).toEqual(snapshot);
expect(messages[0]).toBe(originalRef0);
diff --git a/packages/eve/src/harness/prompt-cache.ts b/packages/eve/src/harness/prompt-cache.ts
index 010046034..059dee6f8 100644
--- a/packages/eve/src/harness/prompt-cache.ts
+++ b/packages/eve/src/harness/prompt-cache.ts
@@ -1,5 +1,7 @@
import type { ModelMessage, SystemModelMessage, ToolSet } from "ai";
+import type { ModelProfile } from "#harness/model-profile.js";
+
/**
* Cache marker for models that take Anthropic prompt-cache breakpoints.
*
@@ -13,18 +15,25 @@ import type { ModelMessage, SystemModelMessage, ToolSet } from "ai";
* Converse provider, which does not understand `anthropic.cacheControl`.
*
* A provider ignores namespaces it does not own, so carrying both is safe on
- * every request regardless of which provider serves it. Every breakpoint shares
- * this frozen instance.
+ * every request regardless of which provider serves it.
+ *
+ * Every breakpoint in a request carries the same TTL, because Anthropic rejects
+ * a 1-hour breakpoint that follows a 5-minute one. The 5-minute marker omits the
+ * TTL, which is the providers' default.
*/
-const ANTHROPIC_CACHE_MARKER = Object.freeze({
- anthropic: Object.freeze({
- cacheControl: Object.freeze({ type: "ephemeral" as const }),
+const ANTHROPIC_CACHE_MARKERS = Object.freeze({
+ "5m": Object.freeze({
+ anthropic: Object.freeze({ cacheControl: Object.freeze({ type: "ephemeral" }) }),
+ bedrock: Object.freeze({ cachePoint: Object.freeze({ type: "default" }) }),
}),
- bedrock: Object.freeze({
- cachePoint: Object.freeze({ type: "default" as const }),
+ "1h": Object.freeze({
+ anthropic: Object.freeze({ cacheControl: Object.freeze({ type: "ephemeral", ttl: "1h" }) }),
+ bedrock: Object.freeze({ cachePoint: Object.freeze({ type: "default", ttl: "1h" }) }),
}),
});
+type AnthropicCache = NonNullable<ModelProfile["anthropicCache"]>;
+
/**
* Returns a new `providerOptions` object with
* `gateway.caching = "auto"` merged into the existing `gateway` sub-object.
@@ -63,7 +72,7 @@ export function mergeGatewayAutoCaching(
* No-op when `tools` has no entries. Preserves existing `providerOptions`
* on tools (merges the cache marker in via spread).
*/
-export function applyLastToolCacheBreakpoint(tools: ToolSet): ToolSet {
+export function applyLastToolCacheBreakpoint(tools: ToolSet, cache: AnthropicCache): ToolSet {
const entries = Object.entries(tools);
if (entries.length === 0) {
return tools;
@@ -81,7 +90,7 @@ export function applyLastToolCacheBreakpoint(tools: ToolSet): ToolSet {
...tool,
providerOptions: {
...existingProviderOptions,
- ...ANTHROPIC_CACHE_MARKER,
+ ...ANTHROPIC_CACHE_MARKERS[cache.ttl],
},
};
} else {
@@ -103,6 +112,7 @@ export function applyLastToolCacheBreakpoint(tools: ToolSet): ToolSet {
*/
export function applySystemCacheBreakpoint(
instructions: readonly SystemModelMessage[],
+ cache: AnthropicCache,
): SystemModelMessage[] {
if (instructions.length === 0) return [...instructions];
@@ -112,7 +122,7 @@ export function applySystemCacheBreakpoint(
...last,
providerOptions: {
...last.providerOptions,
- ...ANTHROPIC_CACHE_MARKER,
+ ...ANTHROPIC_CACHE_MARKERS[cache.ttl],
},
};
return result;
@@ -140,7 +150,10 @@ export function applySystemCacheBreakpoint(
* appends more content blocks than Anthropic's backward boundary scan
* covers.
*/
-export function applyConversationCacheControl(messages: readonly ModelMessage[]): ModelMessage[] {
+export function applyConversationCacheControl(
+ messages: readonly ModelMessage[],
+ cache: AnthropicCache,
+): ModelMessage[] {
if (messages.length === 0) {
return [...messages];
}
@@ -156,7 +169,7 @@ export function applyConversationCacheControl(messages: readonly ModelMessage[])
...message,
providerOptions: {
...message.providerOptions,
- ...ANTHROPIC_CACHE_MARKER,
+ ...ANTHROPIC_CACHE_MARKERS[cache.ttl],
},
};
};
diff --git a/packages/eve/src/harness/provider-safety.test.ts b/packages/eve/src/harness/provider-safety.test.ts
index 8c4a57e2c..70771cc2e 100644
--- a/packages/eve/src/harness/provider-safety.test.ts
+++ b/packages/eve/src/harness/provider-safety.test.ts
@@ -101,7 +101,7 @@ describe("mergeProviderSafetyIdentifier", () => {
describe("resolveCallProviderOptions", () => {
const openai: ModelProfile = {
- anthropicCache: false,
+ anthropicCache: undefined,
filesOutsideToolResults: false,
gateway: false,
googleSearchDropsTools: false,
diff --git a/packages/eve/src/harness/step-hooks.ts b/packages/eve/src/harness/step-hooks.ts
index 8d8e34e53..f5ced490d 100644
--- a/packages/eve/src/harness/step-hooks.ts
+++ b/packages/eve/src/harness/step-hooks.ts
@@ -164,7 +164,9 @@ export function buildStepHooks(input: StepHooksInput): StepHooks {
const prepareStep: PrepareStepFunction<ToolSet> = async ({ messages }) => {
const { profile } = input;
const stepResult: NonNullable<Awaited<ReturnType<PrepareStepFunction<ToolSet>>>> = {
- messages: profile.anthropicCache ? applyConversationCacheControl(messages) : messages,
+ messages: profile.anthropicCache
+ ? applyConversationCacheControl(messages, profile.anthropicCache)
+ : messages,
};
const modelReference = requireSessionModelReference(session);
diff --git a/packages/eve/src/harness/tool-loop.test.ts b/packages/eve/src/harness/tool-loop.test.ts
index d8b2b8c46..3c211323a 100644
--- a/packages/eve/src/harness/tool-loop.test.ts
+++ b/packages/eve/src/harness/tool-loop.test.ts
@@ -9112,6 +9112,64 @@ describe("createToolLoopHarness", () => {
});
});
+ it("declared Anthropic cache: marks an opaque Bedrock profile with the authored TTL", async () => {
+ setupStopResult();
+ const config: ToolLoopHarnessConfig = {
+ resolveModel: vi.fn().mockResolvedValue({
+ provider: "amazon-bedrock",
+ modelId: "arn:aws:bedrock:us-east-1:123456789012:application-inference-profile/abc",
+ specificationVersion: "v3",
+ } as unknown as LanguageModel),
+ tools: new Map([
+ [
+ "add",
+ {
+ description: "Adds numbers",
+ execute: vi.fn(),
+ inputSchema: jsonSchema({ type: "object" }),
+ name: "add",
+ },
+ ],
+ ]),
+ };
+ const runStep = createToolLoopHarness(config);
+ const session = createTestSession();
+ await runStep(
+ {
+ ...session,
+ agent: {
+ ...session.agent,
+ modelReference: {
+ id: "bedrock-profile",
+ promptCache: { anthropic: { ttl: "1h" } },
+ },
+ },
+ },
+ { message: "c" },
+ );
+
+ const agentCall = vi.mocked(ToolLoopAgent).mock.calls[0]?.[0];
+ const prepareStep = getPrepareStep<
+ Array<{ role: string; content: string; providerOptions?: unknown }>,
+ { messages?: Array<{ providerOptions?: unknown }> }
+ >(agentCall?.prepareStep);
+ const result = await prepareStep({
+ messages: [{ role: "user", content: "c" }],
+ stepNumber: 0,
+ steps: [],
+ model: agentCall?.model,
+ context: undefined,
+ });
+ const hourMarker = {
+ anthropic: { cacheControl: { type: "ephemeral", ttl: "1h" } },
+ bedrock: { cachePoint: { type: "default", ttl: "1h" } },
+ };
+
+ expect(result.messages?.[0]?.providerOptions).toEqual(hourMarker);
+ const tools = agentCall?.tools as Record<string, { providerOptions?: unknown }>;
+ expect(Object.values(tools).at(-1)?.providerOptions).toEqual(hourMarker);
+ });
+
it("none path: direct OpenAI instance gets only a prompt cache key", async () => {
setupStopResult();
const config: ToolLoopHarnessConfig = {
diff --git a/packages/eve/src/internal/authored-definition/core.test.ts b/packages/eve/src/internal/authored-definition/core.test.ts
index 24cd6f5ef..cad09575a 100644
--- a/packages/eve/src/internal/authored-definition/core.test.ts
+++ b/packages/eve/src/internal/authored-definition/core.test.ts
@@ -99,6 +99,24 @@ describe("normalizeAgentDefinition", () => {
).toThrow('"compaction.model" does not support defineDynamic');
});
+ it("accepts Anthropic prompt cache options", () => {
+ const definition = normalizeAgentDefinition(
+ { model: "openai/gpt-5.5", modelOptions: { promptCache: { anthropic: { ttl: "1h" } } } },
+ FAILURE_MESSAGE,
+ );
+
+ expect(definition.modelOptions).toEqual({ promptCache: { anthropic: { ttl: "1h" } } });
+ });
+
+ it("rejects an unsupported Anthropic prompt cache TTL", () => {
+ expect(() =>
+ normalizeAgentDefinition(
+ { model: "openai/gpt-5.5", modelOptions: { promptCache: { anthropic: { ttl: "24h" } } } },
+ FAILURE_MESSAGE,
+ ),
+ ).toThrow('"modelOptions.promptCache.anthropic.ttl" must be "5m" or "1h"; received "24h".');
+ });
+
it("rejects unsupported reasoning effort", () => {
expect(() =>
normalizeAgentDefinition(
diff --git a/packages/eve/src/internal/authored-definition/core.ts b/packages/eve/src/internal/authored-definition/core.ts
index abfb86db6..e69cd6c70 100644
--- a/packages/eve/src/internal/authored-definition/core.ts
+++ b/packages/eve/src/internal/authored-definition/core.ts
@@ -19,6 +19,9 @@ import {
} from "#internal/authored-module.js";
import {
AGENT_WORKFLOW_RETENTION_VALUES,
+ ANTHROPIC_PROMPT_CACHE_TTLS,
+ type AgentPromptCacheDefinition,
+ type AnthropicPromptCacheTtl,
type PublicAgentStaticModelDefinition,
} from "#shared/agent-definition.js";
import {
@@ -322,16 +325,45 @@ function normalizeAgentModelOptions(
message: string,
): NonNullable<NormalizedAgentDefinition["modelOptions"]> {
const record = expectObjectRecord(value, message);
- expectOnlyKnownKeys(record, ["providerOptions"], message);
- const providerOptions = record.providerOptions;
+ expectOnlyKnownKeys(record, ["promptCache", "providerOptions"], message);
+ const options: Mutable<NonNullable<NormalizedAgentDefinition["modelOptions"]>> = {};
- if (providerOptions === undefined) {
- return {};
+ if (record.providerOptions !== undefined) {
+ options.providerOptions = expectProviderOptions(record.providerOptions, message);
}
- return {
- providerOptions: expectProviderOptions(providerOptions, message),
- };
+ if (record.promptCache !== undefined) {
+ options.promptCache = normalizeAgentPromptCache(record.promptCache, message);
+ }
+
+ return options;
+}
+
+/** Validates an authored `modelOptions.promptCache` value. */
+export function normalizeAgentPromptCache(
+ value: unknown,
+ message: string,
+): AgentPromptCacheDefinition {
+ const record = expectObjectRecord(
+ value,
+ `${message} "modelOptions.promptCache" must be an object.`,
+ );
+ expectOnlyKnownKeys(record, ["anthropic"], `${message} In "modelOptions.promptCache":`);
+ if (record.anthropic === undefined) return {};
+
+ const anthropic = expectObjectRecord(
+ record.anthropic,
+ `${message} "modelOptions.promptCache.anthropic" must be an object.`,
+ );
+ expectOnlyKnownKeys(anthropic, ["ttl"], `${message} In "modelOptions.promptCache.anthropic":`);
+ const ttl = anthropic.ttl;
+ if (ttl === undefined) return { anthropic: {} };
+ if (!ANTHROPIC_PROMPT_CACHE_TTLS.includes(ttl as AnthropicPromptCacheTtl)) {
+ throw new Error(
+ `${message} "modelOptions.promptCache.anthropic.ttl" must be "5m" or "1h"; received ${JSON.stringify(ttl)}.`,
+ );
+ }
+ return { anthropic: { ttl: ttl as AnthropicPromptCacheTtl } };
}
function normalizeAgentCompactionDefinition(
diff --git a/packages/eve/src/runtime/agent/resolve-model.test.ts b/packages/eve/src/runtime/agent/resolve-model.test.ts
index 9e14674d0..84be2dc0d 100644
--- a/packages/eve/src/runtime/agent/resolve-model.test.ts
+++ b/packages/eve/src/runtime/agent/resolve-model.test.ts
@@ -47,6 +47,36 @@ describe("dynamic runtime model resolution", () => {
expect(catalog.getByGatewayId).not.toHaveBeenCalled();
});
+ it("carries prompt cache options on a direct provider selection", async () => {
+ const result = await resolveRuntimeModelSelection({
+ selection: {
+ model: createLanguageModel("amazon-bedrock", "application-inference-profile/abc"),
+ modelContextWindowTokens: 200_000,
+ modelOptions: { promptCache: { anthropic: { ttl: "1h" } } },
+ },
+ durability: "live",
+ state: new ContextContainer(),
+ });
+
+ expect(result.reference.promptCache).toEqual({ anthropic: { ttl: "1h" } });
+ });
+
+ it("rejects prompt cache options on an AI Gateway selection", async () => {
+ const catalog = createCatalog();
+ await expect(
+ resolveRuntimeModelSelection({
+ catalog,
+ selection: {
+ model: "anthropic/claude-opus-5",
+ modelOptions: { promptCache: { anthropic: {} } },
+ },
+ durability: "durable",
+ state: new ContextContainer(),
+ }),
+ ).rejects.toThrow('modelOptions.promptCache for "anthropic/claude-opus-5"');
+ expect(catalog.getByGatewayId).not.toHaveBeenCalled();
+ });
+
afterEach(() => {
vi.restoreAllMocks();
vi.unstubAllEnvs();
diff --git a/packages/eve/src/runtime/agent/resolve-model.ts b/packages/eve/src/runtime/agent/resolve-model.ts
index 356eb0a92..c3b4e5afa 100644
--- a/packages/eve/src/runtime/agent/resolve-model.ts
+++ b/packages/eve/src/runtime/agent/resolve-model.ts
@@ -3,7 +3,10 @@ import type { LanguageModel } from "ai";
import type { CompiledModuleMap } from "#compiler/module-map.js";
import type { ContextAccessor } from "#context/key.js";
import { RuntimeModelMetadataCacheKey, type CachedModelMetadata } from "#context/keys.js";
-import { normalizeAgentDefinition } from "#internal/authored-definition/core.js";
+import {
+ normalizeAgentDefinition,
+ normalizeAgentPromptCache,
+} from "#internal/authored-definition/core.js";
import {
formatLanguageModelGatewayId,
isRuntimeLanguageModel,
@@ -164,11 +167,19 @@ export async function resolveRuntimeModelSelection(input: {
selection.modelOptions?.providerOptions === undefined
? undefined
: parseProviderOptionsRecord(selection.modelOptions.providerOptions);
+ const promptCache =
+ selection.modelOptions?.promptCache === undefined
+ ? undefined
+ : normalizeAgentPromptCache(
+ selection.modelOptions.promptCache,
+ "Dynamic model resolver returned an invalid selection.",
+ );
const selectedModel = selection.model;
const catalog = input.catalog ?? runtimeModelCatalogForState(input.state);
if (typeof selectedModel === "string") {
const id = formatLanguageModelGatewayId(selectedModel);
+ if (promptCache !== undefined) throw gatewayPromptCacheError(id);
const metadata = await resolveSelectionMetadata({
cacheKey: `gateway:${id}`,
catalog,
@@ -197,6 +208,9 @@ export async function resolveRuntimeModelSelection(input: {
const formattedId = formatLanguageModelGatewayId(selectedModel);
const topLevelProvider = selectedModel.provider.split(".")[0]!;
+ if (promptCache !== undefined && topLevelProvider === "gateway") {
+ throw gatewayPromptCacheError(formattedId);
+ }
const metadata = await resolveSelectionMetadata({
cacheKey:
topLevelProvider === "gateway"
@@ -218,12 +232,19 @@ export async function resolveRuntimeModelSelection(input: {
id: metadata.resolvedModelId,
contextWindowTokens: metadata.contextWindowTokens,
maxOutputTokens: metadata.maxOutputTokens,
+ promptCache,
providerOptions,
reasoning: selection.reasoning,
},
};
}
+function gatewayPromptCacheError(modelId: string): Error {
+ return new Error(
+ `Dynamic model resolver returned modelOptions.promptCache for "${modelId}", which routes through AI Gateway. promptCache applies only to models eve calls directly; Gateway manages its own prompt caching through providerOptions.gateway.caching.`,
+ );
+}
+
const runtimeModelCatalogsByState = new WeakMap<ContextAccessor, RuntimeModelCatalog>();
function runtimeModelCatalogForState(state: ContextAccessor): RuntimeModelCatalog {
diff --git a/packages/eve/src/runtime/resolve-agent.ts b/packages/eve/src/runtime/resolve-agent.ts
index 12c60e535..2be44552a 100644
--- a/packages/eve/src/runtime/resolve-agent.ts
+++ b/packages/eve/src/runtime/resolve-agent.ts
@@ -294,6 +294,7 @@ function createResolvedAgentConfig(
contextWindowTokens: model.contextWindowTokens,
id: model.id,
maxOutputTokens: model.maxOutputTokens,
+ promptCache: model.promptCache,
providerOptions: model.providerOptions,
source: {
exportName: model.source.exportName,
diff --git a/packages/eve/src/shared/agent-definition.ts b/packages/eve/src/shared/agent-definition.ts
index 0731c7397..fbe2406b6 100644
--- a/packages/eve/src/shared/agent-definition.ts
+++ b/packages/eve/src/shared/agent-definition.ts
@@ -13,8 +13,36 @@ import {
*/
export interface AgentModelOptionsDefinition {
readonly providerOptions?: Record<string, JsonObject>;
+ /** Prompt-cache breakpoints eve places for this model. */
+ readonly promptCache?: AgentPromptCacheDefinition;
}
+/**
+ * Prompt-cache breakpoints eve places for a model it calls directly. AI Gateway models cache
+ * through `providerOptions.gateway.caching` instead and reject this option.
+ */
+export interface AgentPromptCacheDefinition {
+ /**
+ * Places Anthropic cache breakpoints on every call. eve already does this for Anthropic models
+ * it recognizes by provider or model id; set it for one it can't, such as a Bedrock application
+ * inference profile, or to change the cache lifetime.
+ */
+ readonly anthropic?: AgentAnthropicPromptCacheDefinition;
+}
+
+export interface AgentAnthropicPromptCacheDefinition {
+ /**
+ * How long each breakpoint's cache entry lives: `"5m"` (the default) or `"1h"`. A 1-hour write
+ * costs more than a 5-minute one, so `"1h"` pays off when turns often land between five minutes
+ * and an hour apart. Only some Claude models accept a 1-hour TTL.
+ */
+ readonly ttl?: AnthropicPromptCacheTtl;
+}
+
+export type AnthropicPromptCacheTtl = "5m" | "1h";
+
+export const ANTHROPIC_PROMPT_CACHE_TTLS: readonly AnthropicPromptCacheTtl[] = ["5m", "1h"];
+
/**
* Provider-agnostic reasoning effort forwarded to the AI SDK model call.
*/
@@ -48,6 +76,7 @@ export type InternalAgentModelDefinition = {
maxOutputTokens?: number;
source?: ModuleSourceRef;
providerOptions?: Record<string, JsonObject>;
+ promptCache?: AgentPromptCacheDefinition;
};
/**