Skip to content

A fast 2D and 3D game engine — TypeScript SDK, C++/WebAssembly core, visual editor. Ship one project to web, desktop, WeChat MiniGames, playable ads, and native Android / iOS.

feat(agent): your providers are a list, and each says what it can do

yhh8ef59ed8caee
diff --git a/desktop/electron/agent/anthropic.ts b/desktop/electron/agent/anthropic.ts
index f757b5a54..2a507b2c1 100644
--- a/desktop/electron/agent/anthropic.ts
+++ b/desktop/electron/agent/anthropic.ts
@@ -48,6 +48,16 @@ export interface AnthropicOptions {
/** Point at an Anthropic-compatible gateway; omit for the API. Its presence
* is what drops the request to the core wire format — see the file header. */
baseURL?: string;
+ /**
+ * Whether this endpoint's models can be handed an image.
+ *
+ * A separate fact from the dialect, though the address once stood in for
+ * both. The dialect is about which EXTENSIONS the request may carry; image
+ * blocks are in the core format, so a gateway may well take them — and a
+ * gateway that does had no way to say so. Omitted falls back to the guess
+ * that fact replaced, for a caller with nothing to declare.
+ */
+ vision?: boolean;
/** How far the conversation may grow before the oldest runs are folded away.
* The window knows which provider this is, so it is the side that says. */
contextWindow?: number;
@@ -127,15 +137,16 @@ export function buildStepRequest(opts: {
* happened: a model handed a silently dropped screenshot concludes the editor is
* broken, while one told the endpoint cannot carry it asks a different way.
*
- * Exported because the native branch cannot be exercised against a local
- * stand-in gateway — pointing at one is exactly what selects the other dialect.
+ * Keyed on `vision` rather than on the dialect, which it once shared: image
+ * blocks are core format, so a gateway that takes them is an ordinary case and
+ * not an exception to be spotted from the address.
*/
export function toolResultContent(
outcome: ToolOutcome,
- dialect: Dialect,
+ vision: boolean,
): Anthropic.Beta.BetaToolResultBlockParam['content'] {
if (!outcome.image) return outcome.content;
- if (dialect !== 'anthropic') {
+ if (!vision) {
return `${outcome.content}\n[the screenshot could not be sent: this endpoint does not accept images]`;
}
return [
@@ -179,7 +190,9 @@ class AnthropicSession implements AgentSession {
constructor(
private readonly client: Anthropic,
- private readonly opts: { model: string; effort: string; dialect: Dialect; contextWindow: number },
+ private readonly opts: {
+ model: string; effort: string; dialect: Dialect; vision: boolean; contextWindow: number;
+ },
private readonly system: string,
private readonly tools: readonly CatalogTool[],
) {
@@ -226,7 +239,7 @@ class AnthropicSession implements AgentSession {
// than quietly handed the text alone: a model that never learns a picture
// was attached answers confidently about something it was not shown, and
// the person watching has every reason to think it looked.
- if (this.opts.dialect !== 'anthropic') {
+ if (!this.opts.vision) {
const note = images.length === 1
? '[the user attached an image; this endpoint cannot receive images, so you have not seen it]'
: `[the user attached ${images.length} images; this endpoint cannot receive images, so you have not seen them]`;
@@ -269,7 +282,7 @@ class AnthropicSession implements AgentSession {
content: outcomes.map((o) => ({
type: 'tool_result' as const,
tool_use_id: o.id,
- content: toolResultContent(o, this.opts.dialect),
+ content: toolResultContent(o, this.opts.vision),
is_error: o.isError,
})),
});
@@ -468,16 +481,20 @@ export function createAnthropicProvider(options: AnthropicOptions): AgentProvide
const effort = options.effort ?? DEFAULT_EFFORT;
const baseURL = options.baseURL || undefined;
const dialect: Dialect = baseURL ? 'compatible' : 'anthropic';
+ // Declared by the provider that was picked. The fallback is the guess this
+ // replaced — right for the API, and the safe way to be wrong for a caller
+ // that says nothing, since a claimed sight an endpoint lacks costs the turn.
+ const vision = options.vision ?? dialect === 'anthropic';
const client = new Anthropic({ apiKey: options.apiKey, baseURL });
return {
id: dialect === 'anthropic' ? 'anthropic' : `compatible:${baseURL ?? ''}`,
model,
- // The same condition toolResultContent substitutes on — one fact, one place.
- acceptsImages: dialect === 'anthropic',
+ // The same flag toolResultContent substitutes on — one fact, one place.
+ acceptsImages: vision,
createSession: ({ system, tools }, memory) => {
const session = new AnthropicSession(
client,
- { model, effort, dialect, contextWindow: options.contextWindow || DEFAULT_CONTEXT_WINDOW },
+ { model, effort, dialect, vision, contextWindow: options.contextWindow || DEFAULT_CONTEXT_WINDOW },
system,
tools,
);
diff --git a/desktop/electron/agent/openai.ts b/desktop/electron/agent/openai.ts
index c6d4e8e3d..a8d0bd927 100644
--- a/desktop/electron/agent/openai.ts
+++ b/desktop/electron/agent/openai.ts
@@ -58,6 +58,10 @@ export interface OpenAIOptions {
* Whether this endpoint's model can see. Declared rather than probed: finding
* out costs a call spent on a screenshot the model will be told it cannot
* look at, and the answer is a property of the endpoint, not of the editor.
+ *
+ * The editor always says (src/agent/providers.ts declares it per provider);
+ * omitted falls back to the guess that used to be the only answer — the
+ * address — for a caller with nothing to declare.
*/
vision?: boolean;
/**
diff --git a/desktop/electron/main.ts b/desktop/electron/main.ts
index d02936227..0c221514b 100644
--- a/desktop/electron/main.ts
+++ b/desktop/electron/main.ts
@@ -690,7 +690,7 @@ ipcMain.handle('secret:clear', (_e, id: string) => clearSecret(id));
// settings live in its localStorage) and merged, since the two rows fire
// separately; read at session creation, so a change lands on the next
// conversation rather than under the one being read.
-let agentEndpoint: { protocol?: string; baseUrl?: string; model?: string; keyId?: string; contextWindow?: number; effort?: string } = {};
+let agentEndpoint: { protocol?: string; baseUrl?: string; model?: string; keyId?: string; contextWindow?: number; effort?: string; vision?: boolean; reasoningEffort?: boolean } = {};
/** Contributed agent tools, as the window last reported them. */
let agentPluginTools: ContributedTool[] = [];
@@ -718,6 +718,11 @@ const agentHost = createAgentHost({
model: agentEndpoint.model,
effort,
contextWindow: agentEndpoint.contextWindow,
+ vision: agentEndpoint.vision,
+ // An endpoint that has never heard of `reasoning_effort` refuses the
+ // whole request over it, so whether to name the depth is the provider's
+ // to declare — the same journey as the window and the sight.
+ reasoningEffort: agentEndpoint.reasoningEffort,
});
}
return createAnthropicProvider({
@@ -726,6 +731,10 @@ const agentHost = createAgentHost({
model: agentEndpoint.model,
effort,
contextWindow: agentEndpoint.contextWindow,
+ // What the provider declared, not what its address implies. The window is
+ // the side that knows which provider was picked, so it is the side that
+ // says — for this the same as for the model and the window size.
+ vision: agentEndpoint.vision,
});
},
// Conversations are kept with the project they are about. With none open
@@ -761,7 +770,7 @@ ipcMain.handle('agent:resumeConversation', async (_e, id: string) => {
ipcMain.handle('agent:deleteConversation', async (_e, id: string) => {
if (projectRoot) await deleteConversation(projectRoot, id);
});
-ipcMain.handle('agent:setEndpoint', (_e, patch: { protocol?: string; baseUrl?: string; model?: string; keyId?: string; contextWindow?: number; effort?: string }) => {
+ipcMain.handle('agent:setEndpoint', (_e, patch: { protocol?: string; baseUrl?: string; model?: string; keyId?: string; contextWindow?: number; effort?: string; vision?: boolean; reasoningEffort?: boolean }) => {
agentEndpoint = { ...agentEndpoint, ...patch };
// `ready` is derived from which key this points at, so it just changed.
agentHost.announce();
diff --git a/desktop/electron/preload.ts b/desktop/electron/preload.ts
index 568a62963..4d10c3851 100644
--- a/desktop/electron/preload.ts
+++ b/desktop/electron/preload.ts
@@ -139,7 +139,7 @@ const api = {
ipcRenderer.invoke('agent:deleteConversation', id),
/** Point the NEXT session at an endpoint/model. Merged, so each settings row
* can push its own field. */
- setEndpoint: (patch: { protocol?: string; baseUrl?: string; model?: string; keyId?: string; contextWindow?: number; effort?: string }): Promise<void> =>
+ setEndpoint: (patch: { protocol?: string; baseUrl?: string; model?: string; keyId?: string; contextWindow?: number; effort?: string; vision?: boolean; reasoningEffort?: boolean }): Promise<void> =>
ipcRenderer.invoke('agent:setEndpoint', patch),
/** Report what loaded plugins contributed for the agent to call. Metadata
* only — the handlers stay in this window, reached through the editor
diff --git a/desktop/src/agent/providers.ts b/desktop/src/agent/providers.ts
index b31f515b4..40323a594 100644
--- a/desktop/src/agent/providers.ts
+++ b/desktop/src/agent/providers.ts
@@ -56,6 +56,21 @@ export interface AgentProviderDef {
* high is what makes a turn fail outright.
*/
contextWindow?: number;
+ /**
+ * Whether a rendered frame can reach this endpoint's models at all.
+ *
+ * DECLARED, never inferred. It was inferred once — "no baseUrl, therefore the
+ * vendor's own API, therefore it can see" — which is only a statement about
+ * the address, and it made every gateway blind with no way to say otherwise.
+ *
+ * Per provider rather than per model, for {@link contextWindow}'s reason: a
+ * table with a row per model goes stale a model at a time. Absent means NO,
+ * deliberately — the two mistakes are not symmetrical. Claiming sight an
+ * endpoint lacks spends the turn on a request it refuses; withholding it costs
+ * `screenshot`'s coarse text grid instead of a picture, which still answers
+ * "did anything draw at all".
+ */
+ vision?: boolean;
/**
* The model list is TYPED, not shipped.
*
@@ -66,8 +81,24 @@ export interface AgentProviderDef {
* shipped, including its key row.
*/
typedModels?: boolean;
- /** Endpoint, protocol AND models all come from settings — see
- * {@link CUSTOM_PROVIDER}, the one provider we cannot ship anything for. */
+ /**
+ * Send the reasoning-depth parameter with each request.
+ *
+ * Only the OpenAI protocol carries it as a named argument (`reasoning_effort`),
+ * and an endpoint that has never heard of one refuses the whole request rather
+ * than ignoring the field — so for a gateway this has to be a switch, not an
+ * assumption. Absent means yes, which is right for every endpoint that
+ * implements the parameter and is the behaviour that predates the field.
+ */
+ reasoningEffort?: boolean;
+ /**
+ * This def came from the person's own table rather than from this file.
+ *
+ * Every OTHER field means the same thing whichever way it arrived — that is
+ * the point of projecting the table into this registry instead of keeping it
+ * beside it. This one exists so a reader that must not offer to edit a shipped
+ * provider can tell the two apart.
+ */
userDefined?: boolean;
}
@@ -76,8 +107,18 @@ export { DEFAULT_CONTEXT_WINDOW } from '@/settings/agentIds';
/** The id a provider's key is stored under (electron/secrets.ts). */
export const agentKeyId = (providerId: string): string => `agents.key.${providerId}`;
+/**
+ * The id the ONE custom provider had, back when there could only be one.
+ *
+ * Kept because a key is filed under its provider's id: the row that setup
+ * becomes on upgrade has to carry this id, or the credential already sealed on
+ * the machine belongs to a provider that no longer exists (agent/userProviders.ts).
+ */
export const CUSTOM_PROVIDER = 'custom';
+/** Who owns the providers the person typed — see {@link setUserProviders}. */
+export const USER_OWNER = 'user';
+
/** A provider whose models the user types. Its list is per provider, so two of
* them do not share one box. */
export const modelsSettingId = (providerId: string): string => `agents.models.${providerId}`;
@@ -91,6 +132,7 @@ registry.registerAll('core', [
baseUrl: '',
models: ['claude-opus-5', 'claude-sonnet-5', 'claude-haiku-4-5'],
contextWindow: 200_000,
+ vision: true,
},
{
id: 'deepseek',
@@ -107,16 +149,11 @@ registry.registerAll('core', [
models: [],
typedModels: true,
contextWindow: 400_000,
- },
- // The escape hatch. Its endpoint, protocol and model list come from settings,
- // because a provider we have not heard of is exactly the one we cannot ship
- // a list for — and Chat Completions is what most of those speak.
- {
- id: CUSTOM_PROVIDER,
- label: '',
- baseUrl: '',
- models: [],
- userDefined: true,
+ // Its models are typed, so this is a claim about the ENDPOINT: it carries
+ // image parts. A text-only model named into that box is the same staleness
+ // `contextWindow` accepts, and it degrades the same way — the model is told
+ // a picture it cannot see was attached.
+ vision: true,
},
]);
@@ -131,6 +168,24 @@ export const providersRevision = (): number => registry.getRevision();
export const registerAgentProvider = (def: AgentProviderDef, owner: Owner = 'core'): Disposable =>
registry.register(owner, def);
+/**
+ * Replace the set of providers the person defined themselves.
+ *
+ * They arrive as a SET rather than one at a time because that is how they are
+ * edited — a table, rewritten whole on each keystroke — and because the removals
+ * matter as much as the additions: a provider deleted from the table must stop
+ * being offered in the picker, and dispose-then-register is the only way to say
+ * that without diffing. Everything downstream reads them through the same door
+ * as the shipped ones; nothing but this function knows they were typed.
+ *
+ * Kept here, taking finished defs, so this module stays free of the settings
+ * store — the projection from one to the other lives in agent/userProviders.ts.
+ */
+export function setUserProviders(defs: readonly AgentProviderDef[]): void {
+ registry.disposeOwner(USER_OWNER);
+ registry.registerAll(USER_OWNER, defs.map((d) => ({ ...d, userDefined: true })));
+}
+
export const AGENT_PROTOCOLS: readonly AgentProtocol[] = ['openai', 'anthropic'];
/** What an endpoint speaks, defaulting to the format that predates the field. */
diff --git a/desktop/src/agent/userProviders.ts b/desktop/src/agent/userProviders.ts
new file mode 100644
index 000000000..308751493
--- /dev/null
+++ b/desktop/src/agent/userProviders.ts
@@ -0,0 +1,187 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file userProviders.ts — the providers a person defined, projected into the
+ * same registry the shipped ones live in.
+ *
+ * There used to be exactly one, spelled out as four global settings
+ * (`agents.customProtocol`, `…BaseUrl`, `…Models`, plus one key). That
+ * contradicted the rule the shipped providers are built on — the unit of
+ * configuration is a PROVIDER, and every one keeps its own key so switching back
+ * to the one you used last week costs nothing. A local runner and a company
+ * gateway could not both exist: configuring the second overwrote the first,
+ * credential included.
+ *
+ * So they are a LIST now, and each row is a provider like any other: address,
+ * protocol, models, its own key, and what it can do. This module is the only
+ * place that knows a provider can come from a table — providers.ts stays a
+ * registry, and every reader downstream sees one kind of provider.
+ */
+import { useSettings, dropPersisted } from '@/store/settingsStore';
+import {
+ type AgentProviderDef, agentKeyId, asProtocol, parseModelList, setUserProviders,
+ CUSTOM_PROVIDER,
+} from '@/agent/providers';
+import { forgetSecret } from '@/store/SecretStore';
+import { DEFAULT_CONTEXT_WINDOW } from '@/settings/agentIds';
+import { t } from '@/i18n';
+
+export const AGENT_PROVIDERS_SETTING = 'agents.providers';
+
+/**
+ * One row of that table, as it is stored.
+ *
+ * Flat and all-scalar because it IS the edited shape — the settings table writes
+ * these fields directly, so a nested or derived form here would be a translation
+ * layer between the control and the file. `models` is the raw text the person
+ * typed for the same reason: splitting it on save would make the field fight the
+ * cursor halfway through a comma.
+ */
+export interface UserProviderRow {
+ /** Stable for the row's life — the key is filed under it. Never re-used. */
+ id: string;
+ label: string;
+ protocol: string;
+ baseUrl: string;
+ models: string;
+ vision: boolean;
+ /** Left blank on a new row, and blank means {@link DEFAULT_CONTEXT_WINDOW} —
+ * a zero typed into the cell reads the same way. */
+ contextWindow?: number;
+ reasoningEffort: boolean;
+}
+
+const str = (v: unknown): string => (typeof v === 'string' ? v : '');
+
+/**
+ * What to call a provider whose name was left blank.
+ *
+ * The picker groups models under this, so an empty string there is a heading of
+ * nothing above a list of models with no way to tell whose they are. The host is
+ * the better guess than a generic word: someone running two local models knows
+ * their ports.
+ */
+function fallbackLabel(baseUrl: string): string {
+ try {
+ return new URL(baseUrl).host || t('agent.picker.custom');
+ } catch {
+ return t('agent.picker.custom');
+ }
+}
+
+/**
+ * A stored row as the rest of the editor sees it, or null for one that cannot be
+ * used yet — a row still being typed is an ordinary state, not an error, and the
+ * table says what is missing (see the setting's `rowError`).
+ */
+export function rowToDef(row: Record<string, unknown>): AgentProviderDef | null {
+ const id = str(row.id).trim();
+ if (!id) return null;
+ const baseUrl = str(row.baseUrl).trim();
+ const contextWindow = Number(row.contextWindow);
+ return {
+ id,
+ label: str(row.label).trim() || fallbackLabel(baseUrl),
+ protocol: asProtocol(row.protocol),
+ baseUrl,
+ models: parseModelList(str(row.models)),
+ vision: row.vision === true,
+ // Absent means yes for a shipped provider, so only an explicit `false` from
+ // the table turns it off — a row written before the field existed must not
+ // read as "this endpoint refuses the parameter".
+ reasoningEffort: row.reasoningEffort !== false,
+ contextWindow: contextWindow > 0 ? contextWindow : DEFAULT_CONTEXT_WINDOW,
+ userDefined: true,
+ };
+}
+
+/** The table, as stored. */
+export const userProviderRows = (): Record<string, unknown>[] =>
+ useSettings.getState().getValue<Record<string, unknown>[]>(AGENT_PROVIDERS_SETTING) ?? [];
+
+/** Project the table into the provider registry. Called on every edit to it. */
+export const syncUserProviders = (): void => {
+ const defs: AgentProviderDef[] = [];
+ for (const row of userProviderRows()) {
+ const def = rowToDef(row);
+ if (def) defs.push(def);
+ }
+ setUserProviders(defs);
+};
+
+/**
+ * A row for a provider being added, with an id no earlier row has held.
+ *
+ * Monotonic rather than "first free": ids that come back around would inherit
+ * whatever is still filed under them, and a key belonging to a provider that was
+ * deleted is not one to hand to its replacement. {@link forgetProviderSecret}
+ * closes the same hole from the other side.
+ */
+export function newProviderRow(rows: readonly Record<string, unknown>[]): UserProviderRow {
+ let next = 1;
+ for (const row of rows) {
+ const m = /^custom-(\d+)$/.exec(str(row.id));
+ if (m) next = Math.max(next, Number(m[1]) + 1);
+ }
+ return {
+ id: `custom-${next}`,
+ label: '',
+ // What most gateways and every local runner implement. Getting this wrong is
+ // refused rather than degraded, so the likelier answer is the better default.
+ protocol: 'openai',
+ baseUrl: '',
+ models: '',
+ vision: false,
+ // Left out rather than zeroed: the cell then shows the default it would use
+ // as its placeholder, instead of a 0 that reads like a window of nothing.
+ reasoningEffort: true,
+ };
+}
+
+/** The legacy singleton's settings, folded into the table on first run. */
+const LEGACY_IDS = [
+ 'agents.customProtocol', 'agents.customBaseUrl', 'agents.customModels', 'agents.customVision',
+] as const;
+
+/**
+ * Carry a pre-table setup across, once.
+ *
+ * The row takes {@link CUSTOM_PROVIDER} as its id on purpose: a key is filed
+ * under its provider's id, and this is the one migration step that cannot be
+ * repaired by hand — main never hands a sealed credential back, so an id that
+ * did not carry over is a key the person has to find again. The saved model pick
+ * (`estella.agent.selection`) names the same id and so keeps resolving too.
+ *
+ * Runs only while the table has never been written. A person who has since
+ * deleted every row means it, and re-adding what they removed on each boot is
+ * the worst way for a migration to be wrong.
+ */
+export function migrateLegacyCustomProvider(): void {
+ const { values } = useSettings.getState();
+ if (values[AGENT_PROVIDERS_SETTING] !== undefined) return;
+ const baseUrl = str(values['agents.customBaseUrl']).trim();
+ const models = str(values['agents.customModels']).trim();
+ if (!baseUrl && !models) {
+ // Nothing was configured, but the legacy ids may still sit in the file as
+ // defaults someone toggled and toggled back.
+ dropPersisted(LEGACY_IDS);
+ return;
+ }
+ const migrated: UserProviderRow = {
+ id: CUSTOM_PROVIDER,
+ label: t('agent.picker.custom'),
+ protocol: asProtocol(values['agents.customProtocol']),
+ baseUrl,
+ models,
+ vision: values['agents.customVision'] === true,
+ reasoningEffort: true,
+ };
+ useSettings.getState().setValue(AGENT_PROVIDERS_SETTING, [migrated as unknown as Record<string, unknown>]);
+ dropPersisted(LEGACY_IDS);
+}
+
+/** Let go of the credential a removed provider held — see {@link newProviderRow}. */
+export function forgetProviderSecret(row: Record<string, unknown>): void {
+ const id = str(row.id).trim();
+ if (id) void forgetSecret(agentKeyId(id));
+}
diff --git a/desktop/src/components/AgentDrawer.tsx b/desktop/src/components/AgentDrawer.tsx
index 2629e1f03..0872b7f27 100644
--- a/desktop/src/components/AgentDrawer.tsx
+++ b/desktop/src/components/AgentDrawer.tsx
@@ -26,7 +26,7 @@
import { memo, useCallback, useEffect, useLayoutEffect, useRef, useState, useSyncExternalStore } from 'react';
import {
X, Plus, PanelRight, ArrowUp, Square, ChevronRight, ChevronDown, Check, TriangleAlert,
- Loader, Copy, Pencil, Eye, KeyRound, Boxes, Stethoscope, Image as ImageIcon, RotateCcw,
+ Loader, Copy, Pencil, Eye, KeyRound, Boxes, Stethoscope, Image as ImageIcon, ImageOff, RotateCcw,
File as FileIcon, ArrowRight, History as HistoryIcon, Trash2, FoldVertical,
} from 'lucide-react';
import { useEditorStore } from '@/store/editorStore';
@@ -34,13 +34,13 @@ import {
useAgent, agentEffort, sendAgentMessage, stopAgentTurn, confirmAgentCall, startNewConversation,
peekEntities, entitiesInInput, effectiveSelection, selectAgentModel, retryAgentTurn, setAgentDraft,
RESUMABLE, openAgentHistory, closeAgentHistory, resumeConversation, forgetConversation,
- addAgentAttachments, removeAgentAttachment, latestContext,
+ addAgentAttachments, removeAgentAttachment, latestContext, agentAcceptsImages, resolveProvider,
type AgentTurn, type AgentEntry, type AgentToolEntry, type AgentProseEntry,
} from '@/store/AgentStore';
import { COMPACT_AT, shouldCompact } from '@/settings/agentIds';
import type { ConfirmAnswer } from '../../electron/agent/types';
import {
- agentProviders, agentProvider, agentKeyId, parseModelList, subscribeProviders, providersRevision,
+ agentProviders, agentKeyId, subscribeProviders, providersRevision,
} from '@/agent/providers';
import { secretStatus, subscribeSecrets, secretRevision } from '@/store/SecretStore';
import { useSettings } from '@/store/settingsStore';
@@ -952,7 +952,11 @@ function ModelPicker() {
{open && (
<div className="ag-picker-menu">
{providers.map((p) => {
- const def = resolveProviderModels(p.id);
+ // Its models, with the typed list filled in for the one vendor whose
+ // names are not shippable. A provider the person defined arrives
+ // whole — nothing here can tell it from a built-in, which is the
+ // point of projecting the table into the same registry.
+ const def = resolveProvider(p.id) ?? p;
const keyed = secretStatus(agentKeyId(p.id))?.configured === true;
if (def.models.length === 0 && !def.label) return null;
return (
@@ -992,19 +996,6 @@ function ModelPicker() {
const openAgentSettings = () => useEditorStore.getState().openSettings('agents');
-/** The custom provider's list comes from settings; the rest ship theirs. */
-function resolveProviderModels(id: string): { label: string; models: readonly string[] } {
- const def = agentProvider(id);
- if (!def) return { label: '', models: [] };
- if (def.userDefined) {
- return {
- label: t('agent.picker.custom'),
- models: parseModelList(String(useSettings.getState().getValue('agents.customModels') ?? '')),
- };
- }
- return { label: def.label, models: def.models };
-}
-
/** One thing `@` can name: something in the scene, or something on disk. */
interface Mention {
key: string;
@@ -1185,6 +1176,14 @@ function Compose({ autoFocus }: { autoFocus?: boolean }) {
)}
{/* What is going along with the message, as the picture itself: a row of
filenames would be a list of things you cannot check you picked. */}
+ {/* An endpoint that cannot carry them still takes the turn — the model is
+ told a picture was attached rather than handed the text alone — but
+ that is a thing to learn before pressing send, not from the reply. */}
+ {attachments.length > 0 && !agentAcceptsImages() && (
+ <button type="button" className="ag-att-blind" onClick={openAgentSettings}>
+ <ImageOff size={12} strokeWidth={1.8} />{t('agent.attach.blind')}
+ </button>
+ )}
{attachments.length > 0 && (
<div className="ag-atts">
{attachments.map((a) => (
diff --git a/desktop/src/contrib/ContributionRegistry.ts b/desktop/src/contrib/ContributionRegistry.ts
index 881cdfa43..a34970f55 100644
--- a/desktop/src/contrib/ContributionRegistry.ts
+++ b/desktop/src/contrib/ContributionRegistry.ts
@@ -19,8 +19,16 @@
* are recorded in {@link ContributionRegistry.conflicts} for the Plugins panel.
*/
-/** Who owns a contribution. `'core'` is the editor itself. */
-export type Owner = 'core' | `plugin:${string}`;
+/**
+ * Who owns a contribution. `'core'` is the editor itself.
+ *
+ * `'user'` is the person's own configuration — a set they edit in Settings that
+ * is projected into a registry rather than kept beside it (agent providers are
+ * the first). It disposes and re-registers as a group on every edit, which is
+ * the same operation a plugin reload performs, so it belongs here rather than in
+ * a parallel list that every reader would then have to remember to merge.
+ */
+export type Owner = 'core' | 'user' | `plugin:${string}`;
export interface Disposable {
dispose(): void;
diff --git a/desktop/src/i18n/messages/agent.ts b/desktop/src/i18n/messages/agent.ts
index 8a3fa782a..65d101335 100644
--- a/desktop/src/i18n/messages/agent.ts
+++ b/desktop/src/i18n/messages/agent.ts
@@ -32,6 +32,10 @@ export const agentMessages = defineMessages({
// — Images the person attaches —
'agent.attach.remove': { en: 'Remove', zh: '移除' },
'agent.attach.hint': { en: 'drop or paste an image', zh: '拖入或粘贴图片' },
+ 'agent.attach.blind': {
+ en: 'This endpoint does not accept images — it will be told one was attached, not shown it. Settings › AI Agents.',
+ zh: '当前接口不接受图片——只会告诉模型「有人附了一张图」,但它看不到。可在「设置 › AI 代理」里调整。',
+ },
// — Conversations this project has had —
'agent.history.open': { en: 'Earlier conversations', zh: '历史对话' },
diff --git a/desktop/src/i18n/messages/settings.ts b/desktop/src/i18n/messages/settings.ts
index b39cfb24a..6b2f307ec 100644
--- a/desktop/src/i18n/messages/settings.ts
+++ b/desktop/src/i18n/messages/settings.ts
@@ -100,6 +100,7 @@ export const settingsMessages = defineMessages({
'set.secret.stored': { en: 'Stored', zh: '已配置' },
'set.secret.store': { en: 'Store it', zh: '保存' },
'set.secret.forget': { en: 'Forget it', zh: '清除' },
+ 'set.secret.needed': { en: 'Add key', zh: '添加密钥' },
'set.secret.noKeychain': {
en: 'This machine has no keychain to encrypt with, so nothing can be stored here. On Linux, install GNOME Keyring or KWallet, or start the editor with --password-store=gnome-libsecret.',
zh: '这台机器没有可用于加密的密钥库,因此无法保存。Linux 上请安装 GNOME Keyring 或 KWallet,或以 --password-store=gnome-libsecret 启动编辑器。',
@@ -119,7 +120,7 @@ export const settingsMessages = defineMessages({
en: 'Held by the system keychain — never written to the project or the settings file, and never handed back to this window. Every provider keeps its own, so switching back to one costs nothing.',
zh: '存在系统钥匙串里——不写进项目、不写进设置文件,也不会交回给这个窗口。每个提供方各存各的,切回来不用重输。',
},
- 'set.group.customProvider': { en: 'Custom Provider', zh: '自定义提供方' },
+ 'set.group.customProvider': { en: 'Your providers', zh: '你的提供方' },
'set.agents.effort': { en: 'Reasoning depth', zh: '思考深度' },
'set.agents.effort.desc': {
en: 'How hard the model is asked to think before it acts. `xhigh` is what agentic work is for and is the default; drop it when a turn costs more or takes longer than the task deserved. Applies to the next conversation.',
@@ -130,24 +131,37 @@ export const settingsMessages = defineMessages({
en: 'Which models to offer in the composer, comma- or newline-separated. Typed rather than shipped: a list here would be right until the vendor\'s next release, and a name that no longer exists is not refused — the endpoint serves something smaller instead, for the rest of the session.',
zh: '在输入框旁边可选的型号,用逗号或换行分隔。要自己填而不是内置:内置的列表只能正确到厂商下次发布为止,而写了个已经不存在的名字并不会被拒绝——接口会改用一个更小的模型顶上,并且一整个会话都是如此。',
},
- 'set.agents.customProtocol': { en: 'Protocol', zh: '协议' },
- 'set.agents.customProtocol.desc': {
- en: 'Which API this endpoint speaks. The two formats differ in every message they send, so a wrong pick is refused rather than degraded. Chat Completions is what most gateways and every local runner implement.',
- zh: '这个接口讲哪种 API。两种格式发出的每一条消息都不一样,所以选错会被直接拒绝,而不是降级运行。绝大多数网关和所有本地运行器讲的都是 Chat Completions。',
- },
'set.agents.protocol.openai': { en: 'OpenAI (Chat Completions)', zh: 'OpenAI(Chat Completions)' },
'set.agents.protocol.anthropic': { en: 'Anthropic (Messages)', zh: 'Anthropic(Messages)' },
- 'set.agents.customBaseUrl': { en: 'Endpoint', zh: 'API 地址' },
- 'set.agents.customBaseUrl.desc': {
- en: 'The address of an endpoint speaking the protocol above. Only needed for a provider not in the list — the built-in ones already know theirs. On the Anthropic protocol the agent then sends only the core Messages format, without extended thinking, prompt caching or automatic fallbacks.',
- zh: '讲上面那个协议的接口地址。只有列表里没有的提供方才需要填——内置的那些自己知道地址。走 Anthropic 协议时,Agent 只发送基础 Messages 格式,不带扩展思考、提示缓存和自动回退。',
- },
- 'set.agents.customModels': { en: 'Models', zh: '模型' },
- 'set.agents.customModels.desc': {
- en: 'The model names this endpoint offers, comma- or newline-separated. They become choices in the composer.',
- zh: '这个接口提供的模型名,用逗号或换行分隔。它们会出现在输入框的模型选择器里。',
+ // In a table cell the long form is truncated to something no shorter and far
+ // less readable; which format each name means is the row's description.
+ 'set.agents.protocolShort.openai': { en: 'OpenAI', zh: 'OpenAI' },
+ 'set.agents.protocolShort.anthropic': { en: 'Anthropic', zh: 'Anthropic' },
+ 'set.agents.providers': { en: 'Endpoints', zh: '接口' },
+ 'set.agents.providers.desc': {
+ en: 'Endpoints of your own — a local runner, a company gateway, a vendor not in the list above. Each keeps its own key and appears in the composer under its own name. The expander holds what the endpoint can DO: what it cannot be told, the agent has to work around.',
+ zh: '你自己的接口——本地运行器、公司网关,或者上面列表里没有的厂商。每一条各存各的密钥,并以自己的名字出现在输入框的模型选择器里。展开项里是这个接口「能做什么」:没告诉它的部分,Agent 只能绕着走。',
+ },
+ 'set.agents.providers.empty': {
+ en: 'None yet. The built-in providers above cover the major vendors; add one here for anything else that speaks either protocol.',
+ zh: '还没有。上面的内置提供方覆盖了主要厂商;讲这两种协议中任意一种的其它接口,在这里添加。',
+ },
+ 'set.agents.addProvider': { en: 'Add a provider', zh: '添加提供方' },
+ 'set.agents.col.label': { en: 'Name', zh: '名称' },
+ 'set.agents.col.protocol': { en: 'Protocol', zh: '协议' },
+ 'set.agents.col.baseUrl': { en: 'Endpoint', zh: 'API 地址' },
+ 'set.agents.col.models': { en: 'Models', zh: '模型' },
+ 'set.agents.col.key': { en: 'Key', zh: '密钥' },
+ 'set.agents.capabilities': { en: 'What it can do', zh: '它能做什么' },
+ 'set.agents.col.contextWindow': { en: 'Context window', zh: '上下文窗口' },
+ 'set.agents.col.reasoningEffort': { en: 'Takes a reasoning-depth argument', zh: '接受思考深度参数' },
+ 'set.agents.err.baseUrl': { en: 'This provider has no endpoint address.', zh: '这个提供方还没有填接口地址。' },
+ 'set.agents.err.models': { en: 'No models, so nothing to pick in the composer.', zh: '没有填模型,输入框里就没有可选项。' },
+ 'set.agents.customVision': { en: 'Accepts images', zh: '接受图片' },
+ 'set.agents.customVision.desc': {
+ en: 'Turn on only if this endpoint\'s models can see. The agent takes screenshots to check its own work; where they cannot be sent it asks for the same frame as a coarse text grid instead, and says so in the transcript. Claiming sight an endpoint lacks costs a whole turn to a refused request, which is why this starts off.',
+ zh: '只有这个接口的模型确实能看图时才打开。Agent 会截图来检查自己干的活;发不出去图片时,它会改成把同一帧要成粗颗粒的文字色块网格,并在对话里说明。给一个其实看不了图的接口打开它,代价是整轮请求被拒——所以默认是关的。',
},
- 'set.agents.customKey': { en: 'Custom API key', zh: '自定义提供方密钥' },
'set.agents.mcpEnabled': { en: 'Allow AI agents to connect', zh: '允许 AI 代理连接' },
'set.agents.mcpEnabled.desc': {
en: 'Serve this editor over MCP on a local-only port, so an agent set up with --attach can drive the project you have open. Off means nothing is listening.',
diff --git a/desktop/src/settings/editorSettings.ts b/desktop/src/settings/editorSettings.ts
index db9247e9b..b98e7a7ef 100644
--- a/desktop/src/settings/editorSettings.ts
+++ b/desktop/src/settings/editorSettings.ts
@@ -20,10 +20,15 @@ import { Toasts } from '@/store/Toasts';
import { mcpStatus, setMcpEnabled, subscribeMcp } from '@/store/McpStore';
import { secretStatusLine, subscribeSecrets } from '@/store/SecretStore';
import { syncAgentEndpoint } from '@/store/AgentStore';
+import { useSettings } from '@/store/settingsStore';
import {
- agentProviders, agentKeyId, modelsSettingId, AGENT_PROTOCOLS, CUSTOM_PROVIDER,
+ agentProviders, agentKeyId, modelsSettingId, AGENT_PROTOCOLS, parseModelList,
} from '@/agent/providers';
-import { AGENT_EFFORTS, DEFAULT_EFFORT } from '@/settings/agentIds';
+import {
+ AGENT_PROVIDERS_SETTING, syncUserProviders, newProviderRow, forgetProviderSecret,
+ migrateLegacyCustomProvider,
+} from '@/agent/userProviders';
+import { AGENT_EFFORTS, DEFAULT_EFFORT, DEFAULT_CONTEXT_WINDOW } from '@/settings/agentIds';
import { applyUiZoom, UI_SCALE_SETTING, ZOOM_DEFAULT, ZOOM_MIN, ZOOM_MAX } from '@/layout/uiZoom';
import { t, type MsgKey, editorLocale, systemDefaultLocale, EDITOR_LOCALES, LANGUAGE_SETTING_ID } from '@/i18n';
@@ -290,62 +295,68 @@ settingsRegistry.register({
effect: () => syncAgentEndpoint(),
});
-// The escape hatch, for a provider we have not heard of — which is exactly the
-// one we cannot ship an endpoint or a model list for.
+// Providers we have not heard of — which are exactly the ones we cannot ship an
+// address, a model list, or a word about what they accept.
//
-// Which PROTOCOL it speaks has to be asked, not guessed: the two formats differ
-// in every message they send, so getting it wrong is a 400 about a field the
-// person never wrote. Chat Completions is offered first because it is what most
-// endpoints — and every local runner — implement.
-settingsRegistry.register({
- id: 'agents.customProtocol',
- type: 'enum',
- scope: 'editor',
- section: 'agents',
- group: t('set.group.customProvider'),
- label: t('set.agents.customProtocol'),
- description: t('set.agents.customProtocol.desc'),
- default: 'openai',
- options: AGENT_PROTOCOLS.map((value) => ({ value, label: t(`set.agents.protocol.${value}` as MsgKey) })),
- effect: () => syncAgentEndpoint(),
-});
-
-settingsRegistry.register({
- id: 'agents.customBaseUrl',
- type: 'string',
- scope: 'editor',
- section: 'agents',
- group: t('set.group.customProvider'),
- label: t('set.agents.customBaseUrl'),
- description: t('set.agents.customBaseUrl.desc'),
- placeholder: 'https://example.com/v1',
- default: '',
- effect: () => syncAgentEndpoint(),
-});
-
+// A TABLE, not a set of fields, because the unit of configuration is a provider
+// and a person has more than one: a local runner and a company gateway are not
+// two states of one setting. Each row keeps its own key, for the same reason the
+// shipped rows above do.
+//
+// The main columns say where it is and what it speaks; the expander says what it
+// can DO. That split is the point — an endpoint that could only be given an
+// address had no way to tell the agent it accepts screenshots, and the agent
+// spent every session working blind without either side deciding that.
settingsRegistry.register({
- id: 'agents.customModels',
- type: 'string',
+ id: AGENT_PROVIDERS_SETTING,
+ type: 'objectList',
scope: 'editor',
section: 'agents',
group: t('set.group.customProvider'),
- label: t('set.agents.customModels'),
- description: t('set.agents.customModels.desc'),
- placeholder: 'model-a, model-b',
- default: '',
- effect: () => syncAgentEndpoint(),
+ label: t('set.agents.providers'),
+ description: t('set.agents.providers.desc'),
+ layout: 'block',
+ default: [],
+ columns: [
+ { key: 'label', label: t('set.agents.col.label'), type: 'text', width: '0.9fr', placeholder: 'Local llama' },
+ {
+ key: 'protocol',
+ label: t('set.agents.col.protocol'),
+ type: 'enum',
+ width: '108px',
+ options: AGENT_PROTOCOLS.map((value) => ({ value, label: t(`set.agents.protocolShort.${value}` as MsgKey) })),
+ },
+ { key: 'baseUrl', label: t('set.agents.col.baseUrl'), type: 'text', width: '1.6fr', placeholder: 'http://localhost:11434/v1' },
+ { key: 'models', label: t('set.agents.col.models'), type: 'text', width: '1.1fr', placeholder: 'model-a, model-b' },
+ { key: 'apiKey', label: t('set.agents.col.key'), type: 'secret', width: '96px', secretId: (row) => agentKeyId(String(row.id ?? '')) },
+ ],
+ detailColumns: [
+ { key: 'vision', label: t('set.agents.customVision'), type: 'boolean' },
+ { key: 'contextWindow', label: t('set.agents.col.contextWindow'), type: 'number', width: '104px', min: 0, placeholder: String(DEFAULT_CONTEXT_WINDOW) },
+ { key: 'reasoningEffort', label: t('set.agents.col.reasoningEffort'), type: 'boolean' },
+ ],
+ detailLabel: t('set.agents.capabilities'),
+ addLabel: t('set.agents.addProvider'),
+ emptyHint: t('set.agents.providers.empty'),
+ newRow: () => newProviderRow(
+ useSettings.getState().getValue<Record<string, unknown>[]>(AGENT_PROVIDERS_SETTING) ?? [],
+ ) as unknown as Record<string, unknown>,
+ rowError: (row) => {
+ if (!String(row.baseUrl ?? '').trim()) return t('set.agents.err.baseUrl');
+ if (parseModelList(String(row.models ?? '')).length === 0) return t('set.agents.err.models');
+ return null;
+ },
+ // A key outlives the row it belonged to otherwise: sealed on the machine,
+ // unreachable from any UI, and waiting to be inherited by a later provider.
+ onRowRemoved: forgetProviderSecret,
+ effect: () => {
+ syncUserProviders();
+ syncAgentEndpoint();
+ },
});
-settingsRegistry.register({
- id: agentKeyId(CUSTOM_PROVIDER),
- type: 'secret',
- scope: 'editor',
- section: 'agents',
- group: t('set.group.customProvider'),
- label: t('set.agents.customKey'),
- default: false,
- status: { read: () => secretStatusLine(agentKeyId(CUSTOM_PROVIDER)), subscribe: subscribeSecrets },
-});
+// Fold a pre-table setup into the table, before anything reads either.
+migrateLegacyCustomProvider();
// ── External agents (main-owned endpoint, driven from here) ─────────────────
// An external agent reaches this editor through the MCP endpoint main can expose,
diff --git a/desktop/src/store/AgentStore.ts b/desktop/src/store/AgentStore.ts
index 09352e73c..b1a8e54b4 100644
--- a/desktop/src/store/AgentStore.ts
+++ b/desktop/src/store/AgentStore.ts
@@ -20,9 +20,8 @@ import { create } from 'zustand';
import type { AgentStatus, AgentMessage } from '../../electron/agent/host';
import type { AgentEvent, ConfirmAnswer, ConfirmReason, ConfirmRequest } from '../../electron/agent/types';
import {
- agentProviders, agentProvider, agentKeyId, parseModelList, protocolOf, asProtocol,
- modelsSettingId, CUSTOM_PROVIDER,
- DEFAULT_CONTEXT_WINDOW,
+ agentProviders, agentProvider, agentKeyId, parseModelList, protocolOf,
+ modelsSettingId, DEFAULT_CONTEXT_WINDOW, type AgentProviderDef,
} from '@/agent/providers';
import { refreshSecret, secretStatus, subscribeSecrets } from '@/store/SecretStore';
import { useSettings } from '@/store/settingsStore';
@@ -276,26 +275,34 @@ function loadSelection(): AgentSelection | null {
}
}
-/** The custom provider's endpoint and models come from settings — it is the one
- * we cannot ship a list for. */
-function resolveProvider(id: string) {
+/**
+ * A provider, with the one part of it that cannot be shipped filled in.
+ *
+ * Providers the person typed need nothing here — they are whole defs by the time
+ * they reach the registry (agent/userProviders.ts), which is the point of
+ * projecting them into it. What is left is the shipped vendor whose ADDRESS
+ * holds still while its model names do not.
+ */
+export function resolveProvider(id: string): AgentProviderDef | undefined {
const def = agentProvider(id);
- if (!def) return undefined;
- const settings = useSettings.getState();
- // A shipped provider whose model NAMES are not shippable — everything else
- // about it still is, including its endpoint and its key row.
- if (def.typedModels) {
- return { ...def, models: parseModelList(String(settings.getValue(modelsSettingId(id)) ?? '')) };
- }
- if (id !== CUSTOM_PROVIDER) return def;
- return {
- ...def,
- protocol: asProtocol(settings.getValue('agents.customProtocol')),
- baseUrl: String(settings.getValue('agents.customBaseUrl') ?? ''),
- models: parseModelList(String(settings.getValue('agents.customModels') ?? '')),
- };
+ if (!def?.typedModels) return def;
+ const typed = String(useSettings.getState().getValue(modelsSettingId(id)) ?? '');
+ return { ...def, models: parseModelList(typed) };
}
+/**
+ * Whether the picked endpoint can be handed a picture.
+ *
+ * The composer takes attachments whatever is selected, and an endpoint that
+ * cannot carry them is not a failure — the turn goes out with the text and the
+ * model is told what it was not shown. But that is a thing to learn BEFORE
+ * sending, so this is read where the attachment is made too.
+ */
+export const agentAcceptsImages = (): boolean => {
+ const selection = effectiveSelection();
+ return selection ? resolveProvider(selection.providerId)?.vision === true : false;
+};
+
/** The configured reasoning depth, narrowed. */
export const agentEffort = (): AgentEffort =>
asEffort(useSettings.getState().getValue('agents.effort'));
@@ -335,6 +342,14 @@ export function syncAgentEndpoint(): void {
// How far the conversation may grow before it is compacted. Travels with the
// endpoint because it is the same piece of knowledge: which provider this is.
contextWindow: def?.contextWindow ?? DEFAULT_CONTEXT_WINDOW,
+ // Whether a screenshot can reach it. Same journey as the window and for the
+ // same reason: both are facts about which provider this is, and both were
+ // once guessed from the address instead of being told.
+ vision: def?.vision === true,
+ // Whether the depth below may be NAMED in the request. Its own fact because
+ // an endpoint that has never heard of the argument refuses the call rather
+ // than ignoring it — losing the turn over a field nobody asked for.
+ reasoningEffort: def?.reasoningEffort !== false,
// How hard the model is asked to think. A setting rather than part of the
// model pick: the same model is worth running at different depths, and the
// depth is the one thing a person adjusts because a turn cost too much or
diff --git a/desktop/src/theme/agent.css b/desktop/src/theme/agent.css
index 98c201585..ef592550b 100644
--- a/desktop/src/theme/agent.css
+++ b/desktop/src/theme/agent.css
@@ -1069,6 +1069,21 @@
transition: color var(--t-fast), background var(--t-fast);
}
.ag-att-x:hover { color: var(--text-hi); background: var(--pressed); }
+/* The picture will not reach the model. Said before send, next to the thumbnail
+ it is about, and it leads to the setting that decides it. */
+.ag-att-blind {
+ display: flex; align-items: flex-start; gap: 6px;
+ margin: 0 0 6px;
+ padding: 5px 8px;
+ border-radius: var(--r-sm);
+ background: color-mix(in srgb, var(--warn) 12%, transparent);
+ color: var(--text-dim);
+ font: var(--fs-2xs) var(--font-ui);
+ text-align: left;
+ line-height: 1.45;
+}
+.ag-att-blind:hover { color: var(--text-hi); }
+.ag-att-blind svg { color: var(--warn); flex: none; margin-top: 1px; }
/* A drag is over the composer: the whole box says so, since the whole box takes
it — aiming at a one-line textarea is a precision task nobody should be set. */
.ag-compose.dropping .ag-cbox { border-color: var(--acc); background: var(--hover); }
diff --git a/desktop/tests/agent-dialect.test.ts b/desktop/tests/agent-dialect.test.ts
index c719fa23f..accc5ddc4 100644
--- a/desktop/tests/agent-dialect.test.ts
+++ b/desktop/tests/agent-dialect.test.ts
@@ -143,8 +143,8 @@ describe('a tool result carrying a rendered frame', () => {
image: { data: 'BASE64PNG', mediaType: 'image/png' },
};
- it('sends the image itself to Anthropic', () => {
- const content = toolResultContent(shot, 'anthropic');
+ it('sends the image itself where the endpoint takes one', () => {
+ const content = toolResultContent(shot, true);
expect(Array.isArray(content)).toBe(true);
expect((content as { type: string }[])[0]).toMatchObject({
type: 'image',
@@ -152,16 +152,48 @@ describe('a tool result carrying a rendered frame', () => {
});
});
- it('tells a gateway why there is no image rather than dropping it in silence', () => {
- const content = toolResultContent(shot, 'compatible');
+ it('tells a blind endpoint why there is no image rather than dropping it in silence', () => {
+ const content = toolResultContent(shot, false);
expect(typeof content).toBe('string');
expect(content).toContain('does not accept images');
});
- it('leaves an ordinary result a plain string on both', () => {
+ it('leaves an ordinary result a plain string either way', () => {
const plain = { id: 'c2', content: '7 created', isError: false };
- expect(toolResultContent(plain, 'anthropic')).toBe('7 created');
- expect(toolResultContent(plain, 'compatible')).toBe('7 created');
+ expect(toolResultContent(plain, true)).toBe('7 created');
+ expect(toolResultContent(plain, false)).toBe('7 created');
+ });
+});
+
+/**
+ * Sight is DECLARED, not read off the address.
+ *
+ * It was read off the address once, and that made every gateway blind with no
+ * way to say otherwise — including ones whose models see perfectly well, since
+ * image blocks are core format and not one of the extensions the dialect gates.
+ */
+describe('whether the endpoint is told to be blind', () => {
+ const provider = (over = {}) => createAnthropicProvider({ apiKey: 'k', ...over });
+ const gateway = 'https://gateway.example/anthropic';
+
+ it('takes the provider at its word, over what the address implies', () => {
+ expect(provider({ baseURL: gateway, vision: true }).acceptsImages).toBe(true);
+ expect(provider({ vision: false }).acceptsImages).toBe(false);
+ });
+
+ it('falls back to the address for a caller that declares nothing', () => {
+ expect(provider().acceptsImages).toBe(true);
+ expect(provider({ baseURL: gateway }).acceptsImages).toBe(false);
+ });
+
+ // The dialect still gates the EXTENSIONS — a gateway that sees is not a
+ // gateway that took adaptive thinking, caching and server-side fallbacks.
+ it('does not buy the extensions along with the sight', () => {
+ const session = provider({ baseURL: gateway, vision: true })
+ .createSession({ system: 's', tools: TOOLS }) as unknown as {
+ opts: { dialect: string; vision: boolean };
+ };
+ expect([session.opts.dialect, session.opts.vision]).toEqual(['compatible', true]);
});
});
@@ -224,8 +256,8 @@ describe('what a failed request says', () => {
describe('an image on the person\'s turn', () => {
const shot = [{ mediaType: 'image/png', data: 'AAAA' }];
/** The session's messages — what the request would carry. */
- const messagesOf = (baseURL?: string, images = shot, text = 'like this') => {
- const provider = createAnthropicProvider({ apiKey: 'k', baseURL });
+ const messagesOf = (baseURL?: string, images = shot, text = 'like this', vision?: boolean) => {
+ const provider = createAnthropicProvider({ apiKey: 'k', baseURL, vision });
const session = provider.createSession({ system: 's', tools: TOOLS });
session.pushUser(text, images);
return (session.serialize() as { messages: Array<{ role: string; content: unknown }> }).messages;
@@ -255,6 +287,13 @@ describe('an image on the person\'s turn', () => {
expect(msg.content as string).toMatch(/cannot receive images|have not seen/);
});
+ // And a gateway that DOES take them gets the picture, because what decides
+ // this is what the provider declared and not where the endpoint lives.
+ it('rides the turn on a gateway that declared it can see', () => {
+ const [msg] = messagesOf('https://gateway.example/anthropic', shot, 'like this', true);
+ expect((msg.content as Array<{ type: string }>).map((b) => b.type)).toEqual(['image', 'text']);
+ });
+
it('counts them when there are several', () => {
const two = [shot[0], { mediaType: 'image/jpeg', data: 'BBBB' }];
const [msg] = messagesOf('https://gateway.example/anthropic', two);
diff --git a/desktop/tests/agent-openai.test.ts b/desktop/tests/agent-openai.test.ts
index bf9a347fa..827984a1d 100644
--- a/desktop/tests/agent-openai.test.ts
+++ b/desktop/tests/agent-openai.test.ts
@@ -121,12 +121,14 @@ describe('the provider', () => {
expect(provider({ baseURL: 'http://localhost:11434/v1' }).id).toBe('openai:http://localhost:11434/v1');
});
- // A gateway we have never heard of may not take images, and finding out costs
- // a call spent on a screenshot the model is then told it cannot look at.
- it('assumes a gateway cannot see, and OpenAI itself can', () => {
+ // Declared by whoever picked the provider (src/agent/providers.ts). The
+ // address is only the fallback for a caller that says nothing — reading it as
+ // the answer is what left every gateway blind with no way to say otherwise.
+ it('takes the caller at its word, and falls back to the address', () => {
+ expect(provider({ baseURL: 'http://x/v1', vision: true }).acceptsImages).toBe(true);
+ expect(provider({ vision: false }).acceptsImages).toBe(false);
expect(provider().acceptsImages).toBe(true);
expect(provider({ baseURL: 'http://localhost:11434/v1' }).acceptsImages).toBe(false);
- expect(provider({ baseURL: 'http://x/v1', vision: true }).acceptsImages).toBe(true);
});
/**
diff --git a/desktop/tests/agent-providers.test.ts b/desktop/tests/agent-providers.test.ts
index eae08986a..cd1776b96 100644
--- a/desktop/tests/agent-providers.test.ts
+++ b/desktop/tests/agent-providers.test.ts
@@ -10,6 +10,7 @@
import { describe, it, expect } from 'vitest';
import {
agentProviders, agentProvider, agentKeyId, parseModelList, protocolOf, asProtocol, CUSTOM_PROVIDER,
+ setUserProviders,
} from '@/agent/providers';
describe('the shipped providers', () => {
@@ -41,14 +42,55 @@ describe('the shipped providers', () => {
expect(protocolOf(openai)).toBe('openai');
expect(openai?.typedModels).toBe(true);
expect(openai?.models).toEqual([]);
- // Still a shipped provider: it keeps its own key row, unlike the custom slot.
+ // Still a shipped provider: it keeps its own key row, unlike a typed one.
expect(openai?.userDefined).toBeUndefined();
});
- it('ships the custom slot empty — it is the one we cannot know', () => {
- const custom = agentProvider(CUSTOM_PROVIDER);
- expect(custom?.userDefined).toBe(true);
- expect(custom?.models).toEqual([]);
+ // There is no reserved slot for "the custom one" any more. There cannot be:
+ // the person has as many as they have endpoints, and a single slot meant
+ // configuring the second overwrote the first, credential included.
+ it('ships no placeholder for the providers a person types', () => {
+ expect(agentProviders().every((p) => !p.userDefined)).toBe(true);
+ expect(agentProvider(CUSTOM_PROVIDER)).toBeUndefined();
+ });
+});
+
+/**
+ * The typed providers arrive as a SET, because that is how they are edited — a
+ * table rewritten whole — and because the removals matter as much as the
+ * additions: one deleted from the table must stop being offered.
+ */
+describe('the providers a person defined', () => {
+ const def = (id: string, over = {}) => ({ id, label: id, baseUrl: `http://${id}/v1`, models: ['m'], ...over });
+
+ it('joins the shipped ones, marked as theirs, and leaves when withdrawn', () => {
+ setUserProviders([def('mine-a'), def('mine-b')]);
+ expect(agentProvider('mine-a')?.userDefined).toBe(true);
+ expect(agentProviders().map((p) => p.id)).toEqual(
+ expect.arrayContaining(['anthropic', 'deepseek', 'openai', 'mine-a', 'mine-b']),
+ );
+
+ setUserProviders([def('mine-a')]);
+ expect(agentProvider('mine-b')).toBeUndefined();
+ expect(agentProvider('mine-a')).toBeDefined();
+
+ setUserProviders([]);
+ expect(agentProviders().every((p) => !p.userDefined)).toBe(true);
+ });
+
+ // Shipped first: the picker reads top to bottom, and the vendor rows are what
+ // a person who has typed nothing is looking for.
+ it('lists after the shipped ones', () => {
+ setUserProviders([def('mine-a')]);
+ const ids = agentProviders().map((p) => p.id);
+ expect(ids.indexOf('mine-a')).toBeGreaterThan(ids.indexOf('openai'));
+ setUserProviders([]);
+ });
+
+ it('cannot take an id the editor already ships', () => {
+ setUserProviders([def('anthropic', { label: 'Impostor' })]);
+ expect(agentProvider('anthropic')?.label).toBe('Anthropic');
+ setUserProviders([]);
});
});
diff --git a/desktop/tests/agent-vision.test.ts b/desktop/tests/agent-vision.test.ts
new file mode 100644
index 000000000..8541cc033
--- /dev/null
+++ b/desktop/tests/agent-vision.test.ts
@@ -0,0 +1,245 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2024-present ESEngine Team
+/**
+ * @file What an endpoint can DO, from the row that declares it to the endpoint
+ * main builds.
+ *
+ * Sight used to be read off the address — no baseUrl, therefore the
+ * vendor's own API, therefore it can see — which is a statement about
+ * WHERE an endpoint is and not about what it accepts. Every gateway was
+ * blind with no way to say otherwise, and the whole consequence was
+ * silent: the agent was told each turn that screenshots could not reach
+ * it, asked for `format: 'grid'` instead, and nobody could tell that
+ * apart from a preference.
+ *
+ * The wire halves live with their protocols (agent-dialect, agent-openai);
+ * what is checked here is that the facts travel.
+ */
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import '@/settings';
+import { useSettings } from '@/store/settingsStore';
+import { useAgent, syncAgentEndpoint, agentAcceptsImages } from '@/store/AgentStore';
+import { agentProvider, CUSTOM_PROVIDER } from '@/agent/providers';
+import {
+ AGENT_PROVIDERS_SETTING, syncUserProviders, type UserProviderRow,
+} from '@/agent/userProviders';
+
+interface Endpoint {
+ vision?: boolean; baseUrl?: string; protocol?: string; contextWindow?: number;
+ reasoningEffort?: boolean;
+}
+
+let sent: Endpoint[];
+
+/** A row as the settings table stores one. */
+const row = (over: Partial<UserProviderRow> = {}): Record<string, unknown> => ({
+ id: 'custom-1',
+ label: 'Local llama',
+ protocol: 'openai',
+ baseUrl: 'http://localhost:11434/v1',
+ models: 'llama-4',
+ vision: false,
+ contextWindow: 0,
+ reasoningEffort: true,
+ ...over,
+} as unknown as Record<string, unknown>);
+
+/** Put rows in the table and select the first — what picking one in the composer does. */
+const configure = (...rows: Record<string, unknown>[]) => {
+ useSettings.getState().setValue(AGENT_PROVIDERS_SETTING, rows);
+ syncUserProviders();
+ const first = rows[0];
+ useAgent.setState({
+ selection: { providerId: String(first.id), model: String(first.models).split(',')[0].trim() },
+ });
+};
+
+beforeEach(() => {
+ sent = [];
+ (globalThis as { window?: unknown }).window = {
+ estella: { agent: { setEndpoint: async (patch: Endpoint) => { sent.push(patch); } } },
+ };
+ useAgent.setState({ selection: null });
+ useSettings.getState().setValue(AGENT_PROVIDERS_SETTING, []);
+ syncUserProviders();
+});
+
+afterEach(() => {
+ useSettings.getState().setValue(AGENT_PROVIDERS_SETTING, []);
+ syncUserProviders();
+ delete (globalThis as { window?: unknown }).window;
+});
+
+describe('what a shipped provider declares', () => {
+ it('says so where it is true, rather than leaving it to the address', () => {
+ expect(agentProvider('anthropic')?.vision).toBe(true);
+ expect(agentProvider('openai')?.vision).toBe(true);
+ });
+
+ // The two mistakes are not symmetrical: claiming sight an endpoint lacks
+ // spends the turn on a refused request, withholding it costs a text grid.
+ it('withholds it from a provider we have not confirmed', () => {
+ expect(agentProvider('deepseek')?.vision).toBeUndefined();
+ });
+});
+
+describe('an endpoint the person described', () => {
+ it('is blind until they say otherwise', () => {
+ configure(row());
+ expect(agentAcceptsImages()).toBe(false);
+ });
+
+ it('can be declared sighted, and the endpoint hears about it', () => {
+ configure(row({ vision: true }));
+ syncAgentEndpoint();
+ expect(agentAcceptsImages()).toBe(true);
+ expect(sent.at(-1)).toMatchObject({ vision: true, baseUrl: 'http://localhost:11434/v1' });
+ });
+
+ // A gateway address is not an answer to this question — that reading is the
+ // bug. What matters is that the two facts travel separately.
+ it('carries the declaration alongside the address, not derived from it', () => {
+ configure(row({ vision: true }));
+ sent.length = 0;
+ syncAgentEndpoint();
+ expect(sent).toHaveLength(1);
+ expect([sent[0].baseUrl, sent[0].vision]).toEqual(['http://localhost:11434/v1', true]);
+ });
+
+ // Compaction at three quarters of a window that is not this endpoint's folds
+ // a conversation that had room, or fails one that did not.
+ it('says how much room it has, instead of being assumed small', () => {
+ configure(row({ contextWindow: 1_000_000 }));
+ syncAgentEndpoint();
+ expect(sent.at(-1)?.contextWindow).toBe(1_000_000);
+ });
+
+ // An endpoint that has never heard of the argument refuses the whole call.
+ it('can refuse the reasoning-depth argument without losing every turn', () => {
+ configure(row({ reasoningEffort: false }));
+ syncAgentEndpoint();
+ expect(sent.at(-1)?.reasoningEffort).toBe(false);
+ });
+});
+
+describe('a provider that ships its own answer', () => {
+ it('travels without the person configuring anything', () => {
+ useAgent.setState({ selection: { providerId: 'anthropic', model: 'claude-opus-5' } });
+ syncAgentEndpoint();
+ expect(sent.at(-1)).toMatchObject({ vision: true, reasoningEffort: true });
+ expect(agentAcceptsImages()).toBe(true);
+ });
+
+ it('and a shipped gateway that has not claimed sight stays blind', () => {
+ useAgent.setState({ selection: { providerId: 'deepseek', model: 'deepseek-v4-pro' } });
+ syncAgentEndpoint();
+ expect(sent.at(-1)?.vision).toBe(false);
+ expect(agentAcceptsImages()).toBe(false);
+ });
+});
+
+/**
+ * The reason the table exists: a person has more than one of these, and the
+ * shape it replaced could only hold one — configuring a second overwrote the
+ * first, credential included.
+ */
+describe('more than one of them', () => {
+ it('keeps both, each with its own key id and its own capabilities', () => {
+ configure(
+ row({ id: 'custom-1', label: 'Local llama', vision: false }),
+ row({ id: 'custom-2', label: 'Work gateway', baseUrl: 'https://gw.corp/v1', models: 'gpt-x', vision: true }),
+ );
+ const a = agentProvider('custom-1');
+ const b = agentProvider('custom-2');
+ expect([a?.label, a?.vision, a?.baseUrl]).toEqual(['Local llama', false, 'http://localhost:11434/v1']);
+ expect([b?.label, b?.vision, b?.baseUrl]).toEqual(['Work gateway', true, 'https://gw.corp/v1']);
+ });
+
+ it('stops offering one that was deleted', () => {
+ configure(row({ id: 'custom-1' }), row({ id: 'custom-2' }));
+ expect(agentProvider('custom-2')).toBeDefined();
+ useSettings.getState().setValue(AGENT_PROVIDERS_SETTING, [row({ id: 'custom-1' })]);
+ syncUserProviders();
+ expect(agentProvider('custom-2')).toBeUndefined();
+ expect(agentProvider('custom-1')).toBeDefined();
+ });
+
+ // The registry is the one door: a shipped provider and a typed one are the
+ // same kind of thing to everything downstream of it.
+ it('does not let a typed row displace a shipped provider', () => {
+ configure(row({ id: 'anthropic', label: 'Impostor', baseUrl: 'http://evil/v1' }));
+ expect(agentProvider('anthropic')?.label).toBe('Anthropic');
+ });
+});
+
+/**
+ * A key is filed under its provider's id and lives outside the list, so the two
+ * ends of a row's life both have to account for it: a deleted row's credential
+ * must go, and a new row must never be handed an id that could still have one.
+ */
+describe('the credential a row owns', () => {
+ it('goes when the row does, under the id it was filed against', async () => {
+ const cleared: string[] = [];
+ (globalThis as { window?: unknown }).window = {
+ estella: { secrets: { clear: async (id: string) => { cleared.push(id); return undefined; } } },
+ };
+ const { forgetProviderSecret } = await import('@/agent/userProviders');
+ forgetProviderSecret(row({ id: 'custom-2' }));
+ await Promise.resolve();
+ expect(cleared).toEqual(['agents.key.custom-2']);
+ });
+
+ // An id that came back around would inherit whatever is still sealed under it.
+ it('never re-uses an id, even after the rows between are gone', async () => {
+ const { newProviderRow } = await import('@/agent/userProviders');
+ expect(newProviderRow([]).id).toBe('custom-1');
+ expect(newProviderRow([row({ id: 'custom-1' }), row({ id: 'custom-2' })]).id).toBe('custom-3');
+ // custom-2 was deleted; the next row must not step into its place.
+ expect(newProviderRow([row({ id: 'custom-3' })]).id).toBe('custom-4');
+ // The migrated singleton carries a non-numbered id and must not confuse it.
+ expect(newProviderRow([row({ id: CUSTOM_PROVIDER })]).id).toBe('custom-1');
+ });
+});
+
+/**
+ * A key is filed under its provider's id, and main never hands a sealed one
+ * back — so an id that does not carry over is a credential the person has to
+ * find again. This is the one migration step that cannot be repaired by hand.
+ */
+describe('a setup made before the table existed', () => {
+ it('keeps the id its key is filed under', async () => {
+ const { migrateLegacyCustomProvider } = await import('@/agent/userProviders');
+ useSettings.setState({
+ values: {
+ 'agents.customProtocol': 'anthropic',
+ 'agents.customBaseUrl': 'https://gateway.example/anthropic',
+ 'agents.customModels': 'some-model',
+ 'agents.customVision': true,
+ },
+ });
+ migrateLegacyCustomProvider();
+ syncUserProviders();
+
+ const def = agentProvider(CUSTOM_PROVIDER);
+ expect(def).toBeDefined();
+ expect([def?.protocol, def?.baseUrl, def?.vision]).toEqual([
+ 'anthropic', 'https://gateway.example/anthropic', true,
+ ]);
+ expect(def?.models).toEqual(['some-model']);
+ // And the shape it replaced is gone from the file rather than left to be
+ // puzzled over by whoever opens it next.
+ expect(useSettings.getState().values['agents.customBaseUrl']).toBeUndefined();
+ });
+
+ it('leaves a table the person has already emptied alone', async () => {
+ const { migrateLegacyCustomProvider } = await import('@/agent/userProviders');
+ useSettings.setState({
+ values: {
+ [AGENT_PROVIDERS_SETTING]: [],
+ 'agents.customBaseUrl': 'https://gateway.example/anthropic',
+ },
+ });
+ migrateLegacyCustomProvider();
+ expect(useSettings.getState().getValue(AGENT_PROVIDERS_SETTING)).toEqual([]);
+ });
+});
diff --git a/docs/astro/src/content/docs/extending/built-in-agent.mdx b/docs/astro/src/content/docs/extending/built-in-agent.mdx
index 5fb9d3e01..15beadf22 100644
--- a/docs/astro/src/content/docs/extending/built-in-agent.mdx
+++ b/docs/astro/src/content/docs/extending/built-in-agent.mdx
@@ -33,8 +33,10 @@ and configure one provider — that is the whole setup.
name that no longer exists is not refused — the endpoint quietly serves something
smaller instead, for the rest of the session.
-3. Or fill in the **Custom Provider** — any endpoint at all. Pick which **protocol**
- it speaks, then give its address and its model names.
+3. Or add your own under **Your providers** — any endpoint at all, as many as you
+ have. Each row is a provider like the shipped ones: a name, which **protocol**
+ it speaks, its address, its model names, and its own key. Expand a row to say
+ what that endpoint can do.
4. Close Settings. The model picker sits next to the box you type in — it is a
per-message choice, not a preference buried in a dialog.
@@ -56,6 +58,24 @@ settings file, and never handed back to the editor window — only to the client
has to send it.
</Aside>
+## What an endpoint can do
+
+The shipped providers already say. For one of your own the editor has no way to
+find out, and guessing is expensive in both directions — so a row's expander is
+where you tell it. What is left unsaid is worked around, not discovered.
+
+- **Accepts images** — the agent takes screenshots to check its own work. Where a
+ picture cannot be sent it asks for the same frame as a coarse colour grid in
+ text instead, and says so in the transcript. Off by default: claiming sight an
+ endpoint lacks costs a whole turn to a refused request, while withholding it
+ costs detail the agent can mostly do without.
+- **Context window** — how far a conversation may grow before its oldest runs are
+ folded away. Blank assumes a conservative window, which folds a conversation
+ that still had room; too generous fails the turn outright, after the wait.
+- **Takes a reasoning-depth argument** — whether the request may name the depth
+ set above. An endpoint that has never heard of the argument refuses the whole
+ call rather than ignoring it, so turn this off for one that does.
+
## Ask it something
Open the command palette (`Ctrl/Cmd+Shift+P`) and run **Agent: Open**. Or skip that:
diff --git a/docs/astro/src/content/docs/zh-cn/extending/built-in-agent.mdx b/docs/astro/src/content/docs/zh-cn/extending/built-in-agent.mdx
index 62d8adfd4..abddd4093 100644
--- a/docs/astro/src/content/docs/zh-cn/extending/built-in-agent.mdx
+++ b/docs/astro/src/content/docs/zh-cn/extending/built-in-agent.mdx
@@ -31,8 +31,9 @@ import { Aside, Steps } from '@astrojs/starlight/components';
厂商下次发布为止,而写了个已经不存在的名字并不会被拒绝——接口会悄悄改用一个
更小的模型顶上,并且一整个会话都是如此。
-3. 或者填**自定义提供方**——任何接口都行。先选它讲哪种**协议**,
- 再填地址和模型名。
+3. 或者在**你的提供方**里添加自己的——任何接口都行,有几个加几个。每一行
+ 都和内置的提供方一样:名称、讲哪种**协议**、地址、模型名,以及它自己的密钥。
+ 展开一行还能说明这个接口**能做什么**。
4. 关掉设置。模型选择器就在你打字的那个框旁边——它是**逐条消息**的选择,
而不是埋在对话框里的偏好设置。
@@ -52,6 +53,21 @@ import { Aside, Steps } from '@astrojs/starlight/components';
只交给那个必须发送它的客户端。
</Aside>
+## 接口能做什么
+
+内置的提供方已经自己写好了。你自己加的那些,编辑器没有办法探出来,而猜错的
+两个方向代价都不小——所以一行的展开项就是你告诉它的地方。没说的部分不会被
+发现,只会被绕开。
+
+- **接受图片**——Agent 会截图来检查自己干的活。发不出去图片时,它会改成把
+ 同一帧要成粗颗粒的文字色块网格,并在对话里说明。默认关闭:给一个其实看不了
+ 图的接口打开它,代价是整轮请求被拒;而不打开,只是少了些它多半能不依赖的细节。
+- **上下文窗口**——一段对话能长到多大,才开始把最早的几轮折叠掉。留空按一个
+ 保守值算,会把本来还有余量的对话提前折叠;写得太大则是整轮请求直接失败,
+ 而且是在你等完之后。
+- **接受思考深度参数**——请求里可不可以带上上面设的那个深度。没听说过这个参数
+ 的接口会拒掉整次调用,而不是忽略它——遇到这种就把它关掉。
+
## 问它点什么
打开命令面板(`Ctrl/Cmd+Shift+P`)执行 **Agent:打开**。或者干脆跳过这一步: