Skip to content

🤖 Open-source, local-first alternative to Grok Bot, OpenAI dots and Claude Duet. Your own AI bots on any model: GPT, Claude, Gemini, Grok, DeepSeek, Kimi, GLM and 70+ more through one Modelflare key. Reach them in the app, WeChat or Feishu; they remember you, work in the background, run commands and use your computer. Pay per use.

feat(media): 生图生视频先确认模型,按 Modelflare 文档实时传参

dyk982505d7cbf2b3
diff --git a/README.md b/README.md
index 8d21772..bb6b20d 100644
--- a/README.md
+++ b/README.md
@@ -80,7 +80,9 @@ docker exec amb cat /data/ui-token # 打开 http://<主机>:3600/?token=<令
- 余额:侧栏左下角和 设置 → Modelflare 显示账户钱包余额(只用 API Key 时读 `/v1/dashboard/billing/credit_grants`);「充值」打开 Modelflare 网页,回到窗口时自动刷新。余额低于提醒线(默认 $1,可改或关闭)时在应用里提醒一次,也可以设置让机器人发到你的微信 / 飞书。
- 登录后还能在 设置 → Modelflare 查看按天、按模型的用量,Modelflare 发来的通知,以及复制邀请链接。
- 机器人预算(需要登录):机器人设置 → 预算,开启后这个机器人用自己的 Key(内置网关里单独的 provider),最多花设定的美元数;关闭后回到共用的 Key。
-- 生图 / 生视频:代理模式下机器人有 `generate_image`、`generate_video` 工具,由本程序带着该机器人用的 Key 调 Modelflare,结果存进机器人工作区的 `media/`,再交付到聊天。
+- 生图 / 生视频:代理模式下机器人有 `media_models`(可用模型、默认模型、参数范围、参考价)、`media_docs`(从 docs.modelflare.dev 实时读该模型的请求格式)、`generate_image`、`generate_video` 四个工具,并自带一份 media 技能说明。各模型的参数格式不同,机器人按文档写请求体,工作区里的参考图写成 `workspace:路径` 即可;Key 始终只在本程序里,由本程序带着该机器人用的 Key 调 Modelflare,结果存进机器人工作区的 `media/`,再交付到聊天。
+- 生成前确认:每次生成图片或视频之前都会在聊天里问你这次用哪个模型(附提示词、参数和参考价),可以确认、换成别的模型或取消;微信回复编号或模型名,飞书 / 网页点选。10 分钟内没回答就不生成。
+- 默认图片 / 视频模型:首次设置时和默认文本模型一起选,之后在 设置 → Modelflare 修改;选「自动」时用推荐顺序里第一个不需要参考素材的模型。
- 模型选择器里显示上下文长度、视觉 / 工具 / 推理能力,以及每百万 tokens 输入 / 输出的起步参考价(来自公开价格目录,以账单为准)。
- 调用归因:经 Modelflare 的请求都带 `X-Modelflare-Skill: anymodel-bot/<版本>`,链接都带 `utm_source=anymodelbot`。
- 线路:默认国际线路 `https://modelflare.dev/v1`;中国大陆可在「高级」或 设置 → Modelflare → 线路 选「中国大陆优化」(`https://origin.modelflare.dev/v1`)。线路由你选定,不自动切换。
diff --git a/dsh-runtime/plugins/amb-media.mjs b/dsh-runtime/plugins/amb-media.mjs
index b534b9f..b387e91 100644
--- a/dsh-runtime/plugins/amb-media.mjs
+++ b/dsh-runtime/plugins/amb-media.mjs
@@ -1,9 +1,10 @@
// anymodel bot image / video generation tools (MIT, part of anymodel bot).
//
-// Registers generate_image and generate_video. The agent never holds a Modelflare key: each call goes
-// to the host over the loopback bridge (AMB_BRIDGE_URL / AMB_BRIDGE_TOKEN), which calls Modelflare with
-// this bot's key and saves the result under media/ in the workspace. The agent then delivers the file
-// with the present tool.
+// Registers media_models, media_docs, generate_image and generate_video. The agent never holds a
+// Modelflare key: each call goes to the host over the loopback bridge (AMB_BRIDGE_URL / AMB_BRIDGE_TOKEN),
+// which asks the user to confirm the model, calls Modelflare with this bot's key and saves the result
+// under media/ in the workspace. The agent then delivers the file with the present tool. Every model has
+// its own request format, so the agent reads the model's live docs with media_docs and passes the body.
import { defineTool } from '@deepseek-ai/dsh-tools'
export const name = 'amb-media'
@@ -33,13 +34,41 @@ export function apply(ctx) {
return { text: String(r.text ?? '') }
}
+ const BODY = {
+ type: 'object',
+ additionalProperties: true,
+ description: "Request body exactly as the model's media_docs page shows it, without model. Write local files as \"workspace:<path>\" (sent as a data URL) or \"workspace-base64:<path>\" (raw Base64, e.g. Gemini inline_data.data). Omit for a plain text-only request.",
+ }
+
+ ctx.tools.register(defineTool({
+ name: 'media_models',
+ description: 'List the image and video models this bot may use on Modelflare, with the default model, parameter ranges (sizes, ratios, resolutions, seconds, reference media) and reference prices. Free; call it before choosing a model.',
+ parameters: {
+ kind: { type: 'string', enum: ['image', 'video'], description: 'Only this kind. Omit for both.' },
+ },
+ output: TEXT_OUT,
+ execute: (args, exec) => run('models', args, exec),
+ }))
+
+ ctx.tools.register(defineTool({
+ name: 'media_docs',
+ description: "Read the live documentation page of one image or video model from docs.modelflare.dev: endpoint, exact request body, parameter table, reference-media fields and limits. Every model has its own format; read it before passing a body to generate_image or generate_video.",
+ parameters: {
+ model: { type: 'string', required: true, description: 'Exact model id, e.g. kling-3.0-turbo.' },
+ },
+ output: TEXT_OUT,
+ execute: (args, exec) => run('docs', args, exec),
+ }))
+
ctx.tools.register(defineTool({
name: 'generate_image',
- description: 'Generate an image from a text description with an image model on Modelflare. The image is saved in the workspace under media/; deliver it with the present tool when the user should receive it. Write the prompt in detail (subject, style, composition, text in the image).',
+ description: 'Generate or edit an image with an image model on Modelflare. Before anything is generated the user is asked to confirm the model; they may pick another one or cancel. The image is saved in the workspace under media/; deliver it with the present tool when the user should receive it. Write the prompt in detail (subject, style, composition, text in the image).',
parameters: {
- prompt: { type: 'string', required: true, description: 'What the image should show, in detail.' },
- model: { type: 'string', description: 'Image model id. Omit to use the first one this bot may use.' },
- size: { type: 'string', description: 'Optional size such as 1024x1024, for models on the Images API.' },
+ prompt: { type: 'string', required: true, description: 'What the image should show, in detail. Shown to the user in the confirmation; keep it in sync with the prompt inside body.' },
+ model: { type: 'string', description: 'Image model id. Omit to use the default model.' },
+ body: BODY,
+ endpoint: { type: 'string', enum: ['generations', 'edits'], description: 'For OpenAI Images models only: "edits" when the docs say reference images go to /v1/images/edits. Default generations.' },
+ size: { type: 'string', description: 'Optional size such as 1024x1024 for a plain request without body, Images API models only.' },
},
output: TEXT_OUT,
execute: (args, exec) => run('image', args, exec),
@@ -47,12 +76,13 @@ export function apply(ctx) {
ctx.tools.register(defineTool({
name: 'generate_video',
- description: 'Generate a short video from a text description with a video model on Modelflare. It can take several minutes. The video is saved in the workspace under media/; deliver it with the present tool. Only use it when the user asks for a video: video generation costs much more than images.',
+ description: 'Generate a short video with a video model on Modelflare. Before anything is generated the user is asked to confirm the model; they may pick another one or cancel. It can take several minutes. The video is saved in the workspace under media/; deliver it with the present tool. Only use it when the user asks for a video: video generation costs much more than images.',
parameters: {
- prompt: { type: 'string', required: true, description: 'What the video should show, in detail.' },
- model: { type: 'string', description: 'Video model id. Omit to use the first one this bot may use.' },
- seconds: { type: 'number', description: 'Optional length in seconds, if the model supports it.' },
- size: { type: 'string', description: 'Optional size such as 1280x720, if the model supports it.' },
+ prompt: { type: 'string', required: true, description: 'What the video should show, in detail. Shown to the user in the confirmation; keep it in sync with the prompt inside body.' },
+ model: { type: 'string', description: 'Video model id. Omit to use the default model.' },
+ body: BODY,
+ seconds: { type: 'number', description: 'Optional length in seconds for a plain request without body.' },
+ size: { type: 'string', description: 'Optional size such as 1280x720 for a plain request without body.' },
},
output: TEXT_OUT,
execute: (args, exec) => run('video', args, exec),
diff --git a/src/agent/host.ts b/src/agent/host.ts
index fbdced0..e457c66 100644
--- a/src/agent/host.ts
+++ b/src/agent/host.ts
@@ -135,6 +135,8 @@ export interface AgentHostOptions {
turnTimeoutMs?: number;
/** 需要让用户看到的提示(界面通知) */
onNotice?: (text: string) => void;
+ /** 能生图 / 生视频(已连接 Modelflare):写入 media 技能说明 */
+ media?: boolean;
}
export type HostState = "stopped" | "starting" | "running" | "backoff" | "error";
@@ -315,6 +317,7 @@ export class AgentHost {
const lim = MODEL_LIMITS.find(([re]) => re.test(model)) ?? [/./, 128_000, 16_000];
const plugin = path.join(this.o.runtimeDir, "plugins", "amb-bridge.mjs");
const memory = path.join(this.o.runtimeDir, "plugins", "amb-memory.mjs");
+ this.writeMediaSkill();
const text = [
"# 由 anymodel bot 每次启动生成;请勿手改。",
"- id: llm-pi-ai",
@@ -341,7 +344,7 @@ export class AgentHost {
` name: ${yamlStr(path.join(this.o.runtimeDir, "plugins", "amb-permission.mjs"))}`,
" - id: amb-memory",
` name: ${yamlStr(memory)}`,
- // 生图 / 生视频:宿主带着本机器人的 Key 调 Modelflare,文件存进工作区 media/
+ // 生图 / 生视频:宿主带着本机器人的 Key 调 Modelflare,文件存进工作区 media/;技能说明在 skills/media
" - id: amb-media",
` name: ${yamlStr(path.join(this.o.runtimeDir, "plugins", "amb-media.mjs"))}`,
// sdk 配置默认不带这两个工具:ask_user_question 走 user-questions → 插件 → 审批中心;
@@ -358,6 +361,20 @@ export class AgentHost {
return f;
}
+ /** media 技能:怎么挑模型、读文档、写请求体、交付结果。没有连接 Modelflare 时删掉。 */
+ private writeMediaSkill(): void {
+ const dir = path.join(this.home, "skills", "media");
+ try {
+ if (!this.o.media) fs.rmSync(dir, { recursive: true, force: true });
+ else {
+ fs.mkdirSync(dir, { recursive: true });
+ fs.writeFileSync(path.join(dir, "SKILL.md"), MEDIA_SKILL);
+ }
+ } catch (e) {
+ log.warn("media skill setup failed", { err: String(e) });
+ }
+ }
+
/** 当前机器上本机器人所选后端的可用性(不看本机器人的开关)。 */
computerUseStatus(): CuResolved {
return resolveComputerUse(this.settings().computerUse.backend, this.o.runtimeDir);
@@ -868,3 +885,30 @@ Tools are named \`mcp__computer-use__<tool>\`. Actions run automatically; no app
- On Linux, reading window contents needs AT-SPI; on Wayland only the foreground window can be driven. On macOS the app needs Accessibility and Screen Recording permission; if \`check_permissions\` reports them missing, tell the user how to grant them.
`;
+
+/** 生图 / 生视频的技能说明(由 anymodel bot 编写;不含 Key、不让代理自己调接口,参数格式以 Modelflare 文档为准)。 */
+const MEDIA_SKILL = `---
+name: media
+description: Generate or edit images and generate videos with Modelflare image and video models through the media_models, media_docs, generate_image and generate_video tools. Use when the user asks to draw, create, edit or restyle a picture, animate an image, or make a video.
+---
+
+# Images and videos (Modelflare)
+
+You have no Modelflare key and must not call the API yourself (no curl, no scripts): the tools call it with this bot's key. Every generation costs the user money, so the user confirms the model before anything is generated.
+
+## Steps
+1. \`media_models\` (kind \`image\` or \`video\`) lists the models this bot may use, the default one, their parameter ranges and reference prices. Use the default unless the user named a model or the request needs something the default cannot do (a reference image, a ratio or length it does not support).
+2. Plain text-to-image or text-to-video with default settings: call \`generate_image\` / \`generate_video\` with a detailed \`prompt\` only.
+3. Anything else (ratio, resolution, size, length, quality, audio, reference images, first/last frames, editing an image): call \`media_docs\` for the chosen model, then pass \`body\` in exactly the format its page shows, without \`model\`. Every family is different (for example Seedance uses \`content[]\`, Kling \`contents\` and \`settings\`, Wan \`input\` and \`parameters\`, Gemini \`contents[].parts[]\` with \`generationConfig\`), so never reuse field names from another model's page. For OpenAI Images models whose page sends references to \`/v1/images/edits\`, also pass \`endpoint: "edits"\`. Keep \`prompt\` the same as the prompt inside \`body\`: the user sees \`prompt\` in the confirmation.
+4. Reference media: a file in the workspace is written as \`"workspace:media/x.png"\` (sent as a data URL) or \`"workspace-base64:media/x.png"\` (raw Base64, e.g. Gemini \`inline_data.data\`), up to 20 MB each. A public https URL can be used as is. Models marked 必须带参考素材 (image-to-video, reference-to-video) only work with references; many models accept reference videos only as public URLs, see the docs. Images the user sent in chat are not in the workspace: if you have no file or link for a reference, ask the user for one.
+5. The tool result says what happened:
+ - Generated: the file is under \`media/\`; deliver it with \`present\`.
+ - The user picked another model while you had passed a \`body\`: read that model's page with \`media_docs\`, rewrite \`body\` for it and call again with \`model\` set to it. That call is not confirmed again.
+ - The user answered something else: adjust as they said and call again (they will be asked again).
+ - Cancelled or not confirmed in time: nothing was generated or charged; do not retry on your own.
+6. Videos take minutes and cost much more than images: only make a video when the user asks for one, and do not generate extra variants unless asked.
+
+## Errors
+- 这把 Key 不能用这个模型 (model_not_found): this account cannot use that model; pick another one from \`media_models\`.
+- Any other 4xx names the problem: fix the body according to the model's page instead of sending the same request again. If the balance is empty, tell the user to top up on Modelflare.
+`;
diff --git a/src/agent/manager.ts b/src/agent/manager.ts
index a6d373a..4bfb2f0 100644
--- a/src/agent/manager.ts
+++ b/src/agent/manager.ts
@@ -4,8 +4,8 @@ import { logger } from "../log.js";
import { AgentBridge } from "./bridge.js";
import path from "node:path";
import { AgentHost, callDetail, type TurnObserver, type TurnUsage } from "./host.js";
-import type { MediaService } from "./media.js";
-import { ApprovalCenter, type QuestionItem } from "./approvals.js";
+import type { MediaModel, MediaPlan, MediaService } from "./media.js";
+import { APPROVAL_TTL_MS, ApprovalCenter, type QuestionItem } from "./approvals.js";
import { MemoryError, MemoryStore, type Scope } from "./memory.js";
import { DEFAULT_SETTINGS, type BotSettings } from "./settings.js";
import {
@@ -41,6 +41,9 @@ export interface AgentManagerOptions {
onNotice?: (text: string) => void;
}
+/** 生成确认里"取消"那一项 */
+const CANCEL = "取消";
+
function deferred<T>() {
let resolve!: (v: T) => void;
const promise = new Promise<T>((r) => (resolve = r));
@@ -67,6 +70,8 @@ export class AgentManager {
private compactDone = new Map<string, (v: unknown) => void>();
private judgedAt = new Map<string, number>();
private reviewTimers = new Map<string, NodeJS.Timeout>();
+ /** 会话 → 用户在生成确认里换了模型、代理要按新模型文档重写请求:重写后的那一次免确认 */
+ private mediaGrants = new Map<string, { kind: string; model: string; at: number }>();
/** 最近一次管家/整理结果(状态页展示) */
readonly stewardLog: Array<{ at: number; conversationId: string; action: string; detail: string }> = [];
@@ -84,6 +89,11 @@ export class AgentManager {
this.memory = new MemoryStore(path.join(o.dataDir, "memory"));
}
+ /** 生图 / 生视频服务;未接内置网关时没有 */
+ get media(): MediaService | undefined {
+ return this.o.media;
+ }
+
/** 是否可用(已启用且找到 DSH 运行时)。 */
get available(): boolean {
return this.o.enabled && !!this.o.runtimeDir;
@@ -106,6 +116,7 @@ export class AgentManager {
turnTimeoutMs: this.o.turnTimeoutMs,
settings: () => this.settings(botId),
onNotice: this.o.onNotice,
+ media: !!this.o.media,
});
this.hosts.set(botId, h);
}
@@ -338,20 +349,90 @@ export class AgentManager {
return res;
}
- /** 插件 → 宿主:remember / forget / recall。 */
- /** 生图 / 生视频:只在该机器人有回合进行中时受理(工具调用只会发生在回合里) */
+ /**
+ * 生图 / 生视频:只在该机器人有回合进行中时受理(工具调用只会发生在回合里)。
+ * models / docs 直接回答;image / video 每次都先向用户确认这次用哪个模型,确认后才花钱。
+ */
private async mediaOp(b: any): Promise<{ text?: string; error?: string }> {
const botId = String(b?.botId ?? "");
if (!this.hosts.has(botId) || !this.activeConvs.get(botId)?.size) return { error: "unknown bot" };
- if (!this.o.media) return { error: "生成图片和视频需要连接 Modelflare" };
+ const media = this.o.media;
+ if (!media) return { error: "生成图片和视频需要连接 Modelflare" };
+ const op = String(b?.op ?? "");
+ const args = (b?.args ?? {}) as Record<string, unknown>;
try {
- return { text: await this.o.media.run(botId, String(b?.op ?? ""), (b?.args ?? {}) as Record<string, unknown>) };
+ if (op === "models") return { text: await media.listText(botId, args.kind) };
+ if (op === "docs") return { text: await media.docs(args.model) };
+ const plan = await media.prepare(botId, op, args);
+ const sessionId = String(b?.sessionId ?? "");
+ const conv = this.resolveConversation(botId, sessionId);
+ if (!conv) return { error: "找不到这次对话,没法向用户确认,没有生成" };
+ const tool = plan.kind === "image" ? "generate_image" : "generate_video";
+ let model = plan.model;
+ const grant = this.mediaGrants.get(conv);
+ if (grant?.kind === plan.kind && grant.model === model.id && Date.now() - grant.at < (this.o.approvalTtlMs ?? APPROVAL_TTL_MS)) {
+ this.mediaGrants.delete(conv);
+ } else {
+ const r = await this.confirmMedia(botId, conv, sessionId, plan, tool);
+ if (r.cancel) return { text: "用户取消了这次生成(或没有及时确认),没有生成文件,也没有扣费。不要自行重试;需要的话问用户想怎么改。" };
+ if (r.reply !== undefined) return { text: `用户没有确认,而是回复:「${r.reply}」。按用户的意思调整后再调用 ${tool}(会再次确认)。` };
+ if (r.model && r.model.id !== model.id) {
+ if (plan.custom) {
+ // 请求体是按原模型的文档写的,换了模型就得按新模型的文档重写;重写后的那一次不再确认
+ this.mediaGrants.set(conv, { kind: plan.kind, model: r.model.id, at: Date.now() });
+ return { text: `用户选择改用 ${r.model.id}。各模型的请求格式不同:先用 media_docs 查 ${r.model.id} 的文档,按它重写 body,再调用 ${tool}(model 填 ${r.model.id})。这一次不会再要求确认。` };
+ }
+ model = r.model;
+ }
+ }
+ return { text: await media.execute(botId, model, args) };
} catch (e) {
- log.warn("media tool failed", { bot: botId, message: (e as Error).message.slice(0, 300) });
+ log.warn("media tool failed", { bot: botId, op, message: (e as Error).message.slice(0, 300) });
return { error: (e as Error).message };
}
}
+ /** 生成前的确认:问用户这次用哪个模型(本次的、默认的和几个推荐的,或回复别的模型名),等到回答、取消或过期 */
+ private async confirmMedia(botId: string, conv: string, sessionId: string, plan: MediaPlan, tool: string): Promise<{ model?: MediaModel; cancel?: boolean; reply?: string }> {
+ const name = plan.kind === "image" ? "图片" : "视频";
+ const opt = (m: MediaModel, tag = "") => ({ label: m.id, description: [tag, m.price].filter(Boolean).join(" · ") || undefined });
+ const q: QuestionItem = {
+ id: "model",
+ header: `生成${name}`,
+ question: `这次用 ${plan.model.id} 生成${name}吗?`,
+ detail: `${plan.summary}\n也可以直接回复其他模型名。`,
+ options: [opt(plan.model, plan.model === plan.default ? "默认" : "机器人选的"), ...plan.candidates.map((m) => opt(m, m === plan.default ? "默认" : "")), { label: CANCEL, description: "这次不生成" }],
+ };
+ const row = this.approvals.create({ kind: "question", botId, conversationId: conv, sessionId, tool, detail: q.question, reason: "", payload: JSON.stringify([q]) });
+ const deadline = Date.now() + (this.o.approvalTtlMs ?? APPROVAL_TTL_MS) + 10_000;
+ let cur = row;
+ while (cur.status === "pending") {
+ if (Date.now() > deadline) {
+ this.approvals.cancel(row.id, "timeout");
+ return { cancel: true };
+ }
+ const next = await this.approvals.wait(row.id, 60_000);
+ if (!next) return { cancel: true };
+ cur = next;
+ }
+ if (cur.status !== "answered") return { cancel: true };
+ let a: { selected?: string[]; custom?: string } | undefined;
+ try {
+ a = (JSON.parse(cur.result) as { answers?: Array<{ selected?: string[]; custom?: string }> }).answers?.[0];
+ } catch {
+ a = undefined;
+ }
+ const pick = (a?.selected?.[0] ?? "").trim();
+ const text = (a?.custom ?? "").trim();
+ if (pick === CANCEL || /^(取消|不要?|不用|算了|no?|cancel)$/i.test(text)) return { cancel: true };
+ if (!pick && /^(好的?|行|可以|确认|确定|是的?|对|嗯|ok(ay)?|y(es)?|生成吧?|就用它)$/i.test(text)) return { model: plan.model };
+ const want = (pick || text).replace(/^(用|换成|改用)\s*/, "");
+ const hit = plan.all.find((m) => m.id.toLowerCase() === want.toLowerCase());
+ if (hit) return { model: hit };
+ return text ? { reply: text } : { cancel: true };
+ }
+
+ /** 插件 → 宿主:remember / forget / recall。 */
private async memoryOp(b: any): Promise<{ text?: string; error?: string }> {
const botId = String(b?.botId ?? "");
if (!this.hosts.has(botId)) return { error: "unknown bot" };
@@ -423,6 +504,7 @@ export class AgentManager {
resetConversation(conversationId: string): void {
this.o.store.clearAgentSession(conversationId);
this.o.store.setKV(`seed:${conversationId}`, "");
+ this.mediaGrants.delete(conversationId);
for (const r of this.approvals.pending(conversationId)) this.approvals.cancel(r.id, "reset");
}
diff --git a/src/agent/media.ts b/src/agent/media.ts
index 495943d..30b21c3 100644
--- a/src/agent/media.ts
+++ b/src/agent/media.ts
@@ -1,9 +1,12 @@
/**
- * 生图 / 生视频:代理的 generate_image / generate_video 工具经内部 bridge 调到这里,由宿主带着
- * 该机器人用的 Key 直连 Modelflare(代理子进程拿不到任何密钥),结果存进机器人工作区的 media/,
- * 再由代理用 present 交付到聊天。
- * - 模型取自 GET /v1/models?scope=authorized&kind=image|video(这把 Key 真正能用的)。
- * - 图片:Gemini 图片模型走 /v1/models/{id}:generateContent,其余走 /v1/images/generations。
+ * 生图 / 生视频:代理的 media_models / media_docs / generate_image / generate_video 工具经内部 bridge 调到这里,
+ * 由宿主带着该机器人用的 Key 直连 Modelflare(代理子进程拿不到任何密钥),结果存进机器人工作区的 media/,
+ * 再由代理用 present 交付到聊天。每次生成前都由 AgentManager 向用户确认这次用哪个模型(manager.ts mediaOp)。
+ * - 模型取自 GET /v1/models?scope=authorized&kind=image|video(这把 Key 真正能用的);参数范围、是否必须带参考素材、
+ * 参考价、推荐顺序取自公开价格目录 /api/pricing。
+ * - 每个模型的请求格式都不一样,以 docs.modelflare.dev 上该模型的文档为准:代理先用 media_docs 读文档,按文档写
+ * 请求体(body);宿主只填 model、把 "workspace:路径" 换成文件内容、发请求、保存结果。不带 body 时发最简单的文字请求。
+ * - 图片:Gemini 图片模型走 /v1/models/{id}:generateContent,其余走 /v1/images/generations(改图走 /v1/images/edits)。
* - 视频:POST /v1/videos 建任务 → 轮询 → GET /v1/videos/{id}/content 下载。
*/
import fs from "node:fs";
@@ -12,6 +15,8 @@ import { logger } from "../log.js";
const log = logger("agent.media");
+export type MediaKind = "image" | "video";
+
export interface MediaOptions {
/** Modelflare /v1 基址(随线路) */
baseUrl: () => string;
@@ -20,37 +25,192 @@ export interface MediaOptions {
/** 额外请求头(调用归因) */
headers: () => Record<string, string>;
workspace: (botId: string) => string;
+ /** 设置里选的默认图片 / 视频模型;空 = 按推荐顺序的第一个 */
+ defaults?: () => { image: string; video: string };
+ /** 公开价格目录(参数范围、参考价、推荐顺序);取不到时只是少了这些信息 */
+ pricingUrl?: () => string;
+ /** 文档全文(每个模型一节) */
+ docsUrl?: string;
fetchImpl?: typeof fetch;
/** 视频轮询间隔与上限(测试用) */
pollMs?: number;
videoTimeoutMs?: number;
}
-interface MediaModel {
+export interface MediaModel {
id: string;
- kind?: string;
- image_route?: string;
+ kind: MediaKind;
+ /** images = OpenAI Images API;gemini = generateContent;videos = /v1/videos */
+ route: "images" | "gemini" | "videos";
+ /** 价格目录里的 media_options.image / media_options.video */
+ options?: Record<string, unknown>;
+ /** 必须带参考素材(图生视频之类),只有文字时不能用 */
+ requiresReference: boolean;
+ /** 一句参考价,如 "$0.03/张"、"$0.09/秒(720p)" */
+ price: string;
+ priority: number;
+}
+
+/** 一次生成要确认的内容 */
+export interface MediaPlan {
+ kind: MediaKind;
+ /** 这次要用的模型(代理指定的,或默认) */
+ model: MediaModel;
+ /** 默认模型(设置里选的,或推荐的) */
+ default: MediaModel;
+ /** 确认时一并列出的其他选择 */
+ candidates: MediaModel[];
+ /** 这把 Key 能用的全部同类模型 */
+ all: MediaModel[];
+ /** 代理按文档写了自己的请求体:换模型后需要按新模型的文档重写 */
+ custom: boolean;
+ /** 给用户看的说明:提示词摘要、参数、参考素材 */
+ summary: string;
+}
+
+interface CatalogEntry {
+ model_name: string;
+ display_priority?: number;
+ quota_type?: number;
+ model_price?: number;
+ billing_unit?: string;
+ enable_groups?: string[];
+ video_resolution_price?: Record<string, number>;
+ media_options?: { image?: Record<string, unknown>; video?: Record<string, unknown> };
+}
+
+interface Catalog {
+ data?: CatalogEntry[];
+ group_ratio?: Record<string, number>;
+ usable_group?: Record<string, unknown>;
}
const MAX_BYTES = 200 * 1024 * 1024;
+/** 参考素材按 data URL / Base64 嵌进请求,单个文件上限 */
+const MAX_REFERENCE_BYTES = 20 * 1024 * 1024;
+const CACHE_MS = 3600_000;
+const DOCS_URL = "https://docs.modelflare.dev";
const EXT: Record<string, string> = { "image/png": "png", "image/jpeg": "jpg", "image/webp": "webp", "image/gif": "gif", "video/mp4": "mp4", "video/webm": "webm" };
+const MIME: Record<string, string> = {
+ ".png": "image/png",
+ ".jpg": "image/jpeg",
+ ".jpeg": "image/jpeg",
+ ".webp": "image/webp",
+ ".gif": "image/gif",
+ ".bmp": "image/bmp",
+ ".mp4": "video/mp4",
+ ".mov": "video/quicktime",
+ ".webm": "video/webm",
+ ".mp3": "audio/mpeg",
+ ".wav": "audio/wav",
+};
+/** 请求体里的 "workspace:media/a.png"(data URL)或 "workspace-base64:media/a.png"(纯 Base64) */
+const FILE_REF = /^workspace(-base64)?:(.+)$/;
export class MediaError extends Error {}
+const num = (n: number) => String(+n.toFixed(n < 0.1 ? 4 : 3));
+
+/** 推荐顺序:价格目录的 display_priority 小的在前,必须带参考素材的放后面,其余按名字 */
+export function sortModels(list: MediaModel[]): MediaModel[] {
+ return [...list].sort((a, b) => a.priority - b.priority || Number(a.requiresReference) - Number(b.requiresReference) || a.id.localeCompare(b.id, "en", { sensitivity: "base" }));
+}
+
+/** 设置里的默认模型还能用就用它,否则推荐顺序里第一个不需要参考素材的 */
+export function pickDefault(list: MediaModel[], configured: string): MediaModel | undefined {
+ return list.find((m) => m.id === configured) ?? list.find((m) => !m.requiresReference) ?? list[0];
+}
+
+/** 一句参考价;复杂计费(按 token)只说明计费方式 */
+export function priceNote(e: CatalogEntry | undefined, catalog: Catalog): string {
+ if (!e) return "";
+ const ratios = (e.enable_groups ?? [])
+ .filter((g) => !catalog.usable_group || g in catalog.usable_group)
+ .map((g) => catalog.group_ratio?.[g])
+ .filter((r): r is number => typeof r === "number" && Number.isFinite(r) && r >= 0);
+ const factor = ratios.length ? Math.min(...ratios) : 1;
+ const perSecond = Object.entries(e.video_resolution_price ?? {}).filter(([, p]) => typeof p === "number" && p > 0);
+ if (perSecond.length) {
+ const want = String(e.media_options?.video?.default_resolution ?? "");
+ const [res, p] = perSecond.find(([r]) => r === want) ?? perSecond.sort((a, b) => a[1] - b[1])[0];
+ return `$${num(p * factor)}/秒(${res})`;
+ }
+ if (e.quota_type === 1 && typeof e.model_price === "number" && e.model_price > 0) return `$${num(e.model_price * factor)}/${e.billing_unit === "per_call" ? "次" : "张"}`;
+ if (e.billing_unit === "token") return "按 token 计费";
+ return "";
+}
+
+/** 一行参数范围:尺寸、比例、分辨率、时长、参考素材 */
+export function optionsNote(m: MediaModel): string {
+ const o = m.options ?? {};
+ const list = (k: string) => (Array.isArray(o[k]) ? (o[k] as unknown[]).map(String) : []);
+ const parts: string[] = [];
+ if (list("sizes").length) parts.push(`尺寸 ${list("sizes").join("/")}`);
+ if (list("resolutions").length) parts.push(`分辨率 ${list("resolutions").join("/")}`);
+ if (list("ratios").length) parts.push(`比例 ${list("ratios").join("/")}`);
+ const secs = list("seconds").map(Number).filter((n) => n > 0);
+ if (secs.length) parts.push(`时长 ${Math.min(...secs)}–${Math.max(...secs)} 秒`);
+ const refs = list("reference_media");
+ if (refs.length) parts.push(`参考素材 ${refs.join("/")}`);
+ else if (o.reference_images === true || Number(o.max_references) > 0) parts.push(`参考图${Number(o.max_references) > 0 ? ` 最多 ${o.max_references} 张` : ""}`);
+ if (m.requiresReference) parts.push("必须带参考素材");
+ return parts.join(" · ");
+}
+
+/** 从请求体里挑出给用户看的参数(分辨率、比例、时长……),并数一数参考素材 */
+export function describeBody(body: Record<string, unknown> | undefined): string {
+ if (!body) return "";
+ const keys: Record<string, string> = { resolution: "", aspect_ratio: "", ratio: "", size: "", imageSize: "", aspectRatio: "", seconds: "秒", duration: "秒", quality: "" };
+ const seen = new Map<string, string>();
+ let refs = 0;
+ const walk = (v: unknown, depth: number) => {
+ if (typeof v === "string") {
+ if (FILE_REF.test(v) || /^https?:\/\//i.test(v) || v.startsWith("data:")) refs++;
+ return;
+ }
+ if (!v || typeof v !== "object" || depth > 6) return;
+ for (const [k, x] of Object.entries(v as Record<string, unknown>)) {
+ if (k in keys && (typeof x === "string" || typeof x === "number")) {
+ // 时长两种写法只显示一个
+ const label = k === "duration" ? "seconds" : k === "aspectRatio" || k === "aspect_ratio" ? "ratio" : k === "imageSize" ? "size" : k;
+ if (!seen.has(label)) seen.set(label, `${x}${keys[k] && Number(x) > 0 ? ` ${keys[k]}` : ""}`);
+ } else walk(x, depth + 1);
+ }
+ };
+ walk(body, 0);
+ const parts = [...seen.values()];
+ if (refs) parts.push(`${refs} 个参考素材`);
+ return parts.join(" · ");
+}
+
+/** 从文档全文里取出某个模型的那一节(每节以 "# 标题" 开头、下一行是 "Source: <页面地址>") */
+export function docSection(full: string, docsUrl: string, kind: MediaKind, model: string): string {
+ const slug = model.toLowerCase().replace(/\./g, "-");
+ const marker = `\nSource: ${docsUrl}/models/${kind}/${slug}/\n`;
+ const at = full.indexOf(marker);
+ if (at < 0) return "";
+ const start = full.lastIndexOf("\n# ", at);
+ const next = full.indexOf("\n---\n\n# ", at + marker.length);
+ return full.slice(start < 0 ? at : start + 1, next < 0 ? undefined : next).trim();
+}
+
export class MediaService {
+ private catalogCache: { at: number; data: Catalog } | undefined;
+ private docsCache: { at: number; text: string } | undefined;
+
constructor(private o: MediaOptions) {}
private get fetch(): typeof fetch {
return this.o.fetchImpl ?? fetch;
}
+ private get docsUrl(): string {
+ return (this.o.docsUrl ?? DOCS_URL).replace(/\/+$/, "");
+ }
- /** bridge 入口:返回给模型的一段文字 */
+ /** 直接生成(不经确认,测试与内部用):解析模型后执行 */
async run(botId: string, op: string, args: Record<string, unknown>): Promise<string> {
- const prompt = String(args.prompt ?? "").trim();
- if (!prompt) throw new MediaError("缺少 prompt");
- if (op === "image") return this.image(botId, prompt, args);
- if (op === "video") return this.video(botId, prompt, args);
- throw new MediaError(`未知操作 ${op}`);
+ const plan = await this.prepare(botId, op, args);
+ return this.execute(botId, plan.model, args);
}
private key(botId: string): string {
@@ -59,10 +219,10 @@ export class MediaService {
return key;
}
- private async call(botId: string, method: string, p: string, body?: unknown, timeoutMs = 300_000): Promise<Response> {
+ private async call(key: string, method: string, p: string, body?: unknown, timeoutMs = 300_000): Promise<Response> {
const res = await this.fetch(`${this.o.baseUrl()}${p}`, {
method,
- headers: { Authorization: `Bearer ${this.key(botId)}`, ...this.o.headers(), ...(body !== undefined ? { "content-type": "application/json" } : {}) },
+ headers: { Authorization: `Bearer ${key}`, ...this.o.headers(), ...(body !== undefined ? { "content-type": "application/json" } : {}) },
body: body !== undefined ? JSON.stringify(body) : undefined,
signal: AbortSignal.timeout(timeoutMs),
});
@@ -74,24 +234,150 @@ export class MediaService {
return res;
}
- private async models(botId: string, kind: "image" | "video"): Promise<MediaModel[]> {
- const res = await this.call(botId, "GET", `/models?scope=authorized&kind=${kind}`, undefined, 20_000);
- const body = (await res.json()) as { data?: MediaModel[] };
- return (body.data ?? []).filter((m) => m.id && (!m.kind || m.kind === kind));
+ private async catalog(): Promise<Catalog> {
+ if (this.catalogCache && Date.now() - this.catalogCache.at < CACHE_MS) return this.catalogCache.data;
+ let data: Catalog = {};
+ const url = this.o.pricingUrl?.();
+ if (url) {
+ try {
+ const res = await this.fetch(url, { headers: { Accept: "application/json" }, signal: AbortSignal.timeout(10_000) });
+ if (res.ok) data = (await res.json()) as Catalog;
+ } catch (e) {
+ log.warn("pricing catalog unavailable", { message: (e as Error).message.slice(0, 200) });
+ }
+ }
+ this.catalogCache = { at: Date.now(), data };
+ return data;
+ }
+
+ /** 这把 Key 能用的图片 / 视频模型,按推荐顺序 */
+ async available(key: string, kind: MediaKind): Promise<MediaModel[]> {
+ const [res, catalog] = await Promise.all([this.call(key, "GET", `/models?scope=authorized&kind=${kind}`, undefined, 20_000), this.catalog()]);
+ const body = (await res.json()) as { data?: Array<{ id?: string; kind?: string; image_route?: string }> };
+ const entries = new Map((catalog.data ?? []).map((e) => [e.model_name, e]));
+ const out: MediaModel[] = [];
+ for (const m of body.data ?? []) {
+ if (!m.id || (m.kind && m.kind !== kind)) continue;
+ const route = kind === "video" ? "videos" : m.image_route === "gemini-generate-content" ? "gemini" : m.image_route === "images" ? "images" : undefined;
+ if (!route) continue;
+ const e = entries.get(m.id);
+ const options = e?.media_options?.[kind];
+ out.push({
+ id: m.id,
+ kind,
+ route,
+ options,
+ requiresReference: options?.requires_reference === true,
+ price: priceNote(e, catalog),
+ priority: typeof e?.display_priority === "number" ? e.display_priority : 1e9,
+ });
+ }
+ return sortModels(out);
+ }
+
+ async models(botId: string, kind: MediaKind): Promise<MediaModel[]> {
+ return this.available(this.key(botId), kind);
+ }
+
+ defaultId(kind: MediaKind): string {
+ return this.o.defaults?.()[kind] ?? "";
+ }
+
+ /** media_models 工具:列出可用模型、默认模型、参数范围与参考价 */
+ async listText(botId: string, kind: unknown): Promise<string> {
+ const kinds: MediaKind[] = kind === "image" || kind === "video" ? [kind] : ["image", "video"];
+ const blocks: string[] = [];
+ for (const k of kinds) {
+ const list = await this.models(botId, k);
+ const name = k === "image" ? "图片" : "视频";
+ if (!list.length) {
+ blocks.push(`这把 Key 没有可用的${name}模型。`);
+ continue;
+ }
+ const def = pickDefault(list, this.defaultId(k));
+ const lines = list.map((m) => `- ${m.id}${m === def ? "(默认)" : ""}${[m.price, optionsNote(m)].filter(Boolean).map((s) => ` — ${s}`).join("")}`);
+ blocks.push(`可用的${name}模型(按推荐顺序):\n${lines.join("\n")}`);
+ }
+ blocks.push("生成前会由程序向用户确认这次用哪个模型。要用参考图、比例、分辨率、时长等参数时,先用 media_docs 读该模型的文档,按文档写 body。");
+ return blocks.join("\n\n");
+ }
+
+ /** media_docs 工具:该模型在 docs.modelflare.dev 上的那一页(Markdown) */
+ async docs(model: unknown): Promise<string> {
+ const id = String(model ?? "").trim();
+ if (!id) throw new MediaError("缺少 model");
+ if (!this.docsCache || Date.now() - this.docsCache.at > CACHE_MS) {
+ const res = await this.fetch(`${this.docsUrl}/llms-full.txt`, { signal: AbortSignal.timeout(30_000) }).catch((e: Error) => {
+ throw new MediaError(`读取文档失败:${e.message}`);
+ });
+ if (!res.ok) throw new MediaError(`读取文档失败(HTTP ${res.status})`);
+ this.docsCache = { at: Date.now(), text: await res.text() };
+ }
+ const text = docSection(this.docsCache.text, this.docsUrl, "image", id) || docSection(this.docsCache.text, this.docsUrl, "video", id);
+ if (!text) throw new MediaError(`文档里没有 ${id} 的页面,模型列表见 ${this.docsUrl}/models/`);
+ const cap = 24_000;
+ return `${text.length > cap ? `${text.slice(0, cap)}\n…(已截断)` : text}\n\n---\n按上面的请求格式写 generate_* 工具的 body(不用写 model,也不要自己用 curl 调接口:你没有 Key,程序会代发)。本地文件写成 "workspace:相对路径"(换成 data URL)或 "workspace-base64:相对路径"(换成纯 Base64,例如 Gemini 的 inline_data.data)。`;
+ }
+
+ /** 解析这次要用的模型与确认时列出的其他选择 */
+ async prepare(botId: string, op: string, args: Record<string, unknown>): Promise<MediaPlan> {
+ if (op !== "image" && op !== "video") throw new MediaError(`未知操作 ${op}`);
+ const kind: MediaKind = op;
+ const prompt = String(args.prompt ?? "").trim();
+ if (!prompt) throw new MediaError("缺少 prompt");
+ const body = parseBody(args.body);
+ const all = await this.models(botId, kind);
+ if (!all.length) throw new MediaError(kind === "image" ? "这把 Key 没有可用的图片模型" : "这把 Key 没有可用的视频模型");
+ const name = String(args.model ?? "").trim();
+ const def = pickDefault(all, this.defaultId(kind))!;
+ const model = name ? all.find((m) => m.id === name) : def;
+ if (!model) throw new MediaError(`没有这个模型:${name}。可用:${all.map((m) => m.id).join("、")}`);
+ const hasRefs = /"workspace(-base64)?:|"https?:\/\/|"data:/.test(JSON.stringify(body ?? {}));
+ const candidates = [def, ...all].filter((m, i, a) => m !== model && a.indexOf(m) === i && (hasRefs || !m.requiresReference)).slice(0, 3);
+ const params = describeBody(body) || [typeof args.size === "string" ? args.size : "", Number(args.seconds) > 0 ? `${args.seconds} 秒` : ""].filter(Boolean).join(" · ");
+ const summary = [`提示词:${prompt.length > 120 ? `${prompt.slice(0, 120)}…` : prompt}`, params && `参数:${params}`, model.price && `参考价:${model.price}`].filter(Boolean).join("\n");
+ return { kind, model, default: def, candidates, all, custom: !!body, summary };
+ }
+
+ /** 用确定下来的模型生成,返回给模型的一段文字 */
+ async execute(botId: string, model: MediaModel, args: Record<string, unknown>): Promise<string> {
+ const prompt = String(args.prompt ?? "").trim();
+ const body = parseBody(args.body);
+ const resolved = body ? (this.inline(botId, body) as Record<string, unknown>) : undefined;
+ return model.kind === "image" ? this.image(botId, model, prompt, resolved, args) : this.video(botId, model, prompt, resolved, args);
}
- private async pick(botId: string, kind: "image" | "video", wanted: unknown): Promise<MediaModel> {
- const list = await this.models(botId, kind);
- const usable = kind === "image" ? list.filter((m) => m.image_route === "images" || m.image_route === "gemini-generate-content") : list;
- if (!usable.length) throw new MediaError(kind === "image" ? "这把 Key 没有可用的图片模型" : "这把 Key 没有可用的视频模型");
- const name = String(wanted ?? "").trim();
- if (!name) return usable[0];
- const hit = usable.find((m) => m.id === name);
- if (!hit) throw new MediaError(`没有这个模型:${name}。可用:${usable.map((m) => m.id).join("、")}`);
- return hit;
+ /** 把请求体里的 "workspace:路径" 换成 data URL、"workspace-base64:路径" 换成纯 Base64;只认工作区里的文件 */
+ private inline(botId: string, v: unknown): unknown {
+ if (typeof v === "string") {
+ const m = FILE_REF.exec(v);
+ if (!m) return v;
+ const file = this.workspaceFile(botId, m[2].trim());
+ const data = fs.readFileSync(file);
+ const mime = MIME[path.extname(file).toLowerCase()] ?? "application/octet-stream";
+ return m[1] ? data.toString("base64") : `data:${mime};base64,${data.toString("base64")}`;
+ }
+ if (Array.isArray(v)) return v.map((x) => this.inline(botId, x));
+ if (v && typeof v === "object") return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, this.inline(botId, x)]));
+ return v;
}
- private save(botId: string, kind: "image" | "video", data: Buffer, mime: string): string {
+ private workspaceFile(botId: string, rel: string): string {
+ const root = fs.realpathSync(this.o.workspace(botId));
+ let real: string;
+ try {
+ real = fs.realpathSync(path.resolve(root, rel));
+ } catch {
+ throw new MediaError(`工作区里没有这个文件:${rel}`);
+ }
+ if (real !== root && !real.startsWith(root + path.sep)) throw new MediaError(`只能引用工作区里的文件:${rel}`);
+ const st = fs.statSync(real);
+ if (!st.isFile()) throw new MediaError(`不是文件:${rel}`);
+ if (st.size > MAX_REFERENCE_BYTES) throw new MediaError(`参考素材太大(${rel},上限 20 MB)`);
+ return real;
+ }
+
+ private save(botId: string, kind: MediaKind, data: Buffer, mime: string): string {
const dir = path.join(this.o.workspace(botId), "media");
fs.mkdirSync(dir, { recursive: true });
const stamp = new Date().toISOString().replace(/[-:]/g, "").replace(/\..+$/, "").replace("T", "-");
@@ -118,22 +404,24 @@ export class MediaService {
return { data, mime: res.headers.get("content-type") || "" };
}
- private async image(botId: string, prompt: string, args: Record<string, unknown>): Promise<string> {
- const model = await this.pick(botId, "image", args.model);
+ private async image(botId: string, model: MediaModel, prompt: string, body: Record<string, unknown> | undefined, args: Record<string, unknown>): Promise<string> {
+ const key = this.key(botId);
const files: string[] = [];
- if (model.image_route === "gemini-generate-content") {
- const res = await this.call(botId, "POST", `/models/${encodeURIComponent(model.id)}:generateContent`, { contents: [{ role: "user", parts: [{ text: prompt }] }] });
- const body = (await res.json()) as { candidates?: Array<{ content?: { parts?: Array<{ inlineData?: { mimeType?: string; data?: string }; inline_data?: { mime_type?: string; data?: string } }> } }> };
- for (const c of body.candidates ?? [])
+ if (model.route === "gemini") {
+ const req = body ? withoutModel(body) : { contents: [{ role: "user", parts: [{ text: prompt }] }] };
+ const res = await this.call(key, "POST", `/models/${encodeURIComponent(model.id)}:generateContent`, req);
+ const out = (await res.json()) as { candidates?: Array<{ content?: { parts?: Array<{ inlineData?: { mimeType?: string; data?: string }; inline_data?: { mime_type?: string; data?: string } }> } }> };
+ for (const c of out.candidates ?? [])
for (const p of c.content?.parts ?? []) {
const data = p.inlineData?.data ?? p.inline_data?.data;
if (data) files.push(this.save(botId, "image", Buffer.from(data, "base64"), p.inlineData?.mimeType ?? p.inline_data?.mime_type ?? "image/png"));
}
} else {
const size = typeof args.size === "string" && /^\d{2,5}x\d{2,5}$/.test(args.size) ? args.size : undefined;
- const res = await this.call(botId, "POST", "/images/generations", { model: model.id, prompt, ...(size ? { size } : {}) });
- const body = (await res.json()) as { data?: Array<{ b64_json?: string; url?: string }> };
- for (const item of body.data ?? []) {
+ const req = body ? { ...body, model: model.id } : { model: model.id, prompt, ...(size ? { size } : {}) };
+ const res = await this.call(key, "POST", args.endpoint === "edits" ? "/images/edits" : "/images/generations", req);
+ const out = (await res.json()) as { data?: Array<{ b64_json?: string; url?: string }> };
+ for (const item of out.data ?? []) {
if (item.b64_json) files.push(this.save(botId, "image", Buffer.from(item.b64_json, "base64"), "image/png"));
else if (item.url) {
const d = await this.download(item.url);
@@ -146,25 +434,52 @@ export class MediaService {
return `已用 ${model.id} 生成 ${files.length} 张图片,保存在工作区:${files.join("、")}。要发给用户,请用 present 工具交付这些文件。`;
}
- private async video(botId: string, prompt: string, args: Record<string, unknown>): Promise<string> {
- const model = await this.pick(botId, "video", args.model);
+ private async video(botId: string, model: MediaModel, prompt: string, body: Record<string, unknown> | undefined, args: Record<string, unknown>): Promise<string> {
+ const key = this.key(botId);
const seconds = Number(args.seconds);
const size = typeof args.size === "string" && /^\d{2,5}x\d{2,5}$/.test(args.size) ? args.size : undefined;
- const created = (await (await this.call(botId, "POST", "/videos", { model: model.id, prompt, ...(Number.isFinite(seconds) && seconds > 0 ? { seconds: String(Math.round(seconds)) } : {}), ...(size ? { size } : {}) })).json()) as { id?: string; task_id?: string };
+ const req = body
+ ? { ...body, model: model.id }
+ : { model: model.id, prompt, ...(Number.isFinite(seconds) && seconds > 0 ? { seconds: String(Math.round(seconds)) } : {}), ...(size ? { size } : {}) };
+ const created = (await (await this.call(key, "POST", "/videos", req)).json()) as { id?: string; task_id?: string };
const id = created.id || created.task_id;
if (!id) throw new MediaError("Modelflare 没有返回视频任务编号");
const deadline = Date.now() + (this.o.videoTimeoutMs ?? 20 * 60_000);
for (;;) {
- const t = (await (await this.call(botId, "GET", `/videos/${encodeURIComponent(id)}`, undefined, 30_000)).json()) as { status?: string; error?: { message?: string } | null };
+ const t = (await (await this.call(key, "GET", `/videos/${encodeURIComponent(id)}`, undefined, 30_000)).json()) as {
+ status?: string;
+ error?: { message?: string } | null;
+ error_message?: string;
+ };
if (t.status === "completed") break;
- if (t.status === "failed") throw new MediaError(`视频生成失败:${t.error?.message || "未知原因"}`);
+ if (t.status === "failed") throw new MediaError(`视频生成失败:${t.error?.message || t.error_message || "未知原因"}`);
if (Date.now() > deadline) throw new MediaError(`视频任务 ${id} 超时还没完成,可以稍后再试`);
await new Promise((r) => setTimeout(r, this.o.pollMs ?? 5000));
}
- const content = await this.call(botId, "GET", `/videos/${encodeURIComponent(id)}/content`, undefined, 600_000);
+ const content = await this.call(key, "GET", `/videos/${encodeURIComponent(id)}/content`, undefined, 600_000);
const d = await this.readBody(content);
const file = this.save(botId, "video", d.data, d.mime || "video/mp4");
log.info("video generated", { bot: botId, model: model.id });
return `已用 ${model.id} 生成视频,保存在工作区:${file}。要发给用户,请用 present 工具交付这个文件。`;
}
}
+
+/** body 可以是对象,也可以是 JSON 文本(有的模型会把它写成字符串) */
+function parseBody(v: unknown): Record<string, unknown> | undefined {
+ if (v === undefined || v === null || v === "") return undefined;
+ let o = v;
+ if (typeof v === "string") {
+ try {
+ o = JSON.parse(v);
+ } catch {
+ throw new MediaError("body 不是合法的 JSON");
+ }
+ }
+ if (!o || typeof o !== "object" || Array.isArray(o)) throw new MediaError("body 必须是 JSON 对象");
+ return o as Record<string, unknown>;
+}
+
+function withoutModel(body: Record<string, unknown>): Record<string, unknown> {
+ const { model: _model, ...rest } = body;
+ return rest;
+}
diff --git a/src/app.ts b/src/app.ts
index fc2b895..1f357ad 100644
--- a/src/app.ts
+++ b/src/app.ts
@@ -35,7 +35,7 @@ import {
import { AccountManager } from "./account/manager.js";
import { AccountVault, type SecretBox } from "./account/vault.js";
import { loadModelInfo, type ModelInfo } from "./account/model-info.js";
-import { MediaService } from "./agent/media.js";
+import { MediaService, pickDefault } from "./agent/media.js";
import type { BotSettings } from "./agent/settings.js";
const log = logger("app");
@@ -169,6 +169,8 @@ export class App {
keyFor: (botId) => this.botKey(botId),
headers: () => this.requestHeaders(),
workspace: (botId) => path.join(config.dataDir, "workspaces", botId),
+ defaults: () => ({ image: this.config.media.imageModel, video: this.config.media.videoModel }),
+ pricingUrl: () => `${modelflareIssuer()}/api/pricing`,
})
: undefined,
onNotice: (text) => this.notice("warn", text),
@@ -445,6 +447,30 @@ export class App {
saveConfigFile(this.config.dataDir, (f) => (f.llm = { ...(f.llm ?? {}), defaultModel: model }));
}
+ /** 默认的图片 / 视频模型;空字符串 = 自动(推荐顺序的第一个) */
+ setMediaModels(patch: { imageModel?: string; videoModel?: string }): void {
+ Object.assign(this.config.media, patch);
+ saveConfigFile(this.config.dataDir, (f) => (f.media = { ...(f.media ?? {}), ...patch }));
+ }
+
+ /** 设置页与首次设置用:共用 Key 能用的图片 / 视频模型(按推荐顺序)与自动时会选的那个 */
+ async mediaModels(): Promise<{ image: string[]; video: string[]; recommended: { image: string; video: string }; error?: string }> {
+ const media = this.agent.media;
+ const key = this.magpie?.providerKey(this.magpie.modelflareProviderId ?? MODELFLARE_PROVIDER) ?? "";
+ const empty = { image: [], video: [], recommended: { image: "", video: "" } };
+ if (!media || !key) return empty;
+ try {
+ const [image, video] = await Promise.all([media.available(key, "image"), media.available(key, "video")]);
+ return {
+ image: image.map((m) => m.id),
+ video: video.map((m) => m.id),
+ recommended: { image: pickDefault(image, "")?.id ?? "", video: pickDefault(video, "")?.id ?? "" },
+ };
+ } catch (e) {
+ return { ...empty, error: (e as Error).message };
+ }
+ }
+
private seedDefaultBot(): void {
if (this.store.listBots().length) return;
const bot = this.store.createBot({
diff --git a/src/config.ts b/src/config.ts
index 719cfe2..958d3ef 100644
--- a/src/config.ts
+++ b/src/config.ts
@@ -32,6 +32,8 @@ export interface AppConfig {
externalApiKey: string;
};
llm: { defaultModel: string; contextMessages: number; contextChars: number };
+ /** 默认的图片 / 视频模型(生成前确认时默认选中);空 = 按推荐顺序的第一个 */
+ media: { imageModel: string; videoModel: string };
feishu: { throttleMs: number };
weixin: { apiBaseUrl: string };
/** 代理宿主:dsh = DeepSeek Harness(工具/沙箱/审批);off = 纯对话 */
@@ -87,6 +89,10 @@ export function loadConfig(overrides: { dataDir?: string; env?: NodeJS.ProcessEn
contextMessages: num(env.AMB_CONTEXT_MESSAGES, file.llm?.contextMessages ?? 40),
contextChars: num(env.AMB_CONTEXT_CHARS, file.llm?.contextChars ?? 24000),
},
+ media: {
+ imageModel: typeof file.media?.imageModel === "string" ? file.media.imageModel : "",
+ videoModel: typeof file.media?.videoModel === "string" ? file.media.videoModel : "",
+ },
feishu: { throttleMs: num(env.AMB_FEISHU_THROTTLE_MS, file.feishu?.throttleMs ?? 400) },
weixin: { apiBaseUrl: env.AMB_WEIXIN_API_BASE || file.weixin?.apiBaseUrl || "https://ilinkai.weixin.qq.com" },
agent: {
diff --git a/src/web/server.ts b/src/web/server.ts
index c59c8e5..0136465 100644
--- a/src/web/server.ts
+++ b/src/web/server.ts
@@ -252,6 +252,7 @@ export async function startWebServer(app: App, opts: { port?: number; host?: str
await app.setModelflareLine(normalizeLine(b.line));
return send(res, 200, app.modelflareStatus());
}
+ if (p === "/media/models" && m === "GET") return send(res, 200, await app.mediaModels());
if (p === "/models/info" && m === "GET") {
if (!app.magpie) return send(res, 200, { models: {} });
return send(res, 200, { models: await app.modelInfo(url.searchParams.get("refresh") === "1") });
@@ -308,6 +309,12 @@ export async function startWebServer(app: App, opts: { port?: number; host?: str
if (m === "PUT") {
const b = await readJson(req);
if (b.defaultModel !== undefined) app.setDefaultModel(String(b.defaultModel));
+ if (b.imageModel !== undefined || b.videoModel !== undefined) {
+ app.setMediaModels({
+ ...(b.imageModel !== undefined ? { imageModel: String(b.imageModel) } : {}),
+ ...(b.videoModel !== undefined ? { videoModel: String(b.videoModel) } : {}),
+ });
+ }
if (b.closeToTray !== undefined) {
cfg.ui.closeToTray = !!b.closeToTray;
saveConfigFile(cfg.dataDir, (f) => (f.ui = { ...(f.ui ?? {}), closeToTray: cfg.ui.closeToTray }));
@@ -326,6 +333,8 @@ export async function startWebServer(app: App, opts: { port?: number; host?: str
} else if (m !== "GET") throw new HttpError(405, "method not allowed");
return send(res, 200, {
defaultModel: cfg.llm.defaultModel,
+ imageModel: cfg.media.imageModel,
+ videoModel: cfg.media.videoModel,
closeToTray: cfg.ui.closeToTray,
zoom: cfg.ui.zoom,
autostart: app.desktop ? app.desktop.autostart() : { supported: false, on: false },
diff --git a/test/account.test.ts b/test/account.test.ts
index 8f3a6ac..5b0bd8d 100644
--- a/test/account.test.ts
+++ b/test/account.test.ts
@@ -12,7 +12,7 @@ import { discover, startLogin, pkcePair } from "../src/account/oauth.js";
import { AccountManager, summarizeUsage, type AccountHooks } from "../src/account/manager.js";
import { AccountVault } from "../src/account/vault.js";
import { referencePrice } from "../src/account/model-info.js";
-import { MediaService } from "../src/agent/media.js";
+import { MediaService, docSection } from "../src/agent/media.js";
import { AgentManager } from "../src/agent/manager.js";
import { locateDshRuntime } from "../src/agent/host.js";
import { findAppRoot } from "../src/magpie/supervisor.js";
@@ -291,6 +291,102 @@ describe("生图 / 生视频", () => {
await srv.close();
}
});
+
+ it("按文档写的请求体原样发出:填上 model,工作区文件换成 data URL / Base64,工作区外的文件不让引用", async () => {
+ const bodies: Array<{ path: string; body: any }> = [];
+ const srv = await listen(async (req, res) => {
+ const u = new URL(req.url!, "http://x");
+ const json = (o: unknown) => res.writeHead(200, { "content-type": "application/json" }).end(JSON.stringify(o));
+ if (u.pathname === "/v1/models") {
+ return json({ data: u.searchParams.get("kind") === "image" ? [{ id: "gemini-img", kind: "image", image_route: "gemini-generate-content" }, { id: "gpt-image", kind: "image", image_route: "images" }] : [{ id: "kling-x", kind: "video" }] });
+ }
+ if (req.method !== "POST") return res.writeHead(404).end();
+ bodies.push({ path: u.pathname, body: await readJson(req) });
+ if (u.pathname.endsWith(":generateContent")) return json({ candidates: [{ content: { parts: [{ inlineData: { mimeType: "image/png", data: png.toString("base64") } }] } }] });
+ if (u.pathname === "/v1/images/edits") return json({ data: [{ b64_json: png.toString("base64") }] });
+ if (u.pathname === "/v1/videos") return json({ id: "t1" });
+ res.writeHead(404).end();
+ });
+ const ws = fs.mkdtempSync(path.join(os.tmpdir(), "amb-media-"));
+ fs.mkdirSync(path.join(ws, "media"));
+ fs.writeFileSync(path.join(ws, "media", "ref.png"), png);
+ try {
+ const media = new MediaService({ baseUrl: () => srv.url + "/v1", keyFor: () => "sk", headers: () => ({}), workspace: () => ws, pollMs: 5, videoTimeoutMs: 1 });
+ await media.run("b1", "image", {
+ prompt: "换成水彩",
+ model: "gemini-img",
+ body: { model: "ignored", contents: [{ role: "user", parts: [{ text: "换成水彩" }, { inline_data: { mime_type: "image/png", data: "workspace-base64:media/ref.png" } }] }] },
+ });
+ expect(bodies[0].path).toBe("/v1/models/gemini-img:generateContent");
+ expect(bodies[0].body).toEqual({ contents: [{ role: "user", parts: [{ text: "换成水彩" }, { inline_data: { mime_type: "image/png", data: png.toString("base64") } }] }] });
+
+ await media.run("b1", "image", { prompt: "加一顶帽子", model: "gpt-image", endpoint: "edits", body: JSON.stringify({ prompt: "加一顶帽子", images: [{ image_url: "workspace:media/ref.png" }] }) });
+ expect(bodies[1]).toEqual({ path: "/v1/images/edits", body: { prompt: "加一顶帽子", images: [{ image_url: `data:image/png;base64,${png.toString("base64")}` }], model: "gpt-image" } });
+
+ // 视频:body 原样带上 model 发出(这个 mock 不回答轮询,只看建任务的请求)
+ await expect(media.run("b1", "video", { prompt: "猫", body: { contents: [{ type: "prompt", text: "猫" }], settings: { duration: 5 } } })).rejects.toThrow();
+ expect(bodies[2]).toEqual({ path: "/v1/videos", body: { contents: [{ type: "prompt", text: "猫" }], settings: { duration: 5 }, model: "kling-x" } });
+
+ await expect(media.run("b1", "image", { prompt: "x", model: "gemini-img", body: { a: "workspace:../../etc/passwd" } })).rejects.toThrow(/工作区/);
+ await expect(media.run("b1", "image", { prompt: "x", model: "gemini-img", body: "{oops" })).rejects.toThrow(/JSON/);
+ expect(bodies).toHaveLength(3);
+ } finally {
+ await srv.close();
+ }
+ });
+
+ it("默认模型:设置里选的优先;没选时按价格目录的推荐顺序,跳过必须带参考素材的;确认时附参考价", async () => {
+ const srv = await listen(async (req, res) => {
+ const u = new URL(req.url!, "http://x");
+ const json = (o: unknown) => res.writeHead(200, { "content-type": "application/json" }).end(JSON.stringify(o));
+ if (u.pathname === "/v1/models") return json({ data: [{ id: "a-i2v", kind: "video" }, { id: "b-t2v", kind: "video" }, { id: "c-pro", kind: "video" }] });
+ if (u.pathname === "/api/pricing") {
+ return json({
+ group_ratio: { Media: 1 },
+ usable_group: { Media: "" },
+ data: [
+ { model_name: "a-i2v", enable_groups: ["Media"], media_options: { video: { requires_reference: true, resolutions: ["720p"] } } },
+ { model_name: "b-t2v", enable_groups: ["Media"], video_resolution_price: { "720p": 0.05, "1080p": 0.1 }, media_options: { video: { default_resolution: "720p", seconds: [-1, 4, 10] } } },
+ { model_name: "c-pro", enable_groups: ["Media"], display_priority: 5 },
+ ],
+ });
+ }
+ res.writeHead(404).end();
+ });
+ const defaults = { image: "", video: "" };
+ try {
+ const media = new MediaService({ baseUrl: () => srv.url + "/v1", keyFor: () => "sk", headers: () => ({}), workspace: () => os.tmpdir(), defaults: () => defaults, pricingUrl: () => srv.url + "/api/pricing" });
+ const list = await media.models("b1", "video");
+ expect(list.map((m) => m.id)).toEqual(["c-pro", "b-t2v", "a-i2v"]);
+ expect(list[1].price).toBe("$0.05/秒(720p)");
+ let plan = await media.prepare("b1", "video", { prompt: "海浪" });
+ expect(plan.model.id).toBe("c-pro");
+ expect(plan.candidates.map((m) => m.id)).toEqual(["b-t2v"]);
+ defaults.video = "b-t2v";
+ plan = await media.prepare("b1", "video", { prompt: "海浪", body: { duration: 10, ratio: "16:9", content: [{ image_url: { url: "workspace:media/x.png" } }] } });
+ expect(plan.model.id).toBe("b-t2v");
+ expect(plan.custom).toBe(true);
+ // 带了参考素材,必须带参考素材的模型也列进可选
+ expect(plan.candidates.map((m) => m.id)).toEqual(["c-pro", "a-i2v"]);
+ expect(plan.summary).toContain("10 秒");
+ expect(plan.summary).toContain("1 个参考素材");
+ expect(plan.summary).toContain("参考价:$0.05/秒(720p)");
+ expect(await media.listText("b1", "video")).toMatch(/- b-t2v(默认) — \$0\.05\/秒(720p) — 时长 4–10 秒/);
+ } finally {
+ await srv.close();
+ }
+ });
+
+ it("文档:从全文里取出该模型那一节", () => {
+ const full = [
+ "# Overview\n\nSource: https://docs.x/\n\nhello",
+ "# kling-3.0 API\n\nSource: https://docs.x/models/video/kling-3-0/\n\nbody A",
+ "# kling-3.0-turbo API\n\nSource: https://docs.x/models/video/kling-3-0-turbo/\n\nbody B\n\n## Params\n\n| a |",
+ ].join("\n\n---\n\n");
+ expect(docSection(full, "https://docs.x", "video", "kling-3.0")).toBe("# kling-3.0 API\n\nSource: https://docs.x/models/video/kling-3-0/\n\nbody A");
+ expect(docSection(full, "https://docs.x", "video", "kling-3.0-turbo")).toMatch(/^# kling-3\.0-turbo API[\s\S]*\| a \|$/);
+ expect(docSection(full, "https://docs.x", "image", "kling-3.0")).toBe("");
+ });
});
describe("amb-media 插件与 bridge", () => {
@@ -308,15 +404,77 @@ describe("amb-media 插件与 bridge", () => {
const plugin = await import(path.join(runtime!, "plugins", "amb-media.mjs"));
const tools: any[] = [];
plugin.apply({ tools: { register: (t: any) => tools.push(t) } });
- expect(tools.map((t) => t.name)).toEqual(["generate_image", "generate_video"]);
- expect(await tools[0].execute({ prompt: "猫" }, { agent: { session: { id: "s1" } }, signal: AbortSignal.timeout(5000) })).toEqual({ text: "ok" });
- expect(seen).toEqual([{ url: "/dsh/media", auth: "Bearer tok", body: { botId: "b1", sessionId: "s1", op: "image", args: { prompt: "猫" } } }]);
+ expect(tools.map((t) => t.name)).toEqual(["media_models", "media_docs", "generate_image", "generate_video"]);
+ const image = tools.find((t) => t.name === "generate_image");
+ const body = { contents: [{ parts: [{ text: "猫" }, { inline_data: { mime_type: "image/png", data: "workspace-base64:media/a.png" } }] }] };
+ expect(await image.execute({ prompt: "猫", body }, { agent: { session: { id: "s1" } }, signal: AbortSignal.timeout(5000) })).toEqual({ text: "ok" });
+ expect(seen).toEqual([{ url: "/dsh/media", auth: "Bearer tok", body: { botId: "b1", sessionId: "s1", op: "image", args: { prompt: "猫", body } } }]);
} finally {
process.env = env;
await bridge.close();
}
});
+ it("每次生成前都向用户确认模型:确认、换模型、按新模型文档重写后免确认、取消、回复别的话", async () => {
+ const store = new Store(":memory:");
+ const M1 = { id: "m1", kind: "image", price: "$0.03/张" };
+ const M2 = { id: "m2", kind: "image", price: "" };
+ const ran: string[] = [];
+ const media = {
+ prepare: async (_b: string, op: string, args: any) => ({ kind: op, model: args.model === "m2" ? M2 : M1, default: M1, candidates: [M2], all: [M1, M2], custom: !!args.body, summary: "提示词:猫" }),
+ execute: async (_b: string, model: { id: string }) => (ran.push(model.id), `done ${model.id}`),
+ } as unknown as MediaService;
+ const mgr = new AgentManager({ store, dataDir: fs.mkdtempSync(path.join(os.tmpdir(), "amb-mgr-")), runtimeDir: undefined, upstream: () => ({ baseUrl: "", apiKey: "" }), enabled: false, media });
+ // 模拟机器人 b1 在会话 c1 里有回合进行中
+ (mgr as any).hosts.set("b1", {});
+ (mgr as any).activeConvs.set("b1", new Set(["c1"]));
+ await mgr.bridge.start();
+ const call = (args: Record<string, unknown>) =>
+ fetch(`${mgr.bridge.url}/dsh/media`, {
+ method: "POST",
+ headers: { authorization: `Bearer ${mgr.bridge.token}`, "content-type": "application/json" },
+ body: JSON.stringify({ botId: "b1", sessionId: "s1", op: "image", args }),
+ }).then((r) => r.json());
+ const answer = async (a: { selected?: string[]; custom?: string }) => {
+ for (let i = 0; i < 100; i++) {
+ const q = mgr.approvals.pending("c1", "question")[0];
+ if (q) {
+ const [item] = JSON.parse(q.payload);
+ mgr.approvals.answer(q.id, { answers: [{ id: "model", selected: a.selected ?? [], ...(a.custom ? { custom: a.custom } : {}) }] }, "test");
+ return item;
+ }
+ await new Promise((r) => setTimeout(r, 10));
+ }
+ throw new Error("没有等到确认");
+ };
+ try {
+ let [r, q] = await Promise.all([call({ prompt: "猫" }), answer({ selected: ["m1"] })]);
+ expect(q.question).toBe("这次用 m1 生成图片吗?");
+ expect(q.options.map((o: any) => o.label)).toEqual(["m1", "m2", "取消"]);
+ expect(q.options[0].description).toBe("默认 · $0.03/张");
+ expect(r).toEqual({ text: "done m1" });
+
+ [r] = await Promise.all([call({ prompt: "猫" }), answer({ selected: ["m2"] })]);
+ expect(r).toEqual({ text: "done m2" });
+
+ // 请求体是按 m1 的文档写的:换成 m2 时先让代理按 m2 的文档重写,重写后的那一次不再确认
+ [r] = await Promise.all([call({ prompt: "猫", body: { a: 1 } }), answer({ custom: "用 M2" })]);
+ expect(r.text).toMatch(/media_docs 查 m2/);
+ expect(await call({ prompt: "猫", model: "m2", body: { b: 1 } })).toEqual({ text: "done m2" });
+
+ [r] = await Promise.all([call({ prompt: "猫" }), answer({ selected: ["取消"] })]);
+ expect(r.text).toMatch(/取消/);
+ [r] = await Promise.all([call({ prompt: "猫" }), answer({ custom: "好的" })]);
+ expect(r).toEqual({ text: "done m1" });
+ [r] = await Promise.all([call({ prompt: "猫" }), answer({ custom: "换成竖屏的" })]);
+ expect(r.text).toMatch(/「换成竖屏的」/);
+ expect(ran).toEqual(["m1", "m2", "m2", "m1"]);
+ } finally {
+ await mgr.bridge.stop();
+ store.close();
+ }
+ });
+
it("bridge 只在该机器人有回合进行中时受理生图(别的机器人不能借它的 Key)", async () => {
const store = new Store(":memory:");
const calls: string[] = [];
diff --git a/ui/components/model-picker.js b/ui/components/model-picker.js
index 39f0dbb..7b1c804 100644
--- a/ui/components/model-picker.js
+++ b/ui/components/model-picker.js
@@ -4,7 +4,7 @@ import { useEffect, useMemo, useRef, useState } from "preact/hooks";
import { html, Popover, Icon } from "./ui.js";
import { api } from "../lib/api.js";
import { createStore, useStore } from "../lib/store.js";
-import { groupModels, modelFamily, normModel, sameModel } from "../lib/models.js";
+import { groupModels, modelFamily, normModel, sameModel, shortModel } from "../lib/models.js";
export { groupModels, normModel, sameModel, shortModel } from "../lib/models.js";
@@ -154,3 +154,17 @@ export function ModelPicker({ anchor, models, error, current, onPick, onClose, r
${foot && html`<div class="foot"><button type="button" disabled=${foot.disabled} onClick=${() => { onClose(); foot.onClick(); }}>${foot.label}</button></div>`}
<//>`;
}
+
+/**
+ * 菜单按钮 + 模型选择器(设置页、首次设置用)。value 为空时按钮显示 resetLabel;
+ * 给了 resetLabel 时列表顶部有一项"恢复默认",选它 onChange("")。
+ */
+export function PickModel({ value, models, modelsError, reloadModels, onChange, resetLabel, align = "right", label = "选择模型" }) {
+ const [open, setOpen] = useState(false);
+ const ref = useRef(null);
+ return html`<button type="button" class="pick" ref=${ref} aria-haspopup="listbox" aria-expanded=${open} aria-label=${label} onClick=${() => setOpen((v) => !v)}>
+ <span class="v">${value ? shortModel(value) : resetLabel || label}</span><${Icon} name="chevDown" />
+ </button>
+ ${open && html`<${ModelPicker} anchor=${ref.current} align=${align} models=${models} error=${modelsError} onRetry=${reloadModels} current=${value}
+ onPick=${onChange} onClose=${() => setOpen(false)} reset=${resetLabel ? { label: resetLabel, onSelect: () => onChange("") } : undefined} />`}`;
+}
diff --git a/ui/css/settings.css b/ui/css/settings.css
index 9102bb7..6ef863a 100644
--- a/ui/css/settings.css
+++ b/ui/css/settings.css
@@ -140,6 +140,8 @@
.onboard details summary:hover { color: var(--fg); }
/* "高级"里的行直接排在卡片里,左右贴着卡片的内容边,不再套一块玻璃 */
.onboard details .row { --row-pl: 0; --row-pr: 0; margin-top: var(--space-1); }
+.onboard .media-pick { display: grid; gap: var(--space-1); }
+.onboard .media-pick .row { --row-pl: 0; --row-pr: 0; }
.onboard .connected { display: flex; align-items: center; gap: var(--space-2); font-size: var(--fs-lg); font-weight: var(--fw-semibold); }
.login-cta { display: grid; gap: var(--space-2-5); }
.login-wait { display: flex; align-items: center; justify-content: center; gap: var(--space-2); font-size: var(--fs-md); color: var(--fg-2); }
diff --git a/ui/screens/onboarding.js b/ui/screens/onboarding.js
index 73ee74c..304dad2 100644
--- a/ui/screens/onboarding.js
+++ b/ui/screens/onboarding.js
@@ -4,7 +4,7 @@
import { useEffect, useMemo, useRef, useState } from "preact/hooks";
import { html, Icon, Notice, Seg, Spinner, CopyButton } from "../components/ui.js";
import { BotAvatar, MAIN_CHAR } from "../components/avatar.js";
-import { FamilyHead, groupModels, useFamilies } from "../components/model-picker.js";
+import { FamilyHead, PickModel, groupModels, useFamilies } from "../components/model-picker.js";
import { api } from "../lib/api.js";
import { MF_CONSOLE, MF_DOCS, MF_KEYS } from "../lib/links.js";
@@ -173,10 +173,27 @@ function ModelStep({ models, initial, onDone }) {
const groups = useMemo(() => groupModels(models, q), [models, q]);
const { isOpen, toggle } = useFamilies(groups, cur);
const folding = !q.trim() && groups.length > 1;
+ // 默认的图片 / 视频模型:之前选过且还能用就沿用,否则用推荐的
+ const [media, setMedia] = useState(null);
+ const [img, setImg] = useState("");
+ const [vid, setVid] = useState("");
+ useEffect(() => {
+ Promise.all([api("/media/models"), api("/settings").catch(() => ({}))])
+ .then(([d, st]) => {
+ setMedia(d);
+ setImg(d.image.includes(st.imageModel) ? st.imageModel : d.recommended.image || "");
+ setVid(d.video.includes(st.videoModel) ? st.videoModel : d.recommended.video || "");
+ })
+ .catch(() => setMedia({ image: [], video: [] }));
+ }, []);
+ const hasMedia = media && (media.image.length > 0 || media.video.length > 0);
const start = async () => {
setBusy(true);
try {
- await api("/settings", { method: "PUT", body: { defaultModel: cur } });
+ await api("/settings", {
+ method: "PUT",
+ body: { defaultModel: cur, ...(media && media.image.length ? { imageModel: img } : {}), ...(media && media.video.length ? { videoModel: vid } : {}) },
+ });
onDone();
} catch (e) {
setErr(e.message);
@@ -199,7 +216,18 @@ function ModelStep({ models, initial, onDone }) {
return html`<button type="button" class="fam" aria-expanded=${open} onClick=${() => toggle(g)}><${FamilyHead} name=${g} count=${ms.length} current=${hit?.[1]} /></button>${open && rows}`;
})}
${!groups.length && html`<p class="hint">没有匹配的模型</p>`}
- </div>`
+ </div>
+ ${hasMedia && html`<div class="media-pick">
+ <p class="hint">生成图片和视频时默认用的模型。每次生成前都会先问你,也可以当场换。</p>
+ ${media.image.length > 0 && html`<div class="row compact">
+ <div class="who"><div class="name">图片</div></div>
+ <${PickModel} value=${img} models=${media.image} onChange=${setImg} label="默认图片模型" />
+ </div>`}
+ ${media.video.length > 0 && html`<div class="row compact">
+ <div class="who"><div class="name">视频</div></div>
+ <${PickModel} value=${vid} models=${media.video} onChange=${setVid} label="默认视频模型" />
+ </div>`}
+ </div>`}`
: html`<${Notice} tone="amber" action=${html`<a class="btn" href=${MF_CONSOLE} target="_blank" rel="noopener noreferrer">去控制台 ↗</a>`}>这个 Key 所在的分组暂无可用模型。可以在控制台给分组开放模型后再回来刷新。<//>`}
${err && html`<div class="err-text" role="alert">${err}</div>`}
<button type="button" class="btn primary lg block" disabled=${busy} onClick=${start}>${busy ? html`<${Spinner} /> 正在保存…` : models.length ? "开始对话" : "仍然开始"}</button>
diff --git a/ui/screens/settings.js b/ui/screens/settings.js
index 0123ee8..9ab311b 100644
--- a/ui/screens/settings.js
+++ b/ui/screens/settings.js
@@ -3,7 +3,7 @@
import { useEffect, useMemo, useRef, useState } from "preact/hooks";
import { html, Icon, Seg, Switch, Sheet, CopyButton, Skeleton } from "../components/ui.js";
import { AccountButton } from "./sidebar.js";
-import { ModelPicker, shortModel } from "../components/model-picker.js";
+import { PickModel, shortModel } from "../components/model-picker.js";
import { KeyForm, LINE_OPTIONS } from "./onboarding.js";
import { LoginSheet, NotificationsSheet, UsageSheet } from "./account.js";
import { api } from "../lib/api.js";
@@ -44,16 +44,6 @@ function Row({ id, name, sub, title, children, flash, off }) {
</div>`;
}
-function PickModel({ value, models, modelsError, reloadModels, onChange, resetLabel }) {
- const [open, setOpen] = useState(false);
- const ref = useRef(null);
- return html`<button type="button" class="pick" ref=${ref} aria-haspopup="listbox" aria-expanded=${open} onClick=${() => setOpen((v) => !v)}>
- <span class="v">${value ? shortModel(value) : resetLabel}</span><${Icon} name="chevDown" />
- </button>
- ${open && html`<${ModelPicker} anchor=${ref.current} align="right" models=${models} error=${modelsError} onRetry=${reloadModels} current=${value}
- onPick=${onChange} onClose=${() => setOpen(false)} reset=${{ label: resetLabel, onSelect: () => onChange("") }} />`}`;
-}
-
function NoticesSheet({ onClose }) {
const [d, setD] = useState(null);
useEffect(() => {
@@ -99,9 +89,18 @@ export function SettingsScreen({ tab, query: q, onQuery: setQ, onTab, focusRow,
const [flash, setFlash] = useState("");
const mode = useStore(themeMode);
+ // 图片 / 视频模型(共用 Key 能用的,按推荐顺序):连上 Modelflare 后取
+ const [media, setMedia] = useState(null);
+ const loadMedia = () =>
+ api("/media/models")
+ .then(setMedia)
+ .catch((e) => setMedia({ image: [], video: [], recommended: {}, error: e.message }));
useEffect(() => {
api("/settings").then(setS).catch(failed);
}, []);
+ useEffect(() => {
+ if (mf && mf.connected) loadMedia();
+ }, [mf && mf.connected]);
useEffect(() => {
if (bots && bots[0]) api(`/bots/${encodeURIComponent(bots[0].id)}/settings`).then((d) => setCu(d.computerUse)).catch(() => {});
}, [bots && bots[0] && bots[0].id]);
@@ -229,6 +228,14 @@ export function SettingsScreen({ tab, query: q, onQuery: setQ, onTab, focusRow,
{ tab: "modelflare", id: "default-model", name: "默认模型", sub: "机器人没有单独指定时使用", keys: "模型 model",
ctl: html`<${PickModel} value=${s.defaultModel} models=${models} modelsError=${modelsError} reloadModels=${reloadModels} resetLabel="自动(列表第一个)"
onChange=${(m) => put({ defaultModel: m }, { defaultModel: m })} />` },
+ managed && mf.connected && { tab: "modelflare", id: "image-model", name: "默认图片模型", sub: "机器人生成图片时默认选它,每次生成前都会让你确认", keys: "图片 生图 画图 image 模型",
+ ctl: html`<${PickModel} value=${s.imageModel} models=${media ? media.image : null} modelsError=${media && media.error} reloadModels=${loadMedia}
+ resetLabel=${media && media.recommended.image ? `自动(${shortModel(media.recommended.image)})` : "自动"} label="默认图片模型"
+ onChange=${(m) => put({ imageModel: m }, { imageModel: m })} />` },
+ managed && mf.connected && { tab: "modelflare", id: "video-model", name: "默认视频模型", sub: "机器人生成视频时默认选它,每次生成前都会让你确认", keys: "视频 生视频 video 模型",
+ ctl: html`<${PickModel} value=${s.videoModel} models=${media ? media.video : null} modelsError=${media && media.error} reloadModels=${loadMedia}
+ resetLabel=${media && media.recommended.video ? `自动(${shortModel(media.recommended.video)})` : "自动"} label="默认视频模型"
+ onChange=${(m) => put({ videoModel: m }, { videoModel: m })} />` },
{ tab: "modelflare", id: "models", name: "可用模型", sub: modelsError ? `获取失败:${modelsError}` : "这个 Key 所在分组开放的模型", keys: "列表 刷新",
ctl: html`<span class="val num">${models ? `${models.length} 个` : "…"}</span><button type="button" class="text" onClick=${() => reloadModels().then(() => saved())}>刷新</button>` },
{ tab: "modelflare", id: "console", name: "Modelflare 控制台", sub: "账单、发票与 API Key 管理", keys: "控制台 账单 console",