Skip to content

One-click character swap for videos: an open-source, local Higgsfield Genjutsu alternative built on ComfyUI and MiniMax H3, with FLUX.3 Klein relighting.

Add From picture video render settings

D3voze61b9913546f
diff --git a/README.md b/README.md
index 6c6e131..5995ecf 100644
--- a/README.md
+++ b/README.md
@@ -9,7 +9,7 @@ The app is a small Next.js front end over a ComfyUI workflow. It uses [`@comfyor
- **Simple motion transfer**: available. One character replaces the main person (the one who fills the most of the frame). Runs `workflows/simple.api.json`, exported from `H3 One-Click Single Swap (DMAD 6 step).json`.
- **Advanced motion transfer**: available. 1–4 characters, each with an optional role; every tracked person is matched to a role per shot, and people without a character stay untouched. Runs `workflows/advanced.api.json`, exported from `H3 One-Click Advanced Swap (DMAD 6 step).json`; **One pass per person** (two characters) runs `workflows/advanced_maxq.api.json`.
- **Object swap**: available. **Change** what SAM tracks (hair, a jacket, a car) into what you describe or show in a picture; **Remove** it; or change the **Background** around it (`person` stays). Only that area is regenerated. Runs `workflows/object.api.json`, exported from `H3 One-Click Object Swap (DMAD 6 step).json`.
-- **From picture** (under Object swap): FLUX prepares the character in the raw video frame's pose and body cut while keeping the character picture's background. Review or regenerate the image before choosing **Use this image · render video**. The video step uses the exact saved preview with Metric3D normal-map motion guidance, for one 5.875-second clip at a 480 px short edge with the source soundtrack. See [the workflow requirements](workflows/picture/README.md).
+- **From picture** (under Object swap): FLUX prepares the character in the raw video frame's pose and body cut while keeping the character picture's background. Review or regenerate the image, then choose **Use this image** to set the video length (up to 15 seconds), video and motion-reference resolutions, models, LoRAs, steps and seed. **Render video** reuses the saved preview with normal-map motion guidance and the source soundtrack, in one clip without a context loop. H3 rounds the requested length down to its valid frame grid. See [the workflow requirements](workflows/picture/README.md).
- **Post-processing**: available, for a finished video. **Fix faces**: H3 draws faces poorly when the person is small in the frame, so this redraws the face of the person in a photo at close-up size and pastes it back. Runs `workflows/face.api.json`, written by `workflows/build/openjutsu_face_build.py`. **Fix colours** softens H3's harsh look without an H3 run: Image Detail Contrast takes some fine detail contrast out, then Orion4D's DCTL Tone Mapper grades every frame. Runs `workflows/color.api.json`.
## Requirements
diff --git a/app/api/picture/route.ts b/app/api/picture/route.ts
index 398cce4..4b03fb7 100644
--- a/app/api/picture/route.ts
+++ b/app/api/picture/route.ts
@@ -4,6 +4,8 @@ import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { apiAccess, HttpError, IMAGE_SIGNATURE, readFormData, StoredFile, validateJobId, VIDEO_SIGNATURE } from "../../../lib/http.ts";
import { canvasSize, isLink, MAX_SEED, type ApiGraph } from "../../../lib/simple.ts";
+import { listModels } from "../../../lib/comfy.ts";
+import { applyPictureVideoSettings, DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, type PictureVideoSettings } from "../../../lib/picture.ts";
export const runtime = "nodejs";
const PREPARED = "OpenJutsu · prepared picture";
@@ -78,14 +80,24 @@ export async function POST(request: Request) {
const file = item?.outputs?.["21"]?.images?.[0];
const source = item?.prompt?.[2];
if (!file || !source || source["21"]?._meta?.title !== PREPARED || source["1"]?.class_type !== "VHS_LoadVideo") throw new HttpError(409, "The prepared picture is not ready. Generate it first.");
+ let settings: PictureVideoSettings;
+ try { settings = parsePictureVideoSettings(body.settings ?? { ...DEFAULT_PICTURE_VIDEO, seed: source["19"].inputs.seed }); }
+ catch (error) { throw new HttpError(400, (error as Error).message); }
+ const available = await listModels();
+ if (!available.models.includes(settings.model)) throw new HttpError(400, "Choose a video model ComfyUI has.");
+ if (settings.useLoras && settings.loras.some((lora) => lora.enabled && !available.loras.includes(lora.name))) throw new HttpError(400, "Choose LoRAs ComfyUI has.");
+ const [clipInfo, vaeInfo] = await Promise.all([comfy("/object_info/CLIPLoader").then((response) => response.json()), comfy("/object_info/VAELoader").then((response) => response.json())]);
+ for (const [name, choices, label] of [
+ [settings.textEncoder, clipInfo.CLIPLoader.input.required.clip_name[0], "H3 text encoder"],
+ [settings.videoVae, vaeInfo.VAELoader.input.required.vae_name[0], "video VAE"],
+ [settings.audioVae, vaeInfo.VAELoader.input.required.vae_name[0], "audio VAE"],
+ ] as const) if (!choices.includes(name)) throw new HttpError(400, `Choose a ${label} ComfyUI has.`);
const picture = await (await comfy(view(file))).blob();
graph = await template("motion");
graph["20"] = { class_type: "LoadImage", inputs: { image: await uploadFile(picture, ".png") } };
graph["1"].inputs.video = source["1"].inputs.video;
graph["1"].inputs.skip_first_frames = source["1"].inputs.skip_first_frames;
- const [width, height] = canvasSize(Number(source["3"].inputs.width), Number(source["3"].inputs.height), "480p");
- for (const id of ["33", "56"]) Object.assign(graph[id].inputs, { width, height });
- graph["58"].inputs.noise_seed = source["19"].inputs.seed;
+ applyPictureVideoSettings(graph, settings, Number(source["3"].inputs.width), Number(source["3"].inputs.height));
graph["63"]._meta = { title: RENDERED };
const keep = new Set<string>();
function visit(id: string) {
@@ -145,7 +157,7 @@ export async function GET(request: Request) {
const file = isVideo ? output?.gifs?.[0] : output?.images?.[0];
if (!file) throw new HttpError(422, "ComfyUI returned no result. Check its console.");
const base = `/api/picture?id=${encodeURIComponent(id)}`;
- if (!media) return Response.json({ status: "ready", url: `${base}&media=result`, ...(!isVideo ? { frame: `${base}&media=frame` } : {}) });
+ if (!media) return Response.json({ status: "ready", url: `${base}&media=result`, ...(!isVideo ? { frame: `${base}&media=frame`, width: item.prompt![2]["3"].inputs.width, height: item.prompt![2]["3"].inputs.height } : {}) });
const response = await comfy(view(file), { headers: request.headers.has("Range") ? { Range: request.headers.get("Range")! } : {} });
return new Response(response.body, { status: response.status, headers: {
"Content-Type": isVideo ? "video/mp4" : "image/png", "X-Content-Type-Options": "nosniff", "Cache-Control": "private, no-store",
diff --git a/components/from-picture.tsx b/components/from-picture.tsx
index e630794..797500e 100644
--- a/components/from-picture.tsx
+++ b/components/from-picture.tsx
@@ -1,10 +1,13 @@
"use client";
import { useEffect, useState } from "react";
-import { MAX_SEED } from "../lib/simple.ts";
+import { canvasSize, CUSTOM_SIZE, MAX_SEED, MOTION_SIZES, RESOLUTIONS } from "../lib/simple.ts";
+import { MAX_LORAS, STEPS, STRENGTH } from "../lib/model.ts";
+import { DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureFrames, type PictureVideoSettings } from "../lib/picture.ts";
const PROMPT = "change <image 1> character pose follow the <image 2> pose. keep the final image as same body cut as <image 2> . do not change the background of <image 1> and do not add anything extra in <image 1>.";
const STORAGE = "openjutsu.picture-preview";
+const VIDEO_STORAGE = "openjutsu.picture-video-settings";
type Job = { id: string; stage: "prepare" | "render" };
export function FromPicture({ initialVideo, initialPicture, start: initialStart = 0 }: { initialVideo?: File; initialPicture?: File; start?: number }) {
@@ -19,6 +22,11 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
const [job, setJob] = useState<Job | null>(null);
const [prepared, setPrepared] = useState("");
const [preview, setPreview] = useState<{ url: string; frame: string } | null>(null);
+ const [approved, setApproved] = useState(false);
+ const [settings, setSettings] = useState<PictureVideoSettings>(() => structuredClone(DEFAULT_PICTURE_VIDEO));
+ const [settingsLoaded, setSettingsLoaded] = useState(false);
+ const [files, setFiles] = useState<{ models: string[]; loras: string[]; textEncoders: string[]; vaes: string[] }>({ models: [], loras: [], textEncoders: [], vaes: [] });
+ const [modelError, setModelError] = useState("");
const [result, setResult] = useState("");
const [submitting, setSubmitting] = useState(false);
const [error, setError] = useState("");
@@ -28,6 +36,7 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
try {
const saved = JSON.parse(localStorage.getItem(STORAGE) ?? "null");
if (typeof saved?.prepared === "string") setPrepared(saved.prepared);
+ setApproved(saved?.approved === true || saved?.job?.stage === "render");
if (typeof saved?.job?.id === "string" && ["prepare", "render"].includes(saved.job.stage)) {
setJob(saved.job);
if (saved.job.stage === "render" && saved.prepared) {
@@ -37,7 +46,17 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
}
else if (typeof saved?.prepared === "string" && saved.prepared) setJob({ id: saved.prepared, stage: "prepare" });
} catch {}
+ try { setSettings(parsePictureVideoSettings(JSON.parse(localStorage.getItem(VIDEO_STORAGE) ?? "{}"))); } catch {}
+ setSettingsLoaded(true);
+ fetch("/api/models").then(async (response) => {
+ const data = await response.json();
+ if (!response.ok) throw new Error(data.error);
+ setFiles({ models: data.models ?? [], loras: data.loras ?? [], textEncoders: data.textEncoders ?? [], vaes: data.vaes ?? [] });
+ }).catch((error) => setModelError(error.message));
}, []);
+ useEffect(() => {
+ if (settingsLoaded) try { localStorage.setItem(VIDEO_STORAGE, JSON.stringify(settings)); } catch {}
+ }, [settings, settingsLoaded]);
useEffect(() => {
if (!video) return;
const url = URL.createObjectURL(video); setVideoUrl(url); setSize(null);
@@ -49,8 +68,8 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
return () => URL.revokeObjectURL(url);
}, [picture]);
- function remember(next: Job | null, image: string) {
- try { localStorage.setItem(STORAGE, JSON.stringify({ job: next, prepared: image })); } catch {}
+ function remember(next: Job | null, image: string, accepted: boolean) {
+ try { localStorage.setItem(STORAGE, JSON.stringify({ job: next, prepared: image, approved: accepted })); } catch {}
}
useEffect(() => {
if (!job) return;
@@ -62,15 +81,15 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
const data = await response.json();
if (!response.ok) {
setError(data.error ?? "Could not read the job.");
- if ([400, 404, 422].includes(response.status)) { setJob(null); remember(null, prepared); }
+ if ([400, 404, 422].includes(response.status)) { setJob(null); remember(null, prepared, job!.stage === "render"); }
else timer = setTimeout(poll, 5000);
return;
}
setError("");
if (data.status !== "ready") { timer = setTimeout(poll, 2000); return; }
if (job!.stage === "prepare") {
- setPrepared(job!.id); setPreview({ url: data.url, frame: data.frame }); remember(null, job!.id);
- } else { setResult(data.url); remember(job, prepared); }
+ setPrepared(job!.id); setPreview({ url: data.url, frame: data.frame }); setSize([data.width, data.height]); remember(null, job!.id, approved);
+ } else { setResult(data.url); remember(job, prepared, true); }
setJob(null);
} catch {
if (!controller.signal.aborted) { setError("Connection interrupted. Retrying…"); timer = setTimeout(poll, 5000); }
@@ -81,7 +100,7 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
}, [job, prepared]);
function invalidate() {
- setPrepared(""); setPreview(null); setResult(""); setError(""); remember(null, "");
+ setPrepared(""); setPreview(null); setApproved(false); setResult(""); setError(""); remember(null, "", false);
}
async function prepare(nextSeed: number) {
if (!video || !picture || !size) return;
@@ -93,25 +112,33 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
const response = await fetch("/api/picture", { method: "POST", body: form });
const data = await response.json();
if (!response.ok) throw new Error(data.error);
- const next: Job = { id: data.id, stage: "prepare" }; remember(next, ""); setJob(next);
+ const next: Job = { id: data.id, stage: "prepare" }; remember(next, "", false); setJob(next);
} catch (error) { setError((error as Error).message); }
finally { setSubmitting(false); }
}
async function render() {
- if (!prepared || !preview) return;
+ if (!prepared || !preview || !approved) return;
setSubmitting(true); setError(""); setResult("");
try {
- const response = await fetch("/api/picture", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prepared }) });
+ const response = await fetch("/api/picture", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prepared, settings }) });
const data = await response.json();
if (!response.ok) throw new Error(data.error);
- const next: Job = { id: data.id, stage: "render" }; remember(next, prepared); setJob(next);
+ const next: Job = { id: data.id, stage: "render" }; remember(next, prepared, true); setJob(next);
} catch (error) { setError((error as Error).message); }
finally { setSubmitting(false); }
}
+ function picker(label: string, value: string, options: string[], onChange: (value: string) => void) {
+ return <label><span>{label}</span>{options.length
+ ? <select className="file-name" aria-label={label} value={value} onChange={(event) => onChange(event.target.value)}>{[...new Set([value, ...options])].map((name) => <option key={name} value={name}>{name || "Choose a file"}</option>)}</select>
+ : <input className="file-name" aria-label={label} value={value} onChange={(event) => onChange(event.target.value)} />}</label>;
+ }
+ const canvas = typeof settings.resolution === "string" ? canvasSize(size?.[0] ?? 768, size?.[1] ?? 1376, settings.resolution) : [settings.resolution.width, settings.resolution.height];
+ const frames = pictureFrames(settings.seconds);
+
return <div className="picture-pipeline">
<p className="hint">Prepare the character in the video's starting pose with FLUX. Review the image first, then render its motion while keeping the picture's background.</p>
- <div className="inputs">
+ {!approved && <div className="inputs">
<section className="panel">
<h2><span>1</span> Character and source video</h2>
<label>Source video<input type="file" accept="video/*,.mkv" disabled={busy} onChange={(event) => { const file = event.target.files?.[0]; if (file) { invalidate(); setVideo(file); setStart(0); } }} /></label>
@@ -119,7 +146,7 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
<label>Character picture · image 1<input type="file" accept="image/png,image/jpeg,image/webp" disabled={busy} onChange={(event) => { const file = event.target.files?.[0]; if (file) { invalidate(); setPicture(file); } }} /></label>
{pictureUrl && <img className="picture-reference" src={pictureUrl} alt="Character and background reference" />}
<label>Start at (seconds)<input className="target-input" type="number" min={0} step={0.1} value={start} disabled={busy} onChange={(event) => { invalidate(); setStart(Number(event.target.value)); }} /></label>
- <small>The raw frame at this time is image 2. Video test: one 5.875-second clip from here, at a 480 px short edge.</small>
+ <small>The raw frame at this time is image 2. Choose the video length and render settings after reviewing the prepared image.</small>
</section>
<section className="panel">
<h2><span>2</span> Prepare the first frame</h2>
@@ -127,12 +154,12 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
<label>FLUX edit prompt<textarea className="report" rows={6} value={prompt} disabled={busy} onChange={(event) => { invalidate(); setPrompt(event.target.value); }} /></label>
<label>Seed<input className="target-input" type="number" min={0} max={MAX_SEED} step={1} value={seed} disabled={busy} onChange={(event) => { invalidate(); setSeed(Number(event.target.value)); }} /></label>
<button className="primary" disabled={busy || !video || !picture || !size} onClick={() => prepare(seed)}>Generate preview image</button>
- <p className="hint">This step only generates an image. Video rendering starts when you choose to use the preview.</p>
+ <p className="hint">This step only generates an image. After reviewing it, choose the video settings and press Render video.</p>
</section>
- </div>
+ </div>}
{busy && <p role="status">{submitting ? "Submitting…" : job?.stage === "prepare" ? "FLUX is preparing the image…" : "Generating normals and rendering the video…"}</p>}
{error && <p className="error" role="alert">{error}</p>}
- {preview && <section className="panel">
+ {preview && !approved && <section className="panel">
<h2><span>3</span> Review before rendering</h2>
<div className="inputs picture-comparison">
<figure><figcaption>Source frame · image 2</figcaption><img src={preview.frame} alt="Raw source video frame used for pose and body cut" /></figure>
@@ -142,9 +169,46 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
<div className="row">
<a href={preview.url} download="prepared-frame.png">Download image</a>
<button className="secondary" disabled={busy || !video || !picture || !size} onClick={() => prepare((seed + 1) % (MAX_SEED + 1))}>Generate another image</button>
- <button className="primary" disabled={busy} onClick={render}>Use this image · render video</button>
+ <button className="primary" disabled={busy} onClick={() => { setApproved(true); remember(null, prepared, true); }}>Use this image</button>
</div>
</section>}
+ {preview && approved && <section className="panel">
+ <h2><span>4</span> Video render settings</h2>
+ <div className="row"><img className="picture-reference" src={preview.url} alt="Approved FLUX image for the video" style={{ width: 120, maxHeight: 180 }} /><button className="secondary" disabled={busy} onClick={() => { setApproved(false); remember(null, prepared, false); }}>Back to image review</button></div>
+ <p className="hint">One clip, up to 15 seconds. No context loop. The approved image is reused without running FLUX again.</p>
+ {modelError && <p className="error" role="alert">{modelError}</p>}
+ <fieldset disabled={busy} style={{ border: 0, padding: 0, margin: 0, minWidth: 0 }}>
+ <div className="inputs">
+ <div className="panel">
+ <label>Video length (seconds)<input className="steps-input" type="number" min={5 / 24} max={15} step={0.1} value={settings.seconds} onChange={(event) => setSettings({ ...settings, seconds: Number(event.target.value) })} /></label>
+ <small>H3 rounds down to a valid clip length: {frames} frames / {(frames / 24).toFixed(3)} s at 24 fps.</small>
+ <label>Video resolution<select className="file-name" value={typeof settings.resolution === "string" ? settings.resolution : "custom"} onChange={(event) => setSettings({ ...settings, resolution: event.target.value === "custom" ? { width: canvas[0], height: canvas[1] } : event.target.value as PictureVideoSettings["resolution"] })}>{RESOLUTIONS.map((value) => <option key={value}>{value}</option>)}<option value="custom">Custom</option></select></label>
+ {typeof settings.resolution === "object" && <div className="row">{(["width", "height"] as const).map((side) => <label key={side}>Custom {side}<input className="steps-input" type="number" min={CUSTOM_SIZE.min} max={CUSTOM_SIZE.max} step={CUSTOM_SIZE.step} value={typeof settings.resolution === "object" ? settings.resolution[side] : 0} onChange={(event) => setSettings({ ...settings, resolution: { width: canvas[0], height: canvas[1], [side]: Number(event.target.value) } })} /></label>)}</div>}
+ <small>Video canvas: {canvas[0]}×{canvas[1]}.</small>
+ <label>Motion reference resolution<select className="file-name" value={settings.motion} onChange={(event) => setSettings({ ...settings, motion: Number(event.target.value) as PictureVideoSettings["motion"] })}><option value={0}>Match video</option>{MOTION_SIZES.map((value) => <option key={value} value={value}>{value} px</option>)}</select></label>
+ <small>The motion/mask resolution control uses a normal-map guide here. Its size is separate from the output video.</small>
+ <label>Video seed<input className="steps-input" type="number" min={0} max={MAX_SEED} step={1} value={settings.seed} onChange={(event) => setSettings({ ...settings, seed: Number(event.target.value) })} /></label>
+ <label>Steps<input className="steps-input" type="number" min={STEPS.min} max={STEPS.max} value={settings.steps} onChange={(event) => setSettings({ ...settings, steps: Number(event.target.value) })} /></label>
+ <small>Use 6 with DMAD; without a speed LoRA, use 20 or more.</small>
+ </div>
+ <div className="panel">
+ <p className="hint">Choose models and VAEs compatible with MiniMax H3.</p>
+ {picker("Base model", settings.model, files.models, (model) => setSettings({ ...settings, model }))}
+ {picker("H3 text encoder", settings.textEncoder, files.textEncoders, (textEncoder) => setSettings({ ...settings, textEncoder }))}
+ {picker("Video VAE", settings.videoVae, files.vaes, (videoVae) => setSettings({ ...settings, videoVae }))}
+ {picker("Audio VAE", settings.audioVae, files.vaes, (audioVae) => setSettings({ ...settings, audioVae }))}
+ <label className="setting toggle"><span>Use LoRAs</span><input type="checkbox" checked={settings.useLoras} onChange={(event) => setSettings({ ...settings, useLoras: event.target.checked })} /><i /></label>
+ {settings.useLoras && <>{settings.loras.map((lora, index) => <div className="panel" key={index}>
+ {picker(`LoRA ${index + 1}`, lora.name, files.loras, (name) => setSettings({ ...settings, loras: settings.loras.map((item, i) => i === index ? { ...item, name } : item) }))}
+ <label>LoRA {index + 1} strength<input className="steps-input" type="number" min={STRENGTH.min} max={STRENGTH.max} step={0.05} value={lora.strength} onChange={(event) => setSettings({ ...settings, loras: settings.loras.map((item, i) => i === index ? { ...item, strength: Number(event.target.value) } : item) })} /></label>
+ <label className="setting toggle"><span>Use LoRA {index + 1}</span><input type="checkbox" checked={lora.enabled} onChange={(event) => setSettings({ ...settings, loras: settings.loras.map((item, i) => i === index ? { ...item, enabled: event.target.checked } : item) })} /><i /></label>
+ <button className="secondary" onClick={() => setSettings({ ...settings, loras: settings.loras.filter((_, i) => i !== index) })}>Remove LoRA {index + 1}</button>
+ </div>)}<button className="secondary" disabled={settings.loras.length >= MAX_LORAS} onClick={() => setSettings({ ...settings, loras: [...settings.loras, { name: files.loras[0] ?? "", strength: 1, enabled: true }] })}>Add LoRA</button></>}
+ </div>
+ </div>
+ </fieldset>
+ <button className="primary" disabled={busy} onClick={render}>Render video</button>
+ </section>}
{result && <section className="panel"><h2>Video result</h2><video src={result} controls /><a href={result} download="picture-motion.mp4">Download video</a></section>}
</div>;
}
diff --git a/lib/picture.ts b/lib/picture.ts
new file mode 100644
index 0000000..57c7d23
--- /dev/null
+++ b/lib/picture.ts
@@ -0,0 +1,68 @@
+import { DEFAULT_MODEL, MAX_LORAS, STEPS, STRENGTH } from "./model.ts";
+import { canvasSize, MAX_SEED, MOTION_SIZES, RESOLUTIONS, type ApiGraph, type MotionSize, type Resolution } from "./simple.ts";
+
+export type PictureVideoSettings = {
+ seconds: number; resolution: Resolution; motion: 0 | MotionSize; seed: number; steps: number;
+ model: string; useLoras: boolean; loras: { name: string; strength: number; enabled: boolean }[];
+ textEncoder: string; videoVae: string; audioVae: string;
+};
+
+export const DEFAULT_PICTURE_VIDEO: PictureVideoSettings = {
+ seconds: 6, resolution: "480p", motion: 0, seed: 0, steps: DEFAULT_MODEL.steps,
+ model: DEFAULT_MODEL.model, useLoras: true, loras: DEFAULT_MODEL.loras.map((lora) => ({ ...lora, enabled: true })),
+ textEncoder: "qwen3vl_32b_minimax_h3_int4_convrot.safetensors",
+ videoVae: "minimax_h3_video_vae_int8_convrot.safetensors", audioVae: "minimax_h3_audio_vae_fp32.safetensors",
+};
+
+export function pictureFrames(seconds: number) {
+ return Math.max(5, Math.floor((Math.floor(seconds * 24 + 1e-6) - 5) / 17) * 17 + 5);
+}
+
+export function parsePictureVideoSettings(value: unknown): PictureVideoSettings {
+ if (value !== undefined && (!value || typeof value !== "object" || Array.isArray(value))) throw new Error("Choose video render settings.");
+ const input = { ...DEFAULT_PICTURE_VIDEO, ...value as Partial<PictureVideoSettings> };
+ const seconds = Number(input.seconds), seed = Number(input.seed), steps = Number(input.steps), motion = Number(input.motion);
+ if (!Number.isFinite(seconds) || seconds < 5 / 24 || seconds > 15) throw new Error("Video length must be between 5 frames and 15 seconds.");
+ if (!Number.isInteger(seed) || seed < 0 || seed > MAX_SEED) throw new Error(`Seed must be a whole number from 0 to ${MAX_SEED}.`);
+ if (!Number.isInteger(steps) || steps < STEPS.min || steps > STEPS.max) throw new Error(`Steps must be ${STEPS.min}-${STEPS.max}.`);
+ if (![0, ...MOTION_SIZES].includes(motion)) throw new Error("Choose a supported motion reference resolution.");
+ let resolution: Resolution;
+ if (typeof input.resolution === "string" && RESOLUTIONS.includes(input.resolution)) resolution = input.resolution;
+ else {
+ const size = input.resolution;
+ if (!size || typeof size !== "object" || ![size.width, size.height].every((side) => Number.isFinite(side) && side > 0)) throw new Error("Choose a video resolution or positive custom dimensions.");
+ resolution = { width: Math.max(32, Math.round(size.width / 32) * 32), height: Math.max(32, Math.round(size.height / 32) * 32) };
+ }
+ if (typeof input.useLoras !== "boolean" || !Array.isArray(input.loras) || input.loras.length > MAX_LORAS) throw new Error(`Use at most ${MAX_LORAS} LoRAs.`);
+ const loras = input.loras.map((lora) => {
+ if (!lora || typeof lora.name !== "string" || !Number.isFinite(lora.strength) || lora.strength < STRENGTH.min || lora.strength > STRENGTH.max) throw new Error(`Choose each LoRA and a strength from ${STRENGTH.min} to ${STRENGTH.max}.`);
+ return { name: lora.name.trim(), strength: lora.strength, enabled: lora.enabled !== false };
+ });
+ for (const key of ["model", "textEncoder", "videoVae", "audioVae"] as const) if (typeof input[key] !== "string" || !input[key].trim()) throw new Error("Choose the video model, H3 text encoder and VAEs.");
+ return { seconds, seed, steps, motion: motion as 0 | MotionSize, resolution, useLoras: input.useLoras, loras,
+ model: input.model.trim(), textEncoder: input.textEncoder.trim(), videoVae: input.videoVae.trim(), audioVae: input.audioVae.trim() };
+}
+
+export function applyPictureVideoSettings(graph: ApiGraph, settings: PictureVideoSettings, sourceWidth: number, sourceHeight: number) {
+ const [width, height] = typeof settings.resolution === "string" ? canvasSize(sourceWidth, sourceHeight, settings.resolution) : [settings.resolution.width, settings.resolution.height];
+ const length = pictureFrames(settings.seconds);
+ graph["1"].inputs.frame_load_cap = length;
+ Object.assign(graph["56"].inputs, { width, height, length });
+ const scale = settings.motion ? settings.motion / Math.max(width, height) : 1;
+ Object.assign(graph["33"].inputs, { width: Math.max(32, Math.round(width * scale / 32) * 32), height: Math.max(32, Math.round(height * scale / 32) * 32), crop: "center" });
+ graph["33"]._meta = { title: "Normal-map motion reference resolution" };
+ graph["58"].inputs.noise_seed = settings.seed;
+ graph["60"].inputs.steps = settings.steps;
+ graph["50"].inputs.unet_name = settings.model;
+ graph["53"].inputs.clip_name = settings.textEncoder;
+ graph["54"].inputs.vae_name = settings.videoVae;
+ graph["55"].inputs.vae_name = settings.audioVae;
+ delete graph["51"];
+ let model: [string, number] = ["50", 0];
+ if (settings.useLoras) settings.loras.filter((lora) => lora.enabled).forEach((lora, index) => {
+ const id = `picture_lora_${index}`;
+ graph[id] = { class_type: "LoraLoaderModelOnly", inputs: { model, lora_name: lora.name, strength_model: lora.strength }, _meta: { title: `LoRA ${index + 1}` } };
+ model = [id, 0];
+ });
+ graph["52"].inputs.model = model;
+}
diff --git a/test/picture-route.test.ts b/test/picture-route.test.ts
index 256d4b1..f74b449 100644
--- a/test/picture-route.test.ts
+++ b/test/picture-route.test.ts
@@ -5,6 +5,7 @@ import type { AddressInfo } from "node:net";
import test from "node:test";
import { GET, POST } from "../app/api/picture/route.ts";
import type { ApiGraph } from "../lib/simple.ts";
+import { DEFAULT_PICTURE_VIDEO } from "../lib/picture.ts";
let calls = 0, uploads = 0;
const prompts: ApiGraph[] = [];
@@ -20,7 +21,16 @@ const server = createServer(async (request, response) => {
response.end("test"); return;
}
let data: unknown;
- if (request.url === "/upload/image") {
+ if (request.url?.startsWith("/object_info/")) {
+ const node = request.url.split("/").pop()!;
+ const inputs = {
+ UNETLoader: { unet_name: [[DEFAULT_PICTURE_VIDEO.model, "another_h3.safetensors"]] },
+ LoraLoaderModelOnly: { lora_name: [[DEFAULT_PICTURE_VIDEO.loras[0].name, "style.safetensors"]] },
+ CLIPLoader: { clip_name: [[DEFAULT_PICTURE_VIDEO.textEncoder, "another_encoder.safetensors"]] },
+ VAELoader: { vae_name: [[DEFAULT_PICTURE_VIDEO.videoVae, DEFAULT_PICTURE_VIDEO.audioVae, "another_vae.safetensors"]] },
+ };
+ data = { [node]: { input: { required: inputs[node as keyof typeof inputs] } } };
+ } else if (request.url === "/upload/image") {
for await (const _ of request) {}
data = { name: `uploaded-${++uploads}.png`, subfolder: "picture" };
} else if (request.url === "/prompt") {
@@ -47,8 +57,8 @@ function inputs() {
form.set("width", "1280"); form.set("height", "720"); form.set("start", "2.5"); form.set("seed", "42");
return form;
}
-function render(prepared: string) {
- return POST(new Request(url, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prepared, filename: "untrusted.png", prompt: { untrusted: true } }) }));
+function render(prepared: string, settings?: unknown) {
+ return POST(new Request(url, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prepared, settings, filename: "untrusted.png", prompt: { untrusted: true } }) }));
}
function completed() {
return { prompt: [0, "picture-1", prompts[0]], status: { status_str: "success", completed: true }, outputs: {
@@ -121,6 +131,31 @@ test("render rejects unfinished, unrelated and failed histories instead of accep
assert.equal((await render("picture-1")).status, 422); assert.equal(prompts.length, count);
});
+test("render applies selected length, independent guide resolution, model components, LoRAs and steps", async () => {
+ history = { "picture-1": completed() };
+ const response = await render("picture-1", { ...DEFAULT_PICTURE_VIDEO, seconds: 15, resolution: { width: 640, height: 960 }, motion: 512,
+ model: "another_h3.safetensors", steps: 12, seed: 99, textEncoder: "another_encoder.safetensors", videoVae: "another_vae.safetensors",
+ loras: [{ name: "style.safetensors", strength: 0.6, enabled: true }, { name: "unused.safetensors", strength: 1, enabled: false }] });
+ assert.equal(response.status, 202);
+ const graph = prompts.at(-1)!;
+ assert.equal(graph["1"].inputs.frame_load_cap, 345); assert.equal(graph["56"].inputs.length, 345);
+ assert.equal(graph["56"].inputs.width, 640); assert.equal(graph["56"].inputs.height, 960);
+ assert.equal(graph["33"].inputs.width, 352); assert.equal(graph["33"].inputs.height, 512);
+ assert.equal(graph["50"].inputs.unet_name, "another_h3.safetensors"); assert.equal(graph["60"].inputs.steps, 12);
+ assert.equal(graph["53"].inputs.clip_name, "another_encoder.safetensors"); assert.equal(graph["54"].inputs.vae_name, "another_vae.safetensors");
+ assert.equal(graph["picture_lora_0"].inputs.strength_model, 0.6); assert.ok(!graph["picture_lora_1"]);
+ assert.equal(graph["58"].inputs.noise_seed, 99);
+ assert.ok(!Object.values(graph).some((node) => /Chain|Loop/.test(node.class_type)));
+});
+
+test("invalid render settings are rejected before submitting or copying the prepared image", async () => {
+ const count = prompts.length, copied = uploads;
+ for (const settings of [{ seconds: 16 }, { seconds: 0 }, { steps: 0 }, { model: "missing.safetensors" }, { textEncoder: "missing.safetensors" }, { motion: 999 }]) {
+ assert.equal((await render("picture-1", { ...DEFAULT_PICTURE_VIDEO, ...settings })).status, 400);
+ }
+ assert.equal(prompts.length, count); assert.equal(uploads, copied);
+});
+
test("video media supports seeking and submission errors identify missing models", async () => {
history = { "picture-2": { prompt: [0, "picture-2", prompts[1]], outputs: { "63": { gifs: [{ filename: "motion.mp4", type: "output" }] } } } };
const response = await GET(new Request(`${url}?id=picture-2&media=result`, { headers: { Range: "bytes=0-3" } }));
diff --git a/test/picture-settings.test.ts b/test/picture-settings.test.ts
new file mode 100644
index 0000000..468e504
--- /dev/null
+++ b/test/picture-settings.test.ts
@@ -0,0 +1,58 @@
+import assert from "node:assert/strict";
+import { readFileSync } from "node:fs";
+import test from "node:test";
+import { applyPictureVideoSettings, DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureFrames } from "../lib/picture.ts";
+import type { ApiGraph } from "../lib/simple.ts";
+
+const graph = (): ApiGraph => JSON.parse(readFileSync(new URL("../workflows/picture/from_picture_motion_flux3.api.json", import.meta.url), "utf8"));
+
+test("single-clip length is aligned down without exceeding the selected duration or 15 seconds", () => {
+ assert.equal(pictureFrames(6), 141);
+ assert.equal(pictureFrames(15), 345);
+ assert.equal(pictureFrames(1), 22);
+ for (const seconds of [5 / 24, 1, 2.5, 6, 9.3, 15]) {
+ const frames = pictureFrames(seconds);
+ assert.equal(frames % 17, 5); assert.ok(frames / 24 <= seconds + 1e-6);
+ }
+ assert.throws(() => parsePictureVideoSettings({ seconds: 15.1 }), /15 seconds/);
+});
+
+test("video and normal-map reference dimensions can be set independently", () => {
+ const value = graph();
+ applyPictureVideoSettings(value, parsePictureVideoSettings({ resolution: "768p", motion: 512 }), 768, 1376);
+ assert.equal(value["56"].inputs.width, 768); assert.equal(value["56"].inputs.height, 1376);
+ assert.equal(value["33"].inputs.width, 288); assert.equal(value["33"].inputs.height, 512);
+ applyPictureVideoSettings(value, parsePictureVideoSettings({ resolution: { width: 650, height: 970 }, motion: 0 }), 768, 1376);
+ assert.equal(value["56"].inputs.width, 640); assert.equal(value["33"].inputs.width, 640);
+ assert.equal(value["56"].inputs.height, 960); assert.equal(value["33"].inputs.height, 960);
+});
+
+test("LoRAs are applied in order, disabled entries are skipped, and turning them off uses the base directly", () => {
+ const value = graph();
+ const settings = parsePictureVideoSettings({ model: "base.safetensors", steps: 25, loras: [
+ { name: "first.safetensors", strength: 0.7, enabled: true },
+ { name: "disabled.safetensors", strength: 1, enabled: false },
+ { name: "last.safetensors", strength: -0.2, enabled: true },
+ ] });
+ applyPictureVideoSettings(value, settings, 768, 1376);
+ assert.deepEqual(value["picture_lora_0"].inputs.model, ["50", 0]);
+ assert.deepEqual(value["picture_lora_1"].inputs.model, ["picture_lora_0", 0]);
+ assert.equal(value["picture_lora_1"].inputs.lora_name, "last.safetensors");
+ assert.deepEqual(value["52"].inputs.model, ["picture_lora_1", 0]);
+ assert.ok(!value["51"]); assert.equal(value["60"].inputs.steps, 25);
+ const plain = graph();
+ applyPictureVideoSettings(plain, { ...settings, useLoras: false }, 768, 1376);
+ assert.deepEqual(plain["52"].inputs.model, ["50", 0]);
+ assert.ok(!Object.values(plain).some((node) => node.class_type === "LoraLoaderModelOnly"));
+});
+
+test("default settings keep the tested canvas, length and DMAD configuration", () => {
+ const value = graph();
+ applyPictureVideoSettings(value, parsePictureVideoSettings(undefined), 768, 1376);
+ assert.equal(value["56"].inputs.length, 141); assert.equal(value["1"].inputs.frame_load_cap, 141);
+ assert.equal(value["56"].inputs.width, 480); assert.equal(value["56"].inputs.height, 864);
+ assert.equal(value["33"].inputs.width, 480); assert.equal(value["33"].inputs.height, 864);
+ assert.equal(value["picture_lora_0"].inputs.lora_name, DEFAULT_PICTURE_VIDEO.loras[0].name);
+ assert.equal(value["60"].inputs.steps, 6);
+ assert.ok(!Object.values(value).some((node) => /Chain|Loop/.test(node.class_type)));
+});
diff --git a/workflows/picture/README.md b/workflows/picture/README.md
index 65208ac..74cfc50 100644
--- a/workflows/picture/README.md
+++ b/workflows/picture/README.md
@@ -2,7 +2,7 @@
These are ComfyUI API-format workflows. Drop a JSON file onto ComfyUI, or use **File → Open**.
-The app uses them under **Object swap → From picture**. Choose the video and character picture, then **Generate preview image**. Compare the FLUX image with the raw frame, adjust the prompt or generate another image if needed, and click **Use this image · render video** when ready. Previewing never starts H3 automatically. The video stage loads the saved preview image and does not rerun FLUX. Both stages use `COMFYUI_URL` directly; comfy-api-proxy is not required for this tab. Pending jobs and the selected preview resume when returning to the tab or refreshing the page.
+The app uses them under **Object swap → From picture**. Choose the video and character picture, then **Generate preview image**. Compare the FLUX image with the raw frame, adjust the prompt or generate another image if needed, and click **Use this image** when ready. Set the video length, video resolution, motion-reference resolution, models, LoRAs, steps and seed, then click **Render video**. Previewing never starts H3 automatically. The video stage loads the saved preview image and does not rerun FLUX. Both stages use `COMFYUI_URL` directly; comfy-api-proxy is not required for this tab. Pending jobs and the selected preview resume when returning to the tab or refreshing the page.
| Workflow | Result |
| --- | --- |
@@ -21,9 +21,9 @@ The image-aware prompt upsampler sees both references and expands that request i
Preparation defaults to a 768×1376 canvas and saves under `ComfyUI/output/openjutsu_picture/flux3/prepared_frame_*.png`. Compare the prepared picture against the raw first-frame preview before using it for video; a successful execution does not guarantee that the requested pose was followed.
-The **motion** variants wire the decoded prepared picture directly into H3. No manual re-upload is needed. They default to **141 frames / 5.875 seconds at 24 fps**, 480×864, six DMAD steps, and the original soundtrack (or silence if absent). They are single-clip workflows; they do not process a whole longer video or add context-loop continuation. Change `frame_load_cap` and H3's `length` together to select another valid H3 clip length.
+The **motion** variants wire the decoded prepared picture directly into H3. No manual re-upload is needed. They default to **141 frames / 5.875 seconds at 24 fps**, 480×864, six DMAD steps, and the original soundtrack (or silence if absent). The app lets you select up to 15 seconds, with no context loop. It rounds down to the H3 17k+5 frame grid: 6 seconds becomes 141 frames / 5.875 seconds; 15 seconds becomes 345 frames / 14.375 seconds. The duration shown in the settings is the actual render length. Models, LoRAs (including individual switches and strengths), H3 text encoder, VAEs, steps and seed can be changed before rendering. These settings persist across refreshes. Change `frame_load_cap` and H3's `length` together to select another valid H3 clip length.
-Normals are resized directly to the H3 output canvas. If changing that canvas, change both **Normals match the output canvas** and **Prepared picture + normal motion reference**. Normals are soft reference-video conditioning, not a trained normal ControlNet. The VRAM cleanup node runs after the image edit and before Metric3D to make room for its model.
+By default normals match the H3 output canvas. The app also offers independent 512, 768 and 1376 px motion-reference sizes, rounded to multiples of 32; native H3 can cap large reference canvases. This is the motion/mask resolution control for this mode; no SAM edit mask is used. If changing that canvas, change both **Normals match the output canvas** and **Prepared picture + normal motion reference**. Normals are soft reference-video conditioning, not a trained normal ControlNet. The VRAM cleanup node runs after the image edit and before Metric3D to make room for its model.
Required installed models are selected in the loader nodes. Defaults match this installation: