One-click character swap for videos: an open-source, local Higgsfield Genjutsu alternative built on ComfyUI and MiniMax H3, with FLUX.3 Klein relighting.
Add SolarRenderer to From picture
D3vozbe8cbda71ae8
diff --git a/README.md b/README.mdindex 50bf4d7..5a45f5a 100644--- a/README.md+++ b/README.md@@ -15,7 +15,7 @@ Choose according to available VRAM and leave room for image/video context and wo- **Simple motion transfer**: available. One character replaces the main person (the one who fills the most of the frame). Runs `workflows/simple.api.json`, exported from `H3 One-Click Single Swap (DMAD 6 step).json`.- **Advanced motion transfer**: available. 1–4 characters, each with an optional role; every tracked person is matched to a role per shot, and people without a character stay untouched. Runs `workflows/advanced.api.json`, exported from `H3 One-Click Advanced Swap (DMAD 6 step).json`; **One pass per person** (two characters) runs `workflows/advanced_maxq.api.json`.- **Object swap**: available. **Change** what SAM tracks (hair, a jacket, a car) into what you describe or show in a picture; **Remove** it; or change the **Background** around it (`person` stays). Only that area is regenerated. Runs `workflows/object.api.json`, exported from `H3 One-Click Object Swap (DMAD 6 step).json`.-- **From picture** (under Object swap): FLUX prepares the character in the raw video frame's pose and body cut while keeping the character picture's background. Review or regenerate the image, then choose **Use this image** to set the video length (up to 15 seconds), video and motion-reference resolutions, models, LoRAs, steps and seed. **Render video** reuses the saved preview with normal-map motion guidance and the source soundtrack, in one clip without a context loop. H3 rounds the requested length down to its valid frame grid. See [the workflow requirements](workflows/picture/README.md).+- **From picture** (under Object swap): FLUX prepares the character in the raw video frame's pose and body cut while keeping the character picture's background. Review or regenerate the image, then choose **Use this image** to set the video length, video and motion-reference resolutions, models, LoRAs, steps and seed. SolarRenderer defaults to four steps with a selectable rank-384 or rank-64 four-step/control pair and supports up to 15 seconds, like the native H3 renderer. SolarRenderer clips longer than six seconds can take considerably longer to render, especially at higher resolutions; see the workflow requirements for the extended-silence node fix. Both LoRA file pickers support `models/loras` and its subfolders. **Render video** reuses the saved preview with normal-map motion guidance and the source soundtrack, in one clip without a context loop. H3 rounds the requested length down to its valid frame grid. See [the workflow requirements](workflows/picture/README.md).- **Post-processing**: available, for a finished video. **Fix faces**: H3 draws faces poorly when the person is small in the frame, so this redraws the face of the person in a photo at close-up size and pastes it back. Runs `workflows/face.api.json`, written by `workflows/build/openjutsu_face_build.py`. **Fix colours** softens H3's harsh look without an H3 run: Image Detail Contrast takes some fine detail contrast out, then Orion4D's DCTL Tone Mapper grades every frame. Runs `workflows/color.api.json`.## Requirements@@ -30,6 +30,7 @@ Install these into `ComfyUI/custom_nodes` (ComfyUI Manager finds the first three| Pack | Used for || --- | --- |+| [SolarRenderer](https://github.com/Enderfga/SolarRenderer/tree/main/comfyui) | From picture: SolarRenderer LoRA loading, conditioning and four-step sampling || [ComfyUI-MiniMaxH3-Context-Loop](https://github.com/ethanfel/ComfyUI-MiniMaxH3-Context-Loop) 0.7.3+ | clip loop, checkpoints, final assembly || [ComfyUI-Fantastic-MiniMaxH3-PromptBuilder](https://github.com/Adudeguyman/ComfyUI-Fantastic-MiniMaxH3-PromptBuilder) | masked video-edit latent || [ComfyUI-KJNodes](https://github.com/kijai/ComfyUI-KJNodes) | Grow Mask With Blur |@@ -47,6 +48,8 @@ Before uploading a job, Openjutsu checks the selected workflow against ComfyUI's### Models+For **From picture → SolarRenderer**, download [solarrenderer_4step_lora.safetensors](https://huggingface.co/Enderfga/SolarRenderer/blob/main/solarrenderer_4step_lora.safetensors) and [solarrenderer_control_lora.safetensors](https://huggingface.co/Enderfga/SolarRenderer/blob/main/solarrenderer_control_lora.safetensors) from [Enderfga/SolarRenderer](https://huggingface.co/Enderfga/SolarRenderer), or the corresponding rank-64 pair from [mdevoz/SolarRendererRank64](https://huggingface.co/mdevoz/SolarRendererRank64). Put both in `ComfyUI/models/loras` or a subfolder, reload the app and choose their paths in the render settings.+| Folder in `ComfyUI/models` | File || --- | --- || `diffusion_models` | `minimax_h3_ref2va_pruned_int8_convrot.safetensors` |diff --git a/app/api/picture/route.ts b/app/api/picture/route.tsindex 4b03fb7..4d0c15b 100644--- a/app/api/picture/route.ts+++ b/app/api/picture/route.ts@@ -5,7 +5,7 @@ import { join } from "node:path";import { apiAccess, HttpError, IMAGE_SIGNATURE, readFormData, StoredFile, validateJobId, VIDEO_SIGNATURE } from "../../../lib/http.ts";import { canvasSize, isLink, MAX_SEED, type ApiGraph } from "../../../lib/simple.ts";import { listModels } from "../../../lib/comfy.ts";-import { applyPictureVideoSettings, DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, type PictureVideoSettings } from "../../../lib/picture.ts";+import { applyPictureVideoSettings, DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureLoras, solarLoraFiles, type PictureVideoSettings } from "../../../lib/picture.ts";export const runtime = "nodejs";const PREPARED = "OpenJutsu · prepared picture";@@ -29,7 +29,7 @@ async function comfy(path: string, init?: RequestInit) {return response;}-async function template(stage: "prepare" | "motion"): Promise<ApiGraph> {+async function template(stage: "prepare" | "motion" | "solar"): Promise<ApiGraph> {return JSON.parse(await readFile(join(process.cwd(), "workflows", "picture", `from_picture_${stage}_flux3.api.json`), "utf8"));}@@ -85,15 +85,22 @@ export async function POST(request: Request) {catch (error) { throw new HttpError(400, (error as Error).message); }const available = await listModels();if (!available.models.includes(settings.model)) throw new HttpError(400, "Choose a video model ComfyUI has.");- if (settings.useLoras && settings.loras.some((lora) => lora.enabled && !available.loras.includes(lora.name))) throw new HttpError(400, "Choose LoRAs ComfyUI has.");+ if (settings.renderer === "solar") {+ const installed = solarLoraFiles(settings.solarRank, available.loras);+ const defaults = solarLoraFiles(settings.solarRank);+ for (const key of ["solarFourStep", "solarControl"] as const) if (settings[key] === defaults[key]) settings[key] = installed[key];+ const nodes = await (await comfy("/object_info/SolarRendererLoraLoader")).json();+ if (!nodes.SolarRendererLoraLoader) throw new HttpError(400, "Install the SolarRenderer custom nodes and restart ComfyUI: https://github.com/Enderfga/SolarRenderer");+ }+ if (pictureLoras(settings).some((name) => !available.loras.includes(name))) throw new HttpError(400, "Choose LoRA files installed in ComfyUI/models/loras (subfolders are supported).");const [clipInfo, vaeInfo] = await Promise.all([comfy("/object_info/CLIPLoader").then((response) => response.json()), comfy("/object_info/VAELoader").then((response) => response.json())]);for (const [name, choices, label] of [[settings.textEncoder, clipInfo.CLIPLoader.input.required.clip_name[0], "H3 text encoder"],[settings.videoVae, vaeInfo.VAELoader.input.required.vae_name[0], "video VAE"],- [settings.audioVae, vaeInfo.VAELoader.input.required.vae_name[0], "audio VAE"],+ ...(settings.renderer === "h3" ? [[settings.audioVae, vaeInfo.VAELoader.input.required.vae_name[0], "audio VAE"] as const] : []),] as const) if (!choices.includes(name)) throw new HttpError(400, `Choose a ${label} ComfyUI has.`);const picture = await (await comfy(view(file))).blob();- graph = await template("motion");+ graph = await template(settings.renderer === "solar" ? "solar" : "motion");graph["20"] = { class_type: "LoadImage", inputs: { image: await uploadFile(picture, ".png") } };graph["1"].inputs.video = source["1"].inputs.video;graph["1"].inputs.skip_first_frames = source["1"].inputs.skip_first_frames;diff --git a/components/from-picture.tsx b/components/from-picture.tsxindex 4790820..cd56854 100644--- a/components/from-picture.tsx+++ b/components/from-picture.tsx@@ -3,7 +3,7 @@import { useEffect, useState } from "react";import { canvasSize, CUSTOM_SIZE, MAX_SEED, MOTION_SIZES, RESOLUTIONS } from "../lib/simple.ts";import { MAX_LORAS, STEPS, STRENGTH } from "../lib/model.ts";-import { DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureFrames, type PictureVideoSettings } from "../lib/picture.ts";+import { DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureFrames, solarLoraFiles, type PictureVideoSettings } from "../lib/picture.ts";const PROMPT = "change <image 1> character pose follow the <image 2> pose. keep the final image as same body cut as <image 2> . do not change the background of <image 1> and do not add anything extra in <image 1>.";const STORAGE = "openjutsu.picture-preview";@@ -52,6 +52,11 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStartconst data = await response.json();if (!response.ok) throw new Error(data.error);setFiles({ models: data.models ?? [], loras: data.loras ?? [], textEncoders: data.textEncoders ?? [], vaes: data.vaes ?? [] });+ setSettings((current) => {+ const defaults = solarLoraFiles(current.solarRank), installed = solarLoraFiles(current.solarRank, data.loras ?? []);+ return { ...current, solarFourStep: current.solarFourStep === defaults.solarFourStep ? installed.solarFourStep : current.solarFourStep,+ solarControl: current.solarControl === defaults.solarControl ? installed.solarControl : current.solarControl };+ });}).catch((error) => setModelError(error.message));}, []);useEffect(() => {@@ -177,28 +182,39 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart{preview && approved && <section className="panel"><h2><span>4</span> Video render settings</h2><div className="row"><img className="picture-reference" src={preview.url} alt="Approved FLUX image for the video" style={{ width: 120, maxHeight: 180 }} /><button className="secondary" disabled={busy} onClick={() => { setApproved(false); remember(null, prepared, false); }}>Back to image review</button></div>- <p className="hint">One clip, up to 15 seconds. No context loop. The approved image is reused without running FLUX again.</p>+ <p className="hint">The approved image is reused without running FLUX again. Native H3 and SolarRenderer render one clip, up to 15 seconds, without a context loop.</p>{modelError && <p className="error" role="alert">{modelError}</p>}<fieldset disabled={busy} style={{ border: 0, padding: 0, margin: 0, minWidth: 0 }}><div className="inputs"><div className="panel">- <label>Video length (seconds)<input className="steps-input" type="number" min={5 / 24} max={15} step={0.1} value={settings.seconds} onChange={(event) => setSettings({ ...settings, seconds: Number(event.target.value) })} /></label>+ <label>Video renderer<select className="file-name" value={settings.renderer} onChange={(event) => setSettings({ ...settings, renderer: event.target.value as PictureVideoSettings["renderer"], steps: event.target.value === "solar" ? 4 : 6 })}><option value="solar">SolarRenderer</option><option value="h3">H3 · native</option></select></label>+ <label>Video length (seconds)<input className="steps-input" type="number" min={(settings.renderer === "solar" ? 22 : 5) / 24} max={15} step={0.1} value={settings.seconds} onChange={(event) => setSettings({ ...settings, seconds: Number(event.target.value) })} /></label><small>H3 rounds down to a valid clip length: {frames} frames / {(frames / 24).toFixed(3)} s at 24 fps.</small>+ {settings.renderer === "solar" && settings.seconds > 6 && <p className="hint" role="note"><strong>Longer render time:</strong> SolarRenderer clips longer than 6 seconds can take considerably longer to render, especially at higher resolutions.</p>}<label>Video resolution<select className="file-name" value={typeof settings.resolution === "string" ? settings.resolution : "custom"} onChange={(event) => setSettings({ ...settings, resolution: event.target.value === "custom" ? { width: canvas[0], height: canvas[1] } : event.target.value as PictureVideoSettings["resolution"] })}>{RESOLUTIONS.map((value) => <option key={value}>{value}</option>)}<option value="custom">Custom</option></select></label>{typeof settings.resolution === "object" && <div className="row">{(["width", "height"] as const).map((side) => <label key={side}>Custom {side}<input className="steps-input" type="number" min={CUSTOM_SIZE.min} max={CUSTOM_SIZE.max} step={CUSTOM_SIZE.step} value={typeof settings.resolution === "object" ? settings.resolution[side] : 0} onChange={(event) => setSettings({ ...settings, resolution: { width: canvas[0], height: canvas[1], [side]: Number(event.target.value) } })} /></label>)}</div>}<small>Video canvas: {canvas[0]}×{canvas[1]}.</small><label>Motion reference resolution<select className="file-name" value={settings.motion} onChange={(event) => setSettings({ ...settings, motion: Number(event.target.value) as PictureVideoSettings["motion"] })}><option value={0}>Match video</option>{MOTION_SIZES.map((value) => <option key={value} value={value}>{value} px</option>)}</select></label><small>The motion/mask resolution control uses a normal-map guide here. Its size is separate from the output video.</small><label>Video seed<input className="steps-input" type="number" min={0} max={MAX_SEED} step={1} value={settings.seed} onChange={(event) => setSettings({ ...settings, seed: Number(event.target.value) })} /></label>- <label>Steps<input className="steps-input" type="number" min={STEPS.min} max={STEPS.max} value={settings.steps} onChange={(event) => setSettings({ ...settings, steps: Number(event.target.value) })} /></label>- <small>Use 6 with DMAD; without a speed LoRA, use 20 or more.</small>+ <label>Steps<input className="steps-input" type="number" min={STEPS.min} max={settings.renderer === "solar" ? 50 : STEPS.max} value={settings.steps} onChange={(event) => setSettings({ ...settings, steps: Number(event.target.value) })} /></label>+ <small>{settings.renderer === "solar" ? "Use 4 with the SolarRenderer four-step and control LoRAs." : "Use 6 with DMAD; without a speed LoRA, use 20 or more."}</small></div><div className="panel"><p className="hint">Choose models and VAEs compatible with MiniMax H3.</p>{picker("Base model", settings.model, files.models, (model) => setSettings({ ...settings, model }))}{picker("H3 text encoder", settings.textEncoder, files.textEncoders, (textEncoder) => setSettings({ ...settings, textEncoder }))}{picker("Video VAE", settings.videoVae, files.vaes, (videoVae) => setSettings({ ...settings, videoVae }))}- {picker("Audio VAE", settings.audioVae, files.vaes, (audioVae) => setSettings({ ...settings, audioVae }))}+ {settings.renderer === "h3" && picker("Audio VAE", settings.audioVae, files.vaes, (audioVae) => setSettings({ ...settings, audioVae }))}+ {settings.renderer === "solar" ? <>+ <label>SolarRenderer LoRA rank<select className="file-name" value={settings.solarRank} onChange={(event) => { const solarRank = Number(event.target.value) as 384 | 64; setSettings({ ...settings, solarRank, ...solarLoraFiles(solarRank, files.loras) }); }}><option value={384}>Rank 384 · original</option><option value={64}>Rank 64 · compressed</option></select></label>+ {picker("SolarRenderer four-step LoRA", settings.solarFourStep, files.loras, (solarFourStep) => setSettings({ ...settings, solarFourStep }))}+ {picker("SolarRenderer control LoRA", settings.solarControl, files.loras, (solarControl) => setSettings({ ...settings, solarControl }))}+ <p className="hint">Both LoRAs are required and run at strength 1. Select their paths inside <code>ComfyUI/models/loras</code>; subfolders such as <code>solar/</code> are supported.</p>+ <p className="hint">Rank 384: from <a href="https://huggingface.co/Enderfga/SolarRenderer" target="_blank" rel="noopener noreferrer">SolarRenderer</a>, download <a href="https://huggingface.co/Enderfga/SolarRenderer/blob/main/solarrenderer_4step_lora.safetensors" target="_blank" rel="noopener noreferrer">solarrenderer_4step_lora.safetensors</a> and <a href="https://huggingface.co/Enderfga/SolarRenderer/blob/main/solarrenderer_control_lora.safetensors" target="_blank" rel="noopener noreferrer">solarrenderer_control_lora.safetensors</a>.</p>+ <p className="hint">For rank 64, download the four-step and control files from <a href="https://huggingface.co/mdevoz/SolarRendererRank64" target="_blank" rel="noopener noreferrer">SolarRendererRank64</a>, then select them above.</p>+ <p className="hint">Requires the <a href="https://github.com/Enderfga/SolarRenderer/tree/main/comfyui" target="_blank" rel="noopener noreferrer">SolarRenderer custom nodes</a>. Restart ComfyUI after installing them, and reload this page after adding LoRA files.</p>+ </> : <><label className="setting toggle"><span>Use LoRAs</span><input type="checkbox" checked={settings.useLoras} onChange={(event) => setSettings({ ...settings, useLoras: event.target.checked })} /><i /></label>{settings.useLoras && <>{settings.loras.map((lora, index) => <div className="panel" key={index}>{picker(`LoRA ${index + 1}`, lora.name, files.loras, (name) => setSettings({ ...settings, loras: settings.loras.map((item, i) => i === index ? { ...item, name } : item) }))}@@ -206,6 +222,7 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart<label className="setting toggle"><span>Use LoRA {index + 1}</span><input type="checkbox" checked={lora.enabled} onChange={(event) => setSettings({ ...settings, loras: settings.loras.map((item, i) => i === index ? { ...item, enabled: event.target.checked } : item) })} /><i /></label><button className="secondary" onClick={() => setSettings({ ...settings, loras: settings.loras.filter((_, i) => i !== index) })}>Remove LoRA {index + 1}</button></div>)}<button className="secondary" disabled={settings.loras.length >= MAX_LORAS} onClick={() => setSettings({ ...settings, loras: [...settings.loras, { name: files.loras[0] ?? "", strength: 1, enabled: true }] })}>Add LoRA</button></>}+ </>}</div></div></fieldset>diff --git a/lib/picture.ts b/lib/picture.tsindex 1860b61..681b0f4 100644--- a/lib/picture.ts+++ b/lib/picture.ts@@ -2,18 +2,33 @@ import { DEFAULT_MODEL, MAX_LORAS, STEPS, STRENGTH } from "./model.ts";import { canvasSize, MAX_SEED, MOTION_SIZES, RESOLUTIONS, type ApiGraph, type MotionSize, type Resolution } from "./simple.ts";export type PictureVideoSettings = {+ renderer: "h3" | "solar"; solarRank: 384 | 64; solarFourStep: string; solarControl: string;seconds: number; resolution: Resolution; motion: 0 | MotionSize; seed: number; steps: number;model: string; useLoras: boolean; loras: { name: string; strength: number; enabled: boolean }[];textEncoder: string; videoVae: string; audioVae: string;};export const DEFAULT_PICTURE_VIDEO: PictureVideoSettings = {- seconds: 6, resolution: "480p", motion: 0, seed: 0, steps: DEFAULT_MODEL.steps,+ renderer: "solar", solarRank: 384, ...solarLoraFiles(384),+ seconds: 6, resolution: "480p", motion: 0, seed: 0, steps: 4,model: DEFAULT_MODEL.model, useLoras: true, loras: DEFAULT_MODEL.loras.map((lora) => ({ ...lora, enabled: true })),textEncoder: "qwen3vl_32b_minimax_h3_int4_convrot.safetensors",videoVae: "minimax_h3_video_vae_int8_convrot.safetensors", audioVae: "minimax_h3_audio_vae_fp32.safetensors",};+export function solarLoraFiles(rank: 384 | 64, files: string[] = []) {+ const find = (role: string) => {+ const name = `solarrenderer_${role}_lora${rank === 64 ? "_rank64" : ""}.safetensors`;+ return files.includes(name) ? name : files.find((file) => file.replaceAll("\\", "/").split("/").at(-1) === name) ?? name;+ };+ return { solarFourStep: find("4step"), solarControl: find("control") };+}++export function pictureLoras(settings: PictureVideoSettings) {+ return settings.renderer === "solar" ? [settings.solarFourStep, settings.solarControl]+ : settings.useLoras ? settings.loras.filter((lora) => lora.enabled).map((lora) => lora.name) : [];+}+export function pictureFrames(seconds: number) {return Math.max(5, Math.floor((Math.floor(seconds * 24 + 1e-6) - 5) / 17) * 17 + 5);}@@ -21,10 +36,15 @@ export function pictureFrames(seconds: number) {export function parsePictureVideoSettings(value: unknown): PictureVideoSettings {if (value !== undefined && (!value || typeof value !== "object" || Array.isArray(value))) throw new Error("Choose video render settings.");const input = { ...DEFAULT_PICTURE_VIDEO, ...value as Partial<PictureVideoSettings> };+ if (value && !Object.hasOwn(value, "renderer") && Object.keys(value).length) input.renderer = "h3";+ if (!["h3", "solar"].includes(input.renderer)) throw new Error("Choose H3 or SolarRenderer.");+ if (![384, 64].includes(input.solarRank)) throw new Error("Choose SolarRenderer rank 384 or rank 64.");const seconds = Number(input.seconds), seed = Number(input.seed), steps = Number(input.steps), motion = Number(input.motion);if (!Number.isFinite(seconds) || seconds < 5 / 24 || seconds > 15) throw new Error("Video length must be between 5 frames and 15 seconds.");+ if (input.renderer === "solar" && pictureFrames(seconds) < 22) throw new Error("SolarRenderer needs at least 22 frames.");if (!Number.isInteger(seed) || seed < 0 || seed > MAX_SEED) throw new Error(`Seed must be a whole number from 0 to ${MAX_SEED}.`);if (!Number.isInteger(steps) || steps < STEPS.min || steps > STEPS.max) throw new Error(`Steps must be ${STEPS.min}-${STEPS.max}.`);+ if (input.renderer === "solar" && steps > 50) throw new Error("SolarRenderer supports at most 50 steps; use 4 with its four-step LoRA.");if (![0, ...MOTION_SIZES].includes(motion)) throw new Error("Choose a supported motion reference resolution.");let resolution: Resolution;if (typeof input.resolution === "string" && RESOLUTIONS.includes(input.resolution)) resolution = input.resolution;@@ -38,8 +58,9 @@ export function parsePictureVideoSettings(value: unknown): PictureVideoSettingsif (!lora || typeof lora.name !== "string" || !Number.isFinite(lora.strength) || lora.strength < STRENGTH.min || lora.strength > STRENGTH.max) throw new Error(`Choose each LoRA and a strength from ${STRENGTH.min} to ${STRENGTH.max}.`);return { name: lora.name.trim(), strength: lora.strength, enabled: lora.enabled !== false };});- for (const key of ["model", "textEncoder", "videoVae", "audioVae"] as const) if (typeof input[key] !== "string" || !input[key].trim()) throw new Error("Choose the video model, H3 text encoder and VAEs.");+ for (const key of ["model", "textEncoder", "videoVae", "audioVae", "solarFourStep", "solarControl"] as const) if (typeof input[key] !== "string" || !input[key].trim()) throw new Error("Choose the model components and both SolarRenderer LoRA files.");return { seconds, seed, steps, motion: motion as 0 | MotionSize, resolution, useLoras: input.useLoras, loras,+ renderer: input.renderer, solarRank: input.solarRank, solarFourStep: input.solarFourStep.trim(), solarControl: input.solarControl.trim(),model: input.model.trim(), textEncoder: input.textEncoder.trim(), videoVae: input.videoVae.trim(), audioVae: input.audioVae.trim() };}@@ -56,6 +77,12 @@ export function applyPictureVideoSettings(graph: ApiGraph, settings: PictureVidegraph["50"].inputs.unet_name = settings.model;graph["53"].inputs.clip_name = settings.textEncoder;graph["54"].inputs.vae_name = settings.videoVae;+ if (settings.renderer === "solar") {+ graph["51"].inputs.lora_name = settings.solarFourStep;+ graph["64"].inputs.lora_name = settings.solarControl;+ graph["65"].inputs.seed = settings.seed;+ return;+ }graph["55"].inputs.vae_name = settings.audioVae;delete graph["51"];let model: [string, number] = ["50", 0];diff --git a/patches/solarrenderer-long-clips.patch b/patches/solarrenderer-long-clips.patchnew file mode 100644index 0000000..707dd57--- /dev/null+++ b/patches/solarrenderer-long-clips.patch@@ -0,0 +1,18 @@+--- a/comfyui/nodes.py++++ b/comfyui/nodes.py+@@ -201,10 +201,11 @@+ def __call__(self, executor, x, timestep, context, transformer_options={}, **kwargs):+ video, audio = x[0], x[1]+ t = audio.shape[-1]+- if self.silence.shape[-1] < t:+- raise ValueError(f"the held silence covers 170 frames ({self.silence.shape[-1]} audio latents); "+- f"this clip needs {t}")+- clean = self.silence[..., :t].to(audio.device, torch.float32)++ silence = self.silence++ if silence.shape[-1] < t:++ middle = silence[..., silence.shape[-1] // 2: silence.shape[-1] // 2 + 1].expand(*silence.shape[:-1], t - 40)++ silence = torch.cat((silence[..., :20], middle, silence[..., -20:]), dim=-1)++ clean = silence[..., :t].to(audio.device, torch.float32)+ if self.noise is None or self.noise.shape != clean.shape:+ generator = torch.Generator("cpu").manual_seed(self.seed)+ self.noise = torch.randn(clean.shape, generator=generator, dtype=torch.float32)diff --git a/test/picture-pipeline.test.ts b/test/picture-pipeline.test.tsindex b79c91f..e7458dc 100644--- a/test/picture-pipeline.test.ts+++ b/test/picture-pipeline.test.ts@@ -42,7 +42,7 @@ test("the motion workflow consumes the prepared picture directly and resizes nor});test("picture pipelines have valid links and no disconnected processing nodes", () => {- for (const stage of ["prepare", "motion"]) {+ for (const stage of ["prepare", "motion", "solar"]) {const graph = load(stage);const reached = new Set<string>();function visit(id: string) {diff --git a/test/picture-route.test.ts b/test/picture-route.test.tsindex f74b449..1ea40c4 100644--- a/test/picture-route.test.ts+++ b/test/picture-route.test.ts@@ -10,7 +10,7 @@ import { DEFAULT_PICTURE_VIDEO } from "../lib/picture.ts";let calls = 0, uploads = 0;const prompts: ApiGraph[] = [];let history: Record<string, unknown> = {};-let queued = false, rejectPrompt = false;+let queued = false, rejectPrompt = false, solarInstalled = true;let range: string | undefined;let viewed = "";const server = createServer(async (request, response) => {@@ -25,11 +25,12 @@ const server = createServer(async (request, response) => {const node = request.url.split("/").pop()!;const inputs = {UNETLoader: { unet_name: [[DEFAULT_PICTURE_VIDEO.model, "another_h3.safetensors"]] },- LoraLoaderModelOnly: { lora_name: [[DEFAULT_PICTURE_VIDEO.loras[0].name, "style.safetensors"]] },+ LoraLoaderModelOnly: { lora_name: [[DEFAULT_PICTURE_VIDEO.loras[0].name, "style.safetensors", "solarrenderer_4step_lora.safetensors", "solarrenderer_control_lora.safetensors", "solar\\solarrenderer_4step_lora_rank64.safetensors", "solar\\solarrenderer_control_lora_rank64.safetensors"]] },CLIPLoader: { clip_name: [[DEFAULT_PICTURE_VIDEO.textEncoder, "another_encoder.safetensors"]] },VAELoader: { vae_name: [[DEFAULT_PICTURE_VIDEO.videoVae, DEFAULT_PICTURE_VIDEO.audioVae, "another_vae.safetensors"]] },};- data = { [node]: { input: { required: inputs[node as keyof typeof inputs] } } };+ if (node === "SolarRendererLoraLoader" && !solarInstalled) data = {};+ else data = { [node]: { input: { required: inputs[node as keyof typeof inputs] } } };} else if (request.url === "/upload/image") {for await (const _ of request) {}data = { name: `uploaded-${++uploads}.png`, subfolder: "picture" };@@ -58,7 +59,7 @@ function inputs() {return form;}function render(prepared: string, settings?: unknown) {- return POST(new Request(url, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prepared, settings, filename: "untrusted.png", prompt: { untrusted: true } }) }));+ return POST(new Request(url, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prepared, settings: settings ?? { ...DEFAULT_PICTURE_VIDEO, renderer: "h3", steps: 6, seed: 42 }, filename: "untrusted.png", prompt: { untrusted: true } }) }));}function completed() {return { prompt: [0, "picture-1", prompts[0]], status: { status_str: "success", completed: true }, outputs: {@@ -133,7 +134,7 @@ test("render rejects unfinished, unrelated and failed histories instead of acceptest("render applies selected length, independent guide resolution, model components, LoRAs and steps", async () => {history = { "picture-1": completed() };- const response = await render("picture-1", { ...DEFAULT_PICTURE_VIDEO, seconds: 15, resolution: { width: 640, height: 960 }, motion: 512,+ const response = await render("picture-1", { ...DEFAULT_PICTURE_VIDEO, renderer: "h3", seconds: 15, resolution: { width: 640, height: 960 }, motion: 512,model: "another_h3.safetensors", steps: 12, seed: 99, textEncoder: "another_encoder.safetensors", videoVae: "another_vae.safetensors",loras: [{ name: "style.safetensors", strength: 0.6, enabled: true }, { name: "unused.safetensors", strength: 1, enabled: false }] });assert.equal(response.status, 202);@@ -156,6 +157,41 @@ test("invalid render settings are rejected before submitting or copying the prepassert.equal(prompts.length, count); assert.equal(uploads, copied);});+test("SolarRenderer submits the saved preview with the chosen pair from root or nested folders", async () => {+ history = { "picture-1": completed() };+ for (const solarRank of [384, 64]) {+ const suffix = solarRank === 64 ? "_rank64" : "";+ const seconds = solarRank === 384 ? 9 : 15;+ const response = await render("picture-1", { ...DEFAULT_PICTURE_VIDEO, solarRank, seconds,+ solarFourStep: `solarrenderer_4step_lora${suffix}.safetensors`, solarControl: `solarrenderer_control_lora${suffix}.safetensors` });+ assert.equal(response.status, 202);+ const graph = prompts.at(-1)!;+ assert.equal(graph["56"].class_type, "SolarRendererConditioning");+ assert.deepEqual(graph["56"].inputs.first_frame, ["20", 0]);+ assert.deepEqual(graph["64"].inputs.model, ["51", 0]);+ assert.equal(graph["51"].inputs.lora_name, `${solarRank === 64 ? "solar\\" : ""}solarrenderer_4step_lora${suffix}.safetensors`);+ assert.equal(graph["64"].inputs.lora_name, `${solarRank === 64 ? "solar\\" : ""}solarrenderer_control_lora${suffix}.safetensors`);+ assert.equal(graph["59"].inputs.sampler_name, "lcm");+ assert.equal(graph["60"].inputs.steps, 4);+ assert.equal(graph["56"].inputs.length, solarRank === 384 ? 209 : 345);+ assert.equal(graph["1"].inputs.frame_load_cap, graph["56"].inputs.length);+ assert.equal(graph["1"].inputs.skip_first_frames, 60);+ assert.ok(!graph["55"]);+ assert.ok(!Object.values(graph).some((node) => /TextGenerate|ReferenceMultiLatent/.test(node.class_type)));+ assert.ok(!Object.values(graph).some((node) => /Chain|Loop/.test(node.class_type)));+ }+});++test("missing SolarRenderer nodes and unlisted paths fail before copying the image or submitting", async () => {+ const count = prompts.length, copied = uploads;+ solarInstalled = false;+ const missing = await render("picture-1", DEFAULT_PICTURE_VIDEO);+ assert.equal(missing.status, 400); assert.match((await missing.json()).error, /Install the SolarRenderer/);+ solarInstalled = true;+ assert.equal((await render("picture-1", { ...DEFAULT_PICTURE_VIDEO, solarControl: "../outside.safetensors" })).status, 400);+ assert.equal(uploads, copied); assert.equal(prompts.length, count);+});+test("video media supports seeking and submission errors identify missing models", async () => {history = { "picture-2": { prompt: [0, "picture-2", prompts[1]], outputs: { "63": { gifs: [{ filename: "motion.mp4", type: "output" }] } } } };const response = await GET(new Request(`${url}?id=picture-2&media=result`, { headers: { Range: "bytes=0-3" } }));diff --git a/test/picture-settings.test.ts b/test/picture-settings.test.tsindex f76133b..599cc04 100644--- a/test/picture-settings.test.ts+++ b/test/picture-settings.test.ts@@ -1,10 +1,10 @@import assert from "node:assert/strict";import { readFileSync } from "node:fs";import test from "node:test";-import { applyPictureVideoSettings, DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureFrames } from "../lib/picture.ts";+import { applyPictureVideoSettings, DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureFrames, solarLoraFiles, pictureLoras } from "../lib/picture.ts";import type { ApiGraph } from "../lib/simple.ts";-const graph = (): ApiGraph => JSON.parse(readFileSync(new URL("../workflows/picture/from_picture_motion_flux3.api.json", import.meta.url), "utf8"));+const graph = (renderer = "motion"): ApiGraph => JSON.parse(readFileSync(new URL(`../workflows/picture/from_picture_${renderer}_flux3.api.json`, import.meta.url), "utf8"));test("single-clip length is aligned down without exceeding the selected duration or 15 seconds", () => {assert.equal(pictureFrames(6), 141);@@ -50,7 +50,7 @@ test("LoRAs are applied in order, disabled entries are skipped, and turning themtest("default settings keep the tested canvas, length and DMAD configuration", () => {const value = graph();- applyPictureVideoSettings(value, parsePictureVideoSettings(undefined), 768, 1376);+ applyPictureVideoSettings(value, parsePictureVideoSettings({ renderer: "h3", steps: 6 }), 768, 1376);assert.equal(value["56"].inputs.length, 141); assert.equal(value["1"].inputs.frame_load_cap, 141);assert.equal(value["56"].inputs.width, 480); assert.equal(value["56"].inputs.height, 864);assert.equal(value["33"].inputs.width, 480); assert.equal(value["33"].inputs.height, 864);@@ -71,3 +71,43 @@ test("picture rendering uses the same Comfy Kitchen attention backend as the othassert.deepEqual(value["60"].inputs.model, ["52", 0]);}});++test("SolarRenderer selects both ranks in root or nested LoRA folders without changing custom paths", () => {+ const root = solarLoraFiles(384);+ assert.deepEqual(solarLoraFiles(384, Object.values(root)), root);+ const nested = Object.values(solarLoraFiles(64)).map((name) => `solar\\${name}`);+ assert.deepEqual(Object.values(solarLoraFiles(64, nested)), nested);+ assert.equal(solarLoraFiles(384, [`other/${root.solarFourStep}`, root.solarFourStep]).solarFourStep, root.solarFourStep);+ const custom = parsePictureVideoSettings({ ...DEFAULT_PICTURE_VIDEO, solarFourStep: "my/four.safetensors", solarControl: "renamed/control.safetensors" });+ assert.deepEqual(pictureLoras(custom), ["my/four.safetensors", "renamed/control.safetensors"]);+});++test("SolarRenderer preserves the tested loader order, silent sampling, LCM and frame-aligned conditioning", () => {+ for (const solarRank of [384, 64] as const) {+ const value = graph("solar");+ const settings = parsePictureVideoSettings({ ...DEFAULT_PICTURE_VIDEO, solarRank, ...solarLoraFiles(solarRank), resolution: "768p", seed: 123 });+ applyPictureVideoSettings(value, settings, 1080, 1920);+ assert.equal(value["51"].class_type, "SolarRendererLoraLoader");+ assert.equal(value["64"].inputs.lora_name, settings.solarControl);+ assert.equal(value["51"].inputs.lora_name, settings.solarFourStep);+ assert.deepEqual(value["64"].inputs.model, ["51", 0]);+ assert.deepEqual(value["49"].inputs.model, ["64", 0]);+ assert.equal(value["52"].inputs.shift_audio, 3);+ assert.equal(value["65"].inputs.seed, 123);+ assert.equal(value["56"].class_type, "SolarRendererConditioning");+ assert.deepEqual(value["56"].inputs.first_frame, ["20", 0]);+ assert.deepEqual(value["56"].inputs.structure, ["33", 0]);+ assert.equal(value["56"].inputs.width, 768); assert.equal(value["56"].inputs.height, 1376);+ assert.equal(value["56"].inputs.length, 141);+ assert.equal(value["59"].inputs.sampler_name, "lcm");+ assert.equal(value["60"].class_type, "SolarRendererSigmas"); assert.equal(value["60"].inputs.steps, 4);+ assert.deepEqual(value["62"].inputs.samples, ["66", 0]);+ assert.deepEqual(value["63"].inputs.audio, ["36", 0]);+ assert.ok(!value["55"]);+ }+ assert.equal(parsePictureVideoSettings(undefined).renderer, "solar");+ assert.equal(parsePictureVideoSettings({ seconds: 15 }).renderer, "h3");+ assert.equal(parsePictureVideoSettings({ ...DEFAULT_PICTURE_VIDEO, seconds: 15 }).seconds, 15);+ assert.throws(() => parsePictureVideoSettings({ ...DEFAULT_PICTURE_VIDEO, seconds: 0.5 }), /at least 22 frames/);+ assert.throws(() => parsePictureVideoSettings({ ...DEFAULT_PICTURE_VIDEO, solarRank: 12 }), /rank/);+});diff --git a/workflows/build/openjutsu_picture_pipeline.py b/workflows/build/openjutsu_picture_pipeline.pyindex 4d9bdd8..ce19a91 100644--- a/workflows/build/openjutsu_picture_pipeline.py+++ b/workflows/build/openjutsu_picture_pipeline.py@@ -24,6 +24,27 @@ overall_soundscape:Reuse <Audio 1> unchanged. Add no new voices, music or sound effects."""+SOLAR_PROMPT = """subject_definitions:+<Video 1> is a false-color surface-normal motion guide at 24 fps. It supplies the moving subject's body geometry, poses, gestures, head direction and timing.+<Video 2> is the first frame of the target shot, held still. It supplies the character, outfit, background, materials, colors and lighting.++summary:+[keyframe completion + reference generation] Begin from <Video 2> and animate its character with the frame-by-frame poses and motion of <Video 1>. Keep the identity, clothing and environment of <Video 2> throughout.++retention_analysis:+<Video 1> (frame-by-frame subject pose, motion and framing): attribute_transfer - follow the subject's body position, size, gestures and head direction. Its false colors, source appearance, labels and graphics are not shown.+<Video 2> (first frame): fully_preserved - preserve its character, outfit, background, colors, materials and lighting as the character moves.++detailed_description:+[Shot 1] The shot begins from <Video 2>. The same character performs the changing poses, hand gestures and head movements from <Video 1>, through the final frame. Match the reference performance's timing, framing and camera motion. Keep the character's face, hair and complete outfit consistent, with coherent arms, hands and fingers. Preserve the setting and lighting from <Video 2>, including natural contact shadows. Do not hold the initial pose as the motion guide changes. Do not copy the normal-map colors, source outfit or source setting. Do not add text, comparison dividers, labels or slider graphics.++overall_soundscape:+N/A++non_diegetic_music:+N/A"""++def node(kind, title, **inputs):return {"class_type": kind, "inputs": inputs, "_meta": {"title": title}}@@ -100,6 +121,28 @@ def motion(prepared, width=480, height=864, frames=141):return graph+def solar(prepared, width=480, height=864, frames=141):+ graph = motion(prepared, width, height, frames)+ del graph["55"]+ graph["51"] = node("SolarRendererLoraLoader", "SolarRenderer four-step LoRA", model=["50", 0],+ lora_name="solarrenderer_4step_lora.safetensors", strength=1.0)+ graph["64"] = node("SolarRendererLoraLoader", "SolarRenderer control LoRA", model=["51", 0],+ lora_name="solarrenderer_control_lora.safetensors", strength=1.0)+ graph["49"]["inputs"]["model"] = ["64", 0]+ graph["52"]["inputs"]["shift_audio"] = 3.0+ graph["65"] = node("SolarRendererModel", "SolarRenderer silent sampling", model=["52", 0], seed=2577489884)+ graph["56"] = node("SolarRendererConditioning", "Prepared picture + frame-aligned normal motion", clip=["53", 0],+ vae=["54", 0], structure=["33", 0], first_frame=["20", 0], prompt=SOLAR_PROMPT,+ width=width, height=height, length=frames)+ graph["57"]["inputs"]["model"] = ["65", 0]+ graph["59"]["inputs"]["sampler_name"] = "lcm"+ graph["60"] = node("SolarRendererSigmas", "SolarRenderer four-step schedule", steps=4)+ graph["66"] = node("LTXVSeparateAVLatent", "Separate sampled video", av_latent=["61", 0])+ graph["62"]["inputs"]["samples"] = ["66", 0]+ graph["63"]["inputs"]["filename_prefix"] = "openjutsu_picture/solar/normal_motion"+ return graph++if __name__ == "__main__":parser = argparse.ArgumentParser(description=__doc__)parser.add_argument("--video", default="instagram1_normal_source_20261027.mp4", help="Filename in ComfyUI input/")@@ -108,7 +151,7 @@ if __name__ == "__main__":args = parser.parse_args()args.output_dir.mkdir(parents=True, exist_ok=True)first = prepare(args.video, args.picture)- for name, graph in [("prepare", first), ("motion", motion(first))]:+ for name, graph in [("prepare", first), ("motion", motion(first)), ("solar", solar(first))]:path = args.output_dir/f"from_picture_{name}_flux3.api.json"path.write_text(json.dumps(dict(sorted(graph.items(), key=lambda item: int(item[0]))), indent=2, ensure_ascii=False)+"\n", encoding="utf-8")print(path)diff --git a/workflows/picture/README.md b/workflows/picture/README.mdindex 2361534..206c41f 100644--- a/workflows/picture/README.md+++ b/workflows/picture/README.md@@ -8,6 +8,31 @@ The app uses them under **Object swap → From picture**. Choose the video and c| --- | --- || `from_picture_prepare_flux3.api.json` | FLUX 3 prepares and saves the first-frame picture. || `from_picture_motion_flux3.api.json` | FLUX preparation → Metric3D normals → H3 video. |+| `from_picture_solar_flux3.api.json` | FLUX preparation → Metric3D normals → SolarRenderer with its four-step and control LoRAs. |++## SolarRenderer++The app defaults to **SolarRenderer**, four steps and the rank-384 pair. Install the [official SolarRenderer custom nodes](https://github.com/Enderfga/SolarRenderer/tree/main/comfyui) and restart ComfyUI. These nodes run inside ComfyUI's existing Python environment.++Download both [solarrenderer_4step_lora.safetensors](https://huggingface.co/Enderfga/SolarRenderer/blob/main/solarrenderer_4step_lora.safetensors) and [solarrenderer_control_lora.safetensors](https://huggingface.co/Enderfga/SolarRenderer/blob/main/solarrenderer_control_lora.safetensors) from [Enderfga/SolarRenderer](https://huggingface.co/Enderfga/SolarRenderer). For the compressed pair, download the rank-64 four-step and control files from [mdevoz/SolarRendererRank64](https://huggingface.co/mdevoz/SolarRendererRank64).++Put the pair in `ComfyUI/models/loras` or any subfolder, such as `ComfyUI/models/loras/solar`. Reload the app, select **SolarRenderer LoRA rank**, then check the **four-step LoRA** and **control LoRA** file pickers. Both show the full installed relative paths; files can be selected manually, including renamed files. The selection persists across refreshes. The exported SolarRenderer workflow uses root filenames; select subfolder paths in its two loader nodes when importing directly into ComfyUI.++The four-step LoRA feeds the control LoRA, both at strength 1, through the official SolarRenderer loader. The workflow uses frame-aligned normal-map conditioning, held silent audio during sampling, the SolarRenderer four-step schedule and LCM. The output keeps the original source soundtrack. It needs no audio VAE. Both SolarRenderer and native H3 support up to 15 seconds in one clip, without a context loop. SolarRenderer needs at least 22 frames. Clips longer than six seconds can take considerably longer to render, especially at higher resolutions. Existing saved H3 settings keep using H3.++Both rank pairs completed a 141-frame 768×1376 test on an RTX 4060 Ti 16 GB. Rank 64 reduced adapter file size, with essentially unchanged sampling time and peak VRAM in that test.++Longer SolarRenderer clips need its internal silence buffer to extend beyond the released fixed template. If your node pack still reports "the held silence covers 170 frames", apply the included [extended-silence patch](../../patches/solarrenderer-long-clips.patch) to the official SolarRenderer pack and restart ComfyUI:++```sh+git -C /path/to/ComfyUI/custom_nodes/SolarRenderer apply /path/to/openjutsu/patches/solarrenderer-long-clips.patch+```++The patch preserves the silence template's beginning and end and repeats its steady middle, following the causal sampler's existing extension pattern. It does not enable causal sampling or context looping. Node-pack updates may require checking the fix again.++A rank-64 768×1376 single-clip nine-second selection completed with the fix: 209 frames / 8.708 seconds after frame-grid alignment, 34m 23s total and 15,898 MiB peak measured total VRAM. This confirms that selection on the tested installation; longer selections remain hardware-dependent.++Start with a **prepare** workflow. In **Source video**, choose the uploaded video and its `skip_first_frames`; in **Image 1 / character and background**, choose the original character picture. The first selected raw frame becomes image 2. There is no DWPose or skeleton input.@@ -21,14 +46,14 @@ The image-aware prompt upsampler sees both references and expands that request iPreparation defaults to a 768×1376 canvas and saves under `ComfyUI/output/openjutsu_picture/flux3/prepared_frame_*.png`. Compare the prepared picture against the raw first-frame preview before using it for video; a successful execution does not guarantee that the requested pose was followed.-The **motion** variants wire the decoded prepared picture directly into H3. No manual re-upload is needed. They default to **141 frames / 5.875 seconds at 24 fps**, 480×864, six DMAD steps, and the original soundtrack (or silence if absent). The app lets you select up to 15 seconds, with no context loop. It rounds down to the H3 17k+5 frame grid: 6 seconds becomes 141 frames / 5.875 seconds; 15 seconds becomes 345 frames / 14.375 seconds. The duration shown in the settings is the actual render length. Models, LoRAs (including individual switches and strengths), H3 text encoder, VAEs, steps and seed can be changed before rendering. These settings persist across refreshes. Change `frame_load_cap` and H3's `length` together to select another valid H3 clip length.+The **motion** and **solar** variants wire the decoded prepared picture directly into H3. No manual re-upload is needed. They default to **141 frames / 5.875 seconds at 24 fps**, 480×864, four SolarRenderer steps or six H3/DMAD steps, and the original soundtrack (or silence if absent). Both renderers let you select up to 15 seconds, with no context loop. It rounds down to the H3 17k+5 frame grid: 6 seconds becomes 141 frames / 5.875 seconds; 15 seconds becomes 345 frames / 14.375 seconds. The duration shown in the settings is the actual render length. Models, LoRAs (including individual switches and strengths), H3 text encoder, VAEs, steps and seed can be changed before rendering. These settings persist across refreshes. Change `frame_load_cap` and H3's `length` together to select another valid H3 clip length.By default normals match the H3 output canvas. The app also offers independent 512, 768 and 1376 px motion-reference sizes, rounded to multiples of 32; native H3 can cap large reference canvases. This is the motion/mask resolution control for this mode; no SAM edit mask is used. If changing that canvas, change both **Normals match the output canvas** and **Prepared picture + normal motion reference**. Normals are soft reference-video conditioning, not a trained normal ControlNet. The VRAM cleanup node runs after the image edit and before Metric3D to make room for its model.Required installed models are selected in the loader nodes. Defaults match this installation:- FLUX: `flux3_image_klein_sd_int8_convrot.safetensors`, `qwen3vl_4b_fp8_scaled.safetensors`, `flux2-vae.safetensors`, plus `qwen3.5_9b_flux3_prompt_upsampler_int8_convrot.safetensors`.-- Video: the H3 pruned INT8 ref2va model, DMAD LoRA, H3 text encoder and VAEs used by the existing workflows.+- Video: the H3 pruned INT8 ref2va model, H3 text encoder and video VAE. SolarRenderer needs its four-step/control pair; H3 needs the DMAD LoRA and audio VAE used by the existing workflows.- Custom nodes: VideoHelperSuite; KJNodes for VRAM cleanup; `comfyui_controlnet_aux` for Metric3D; the one-click swap pack for soundless-video handling. Metric3D-small's checkpoint is needed for the motion variants.Regenerate the files from the repository root:diff --git a/workflows/picture/from_picture_solar_flux3.api.json b/workflows/picture/from_picture_solar_flux3.api.jsonnew file mode 100644index 0000000..3465f84--- /dev/null+++ b/workflows/picture/from_picture_solar_flux3.api.json@@ -0,0 +1,689 @@+{+ "1": {+ "class_type": "VHS_LoadVideo",+ "inputs": {+ "video": "instagram1_normal_source_20261027.mp4",+ "force_rate": 24.0,+ "custom_width": 0,+ "custom_height": 0,+ "frame_load_cap": 141,+ "skip_first_frames": 0,+ "select_every_nth": 1,+ "format": "None"+ },+ "_meta": {+ "title": "1 · Source video — selected motion clip"+ }+ },+ "2": {+ "class_type": "ImageFromBatch",+ "inputs": {+ "image": [+ "1",+ 0+ ],+ "batch_index": 0,+ "length": 1+ },+ "_meta": {+ "title": "Raw first frame"+ }+ },+ "3": {+ "class_type": "ImageScale",+ "inputs": {+ "image": [+ "2",+ 0+ ],+ "upscale_method": "lanczos",+ "width": 768,+ "height": 1376,+ "crop": "center"+ },+ "_meta": {+ "title": "Image 2 · First frame / pose and body cut"+ }+ },+ "4": {+ "class_type": "LoadImage",+ "inputs": {+ "image": "Stylish%20Blonde%20in%20a%20Warm%20Cafe.png"+ },+ "_meta": {+ "title": "2 · Image 1 / character and background"+ }+ },+ "5": {+ "class_type": "ImageScale",+ "inputs": {+ "image": [+ "4",+ 0+ ],+ "upscale_method": "lanczos",+ "width": 768,+ "height": 1376,+ "crop": "center"+ },+ "_meta": {+ "title": "Image 1 · Character / editor canvas"+ }+ },+ "6": {+ "class_type": "PrimitiveStringMultiline",+ "inputs": {+ "value": "change <image 1> character pose follow the <image 2> pose. keep the final image as same body cut as <image 2> . do not change the background of <image 1> and do not add anything extra in <image 1>."+ },+ "_meta": {+ "title": "Edit request · image 1 character / image 2 raw frame"+ }+ },+ "7": {+ "class_type": "ImageBatch",+ "inputs": {+ "image1": [+ "5",+ 0+ ],+ "image2": [+ "3",+ 0+ ]+ },+ "_meta": {+ "title": "Character first, raw frame second"+ }+ },+ "8": {+ "class_type": "ImageScaleToMaxDimension",+ "inputs": {+ "image": [+ "7",+ 0+ ],+ "largest_size": 768,+ "upscale_method": "area"+ },+ "_meta": {+ "title": "Images for the prompt upsampler"+ }+ },+ "9": {+ "class_type": "CLIPLoader",+ "inputs": {+ "clip_name": "qwen3.5_9b_flux3_prompt_upsampler_int8_convrot.safetensors",+ "type": "flux3",+ "device": "default"+ },+ "_meta": {+ "title": "FLUX 3 image-edit prompt upsampler"+ }+ },+ "10": {+ "class_type": "UNETLoader",+ "inputs": {+ "unet_name": "flux3_image_klein_sd_int8_convrot.safetensors",+ "weight_dtype": "default"+ },+ "_meta": {+ "title": "FLUX 3 Image Klein SD"+ }+ },+ "11": {+ "class_type": "CLIPLoader",+ "inputs": {+ "clip_name": "qwen3vl_4b_fp8_scaled.safetensors",+ "type": "flux3",+ "device": "default"+ },+ "_meta": {+ "title": "FLUX 3 text encoder"+ }+ },+ "12": {+ "class_type": "VAELoader",+ "inputs": {+ "vae_name": "flux2-vae.safetensors"+ },+ "_meta": {+ "title": "FLUX VAE"+ }+ },+ "13": {+ "class_type": "CLIPTextEncode",+ "inputs": {+ "clip": [+ "11",+ 0+ ],+ "text": [+ "23",+ 0+ ]+ },+ "_meta": {+ "title": "Expanded FLUX editing instruction"+ }+ },+ "14": {+ "class_type": "ConditioningZeroOut",+ "inputs": {+ "conditioning": [+ "13",+ 0+ ]+ },+ "_meta": {+ "title": "Empty negative"+ }+ },+ "15": {+ "class_type": "ReferenceMultiLatent",+ "inputs": {+ "positive": [+ "13",+ 0+ ],+ "negative": [+ "14",+ 0+ ],+ "vae": [+ "12",+ 0+ ],+ "resolution": "keep resolution",+ "images.image0": [+ "5",+ 0+ ],+ "images.image1": [+ "3",+ 0+ ]+ },+ "_meta": {+ "title": "Both raw image references reach FLUX"+ }+ },+ "19": {+ "class_type": "KSampler",+ "inputs": {+ "model": [+ "10",+ 0+ ],+ "seed": 0,+ "steps": 16,+ "cfg": 1.0,+ "sampler_name": "euler",+ "scheduler": "simple",+ "denoise": 1.0,+ "positive": [+ "15",+ 0+ ],+ "negative": [+ "15",+ 1+ ],+ "latent_image": [+ "15",+ 2+ ]+ },+ "_meta": {+ "title": "Prepare pose · FLUX 3"+ }+ },+ "20": {+ "class_type": "VAEDecode",+ "inputs": {+ "samples": [+ "19",+ 0+ ],+ "vae": [+ "12",+ 0+ ]+ },+ "_meta": {+ "title": "Prepared first frame"+ }+ },+ "21": {+ "class_type": "SaveImage",+ "inputs": {+ "images": [+ "20",+ 0+ ],+ "filename_prefix": "openjutsu_picture/flux3/prepared_frame"+ },+ "_meta": {+ "title": "3 · Save prepared first frame"+ }+ },+ "22": {+ "class_type": "PreviewImage",+ "inputs": {+ "images": [+ "3",+ 0+ ]+ },+ "_meta": {+ "title": "Raw first frame — compare composition"+ }+ },+ "23": {+ "class_type": "TextGenerate",+ "inputs": {+ "clip": [+ "9",+ 0+ ],+ "image": [+ "8",+ 0+ ],+ "prompt": [+ "6",+ 0+ ],+ "system_prompt": [+ "25",+ 0+ ],+ "max_length": 700,+ "sampling_mode": "off",+ "thinking": false,+ "use_default_template": true,+ "mtp": "off"+ },+ "_meta": {+ "title": "Describe the raw frame's pose for the edit"+ }+ },+ "24": {+ "class_type": "PreviewAny",+ "inputs": {+ "source": [+ "23",+ 0+ ]+ },+ "_meta": {+ "title": "Expanded editing instruction"+ }+ },+ "25": {+ "class_type": "PrimitiveStringMultiline",+ "inputs": {+ "value": "You are an image-editing prompt upsampler. The attached images are ordered: image 1 is the character image and image 2 is the raw first frame from a video. Expand the user's request into one precise editing instruction. Keep every requested constraint. Use gender-neutral wording such as 'the character'; do not add a fixed gender.\n\nImage 1 supplies the identity, hairstyle, complete outfit, background and lighting. Image 2 supplies only the starting body pose, head orientation, arm positions, hand positions, leg positions, camera perspective, relative character size, position within the frame and body cut. Describe those pose and framing changes concretely from image 2. Replace the original pose of image 1 instead of retaining it. Keep the character and the background of image 1; do not copy the identity, hairstyle, clothes, props, setting or colors of image 2, and do not add extra objects. Describe the intended still image, not later video movements. Output only one English editing instruction, without Markdown or explanation."+ },+ "_meta": {+ "title": "Image-edit prompt expansion rules"+ }+ },+ "30": {+ "class_type": "VRAM_Debug",+ "inputs": {+ "empty_cache": true,+ "gc_collect": true,+ "unload_all_models": true,+ "any_input": [+ "20",+ 0+ ],+ "image_pass": [+ "1",+ 0+ ]+ },+ "_meta": {+ "title": "Release editor VRAM before Metric3D"+ }+ },+ "31": {+ "class_type": "ImageScaleToMaxDimension",+ "inputs": {+ "image": [+ "30",+ 1+ ],+ "largest_size": 768,+ "upscale_method": "area"+ },+ "_meta": {+ "title": "Metric3D input video"+ }+ },+ "32": {+ "class_type": "Metric3D-NormalMapPreprocessor",+ "inputs": {+ "image": [+ "31",+ 0+ ],+ "backbone": "vit-small",+ "fx": 1000,+ "fy": 1000,+ "resolution": 512+ },+ "_meta": {+ "title": "4 · Source motion normals"+ }+ },+ "33": {+ "class_type": "ImageScale",+ "inputs": {+ "image": [+ "32",+ 0+ ],+ "width": 480,+ "height": 864,+ "upscale_method": "lanczos",+ "crop": "disabled"+ },+ "_meta": {+ "title": "Normals match the output canvas"+ }+ },+ "34": {+ "class_type": "ImageFromBatch",+ "inputs": {+ "image": [+ "33",+ 0+ ],+ "batch_index": 0,+ "length": 1+ },+ "_meta": {+ "title": "First normal-map frame"+ }+ },+ "35": {+ "class_type": "PreviewImage",+ "inputs": {+ "images": [+ "34",+ 0+ ]+ },+ "_meta": {+ "title": "Normal-map preview"+ }+ },+ "36": {+ "class_type": "H3SoundtrackOrSilence",+ "inputs": {+ "audio": [+ "1",+ 2+ ],+ "video_info": [+ "1",+ 3+ ]+ },+ "_meta": {+ "title": "Original soundtrack / silence when absent"+ }+ },+ "49": {+ "class_type": "ModelAttentionBackend",+ "inputs": {+ "model": [+ "64",+ 0+ ],+ "attention": "comfy kitchen attention"+ },+ "_meta": {+ "title": "Comfy Kitchen Attention"+ }+ },+ "50": {+ "class_type": "UNETLoader",+ "inputs": {+ "unet_name": "minimax_h3_ref2va_pruned_int8_convrot.safetensors",+ "weight_dtype": "default"+ },+ "_meta": {+ "title": "H3 Diffusion Model"+ }+ },+ "51": {+ "class_type": "SolarRendererLoraLoader",+ "inputs": {+ "model": [+ "50",+ 0+ ],+ "lora_name": "solarrenderer_4step_lora.safetensors",+ "strength": 1.0+ },+ "_meta": {+ "title": "SolarRenderer four-step LoRA"+ }+ },+ "52": {+ "class_type": "MiniMaxH3SigmaShift",+ "inputs": {+ "model": [+ "49",+ 0+ ],+ "shift_video": 12.0,+ "shift_audio": 3.0+ },+ "_meta": {+ "title": "H3 Sigma Shift"+ }+ },+ "53": {+ "class_type": "CLIPLoader",+ "inputs": {+ "clip_name": "qwen3vl_32b_minimax_h3_int4_convrot.safetensors",+ "type": "minimax",+ "device": "default"+ },+ "_meta": {+ "title": "H3 Text Encoder"+ }+ },+ "54": {+ "class_type": "VAELoader",+ "inputs": {+ "vae_name": "minimax_h3_video_vae_int8_convrot.safetensors"+ },+ "_meta": {+ "title": "H3 Video VAE"+ }+ },+ "56": {+ "class_type": "SolarRendererConditioning",+ "inputs": {+ "clip": [+ "53",+ 0+ ],+ "vae": [+ "54",+ 0+ ],+ "structure": [+ "33",+ 0+ ],+ "first_frame": [+ "20",+ 0+ ],+ "prompt": "subject_definitions:\n<Video 1> is a false-color surface-normal motion guide at 24 fps. It supplies the moving subject's body geometry, poses, gestures, head direction and timing.\n<Video 2> is the first frame of the target shot, held still. It supplies the character, outfit, background, materials, colors and lighting.\n\nsummary:\n[keyframe completion + reference generation] Begin from <Video 2> and animate its character with the frame-by-frame poses and motion of <Video 1>. Keep the identity, clothing and environment of <Video 2> throughout.\n\nretention_analysis:\n<Video 1> (frame-by-frame subject pose, motion and framing): attribute_transfer - follow the subject's body position, size, gestures and head direction. Its false colors, source appearance, labels and graphics are not shown.\n<Video 2> (first frame): fully_preserved - preserve its character, outfit, background, colors, materials and lighting as the character moves.\n\ndetailed_description:\n[Shot 1] The shot begins from <Video 2>. The same character performs the changing poses, hand gestures and head movements from <Video 1>, through the final frame. Match the reference performance's timing, framing and camera motion. Keep the character's face, hair and complete outfit consistent, with coherent arms, hands and fingers. Preserve the setting and lighting from <Video 2>, including natural contact shadows. Do not hold the initial pose as the motion guide changes. Do not copy the normal-map colors, source outfit or source setting. Do not add text, comparison dividers, labels or slider graphics.\n\noverall_soundscape:\nN/A\n\nnon_diegetic_music:\nN/A",+ "width": 480,+ "height": 864,+ "length": 141+ },+ "_meta": {+ "title": "Prepared picture + frame-aligned normal motion"+ }+ },+ "57": {+ "class_type": "BasicGuider",+ "inputs": {+ "model": [+ "65",+ 0+ ],+ "conditioning": [+ "56",+ 0+ ]+ },+ "_meta": {+ "title": "H3 Guider"+ }+ },+ "58": {+ "class_type": "RandomNoise",+ "inputs": {+ "noise_seed": 2577489884+ },+ "_meta": {+ "title": "Video seed"+ }+ },+ "59": {+ "class_type": "KSamplerSelect",+ "inputs": {+ "sampler_name": "lcm"+ },+ "_meta": {+ "title": "Video sampler"+ }+ },+ "60": {+ "class_type": "SolarRendererSigmas",+ "inputs": {+ "steps": 4+ },+ "_meta": {+ "title": "SolarRenderer four-step schedule"+ }+ },+ "61": {+ "class_type": "SamplerCustomAdvanced",+ "inputs": {+ "noise": [+ "58",+ 0+ ],+ "guider": [+ "57",+ 0+ ],+ "sampler": [+ "59",+ 0+ ],+ "sigmas": [+ "60",+ 0+ ],+ "latent_image": [+ "56",+ 1+ ]+ },+ "_meta": {+ "title": "Generate motion"+ }+ },+ "62": {+ "class_type": "VAEDecode",+ "inputs": {+ "samples": [+ "66",+ 0+ ],+ "vae": [+ "54",+ 0+ ]+ },+ "_meta": {+ "title": "Decode video"+ }+ },+ "63": {+ "class_type": "VHS_VideoCombine",+ "inputs": {+ "images": [+ "62",+ 0+ ],+ "audio": [+ "36",+ 0+ ],+ "frame_rate": 24.0,+ "loop_count": 0,+ "filename_prefix": "openjutsu_picture/solar/normal_motion",+ "format": "video/h264-mp4",+ "pix_fmt": "yuv420p",+ "crf": 16,+ "save_metadata": false,+ "trim_to_audio": false,+ "pingpong": false,+ "save_output": true+ },+ "_meta": {+ "title": "6 · Result with original soundtrack"+ }+ },+ "64": {+ "class_type": "SolarRendererLoraLoader",+ "inputs": {+ "model": [+ "51",+ 0+ ],+ "lora_name": "solarrenderer_control_lora.safetensors",+ "strength": 1.0+ },+ "_meta": {+ "title": "SolarRenderer control LoRA"+ }+ },+ "65": {+ "class_type": "SolarRendererModel",+ "inputs": {+ "model": [+ "52",+ 0+ ],+ "seed": 2577489884+ },+ "_meta": {+ "title": "SolarRenderer silent sampling"+ }+ },+ "66": {+ "class_type": "LTXVSeparateAVLatent",+ "inputs": {+ "av_latent": [+ "61",+ 0+ ]+ },+ "_meta": {+ "title": "Separate sampled video"+ }+ }+}