Skip to content

One-click character swap for videos: an open-source, local Higgsfield Genjutsu alternative built on ComfyUI and MiniMax H3, with FLUX.3 Klein relighting.

Add Relight to From picture

D3voz5ee5d21d09da
diff --git a/app/api/picture/route.ts b/app/api/picture/route.ts
index 4d0c15b..9792edd 100644
--- a/app/api/picture/route.ts
+++ b/app/api/picture/route.ts
@@ -5,7 +5,7 @@ import { join } from "node:path";
import { apiAccess, HttpError, IMAGE_SIGNATURE, readFormData, StoredFile, validateJobId, VIDEO_SIGNATURE } from "../../../lib/http.ts";
import { canvasSize, isLink, MAX_SEED, type ApiGraph } from "../../../lib/simple.ts";
import { listModels } from "../../../lib/comfy.ts";
-import { applyPictureVideoSettings, DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureLoras, solarLoraFiles, type PictureVideoSettings } from "../../../lib/picture.ts";
+import { applyPictureVideoSettings, applyRelight, DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureLoras, solarLoraFiles, type PictureVideoSettings } from "../../../lib/picture.ts";
export const runtime = "nodejs";
const PREPARED = "OpenJutsu · prepared picture";
@@ -135,6 +135,8 @@ export async function POST(request: Request) {
graph["4"].inputs.image = await uploadFile(await openAsBlob(picture.path), pictureExtension.toLowerCase());
graph["19"].inputs.seed = seed;
graph["21"]._meta = { title: PREPARED };
+ const light = form.get("light");
+ if (typeof light === "string" && light.trim()) applyRelight(graph, light.trim());
}
const result = await (await comfy("/prompt", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prompt: graph }) })).json();
return Response.json({ id: result.prompt_id }, { status: 202 });
diff --git a/components/from-picture.tsx b/components/from-picture.tsx
index cd56854..fa22a55 100644
--- a/components/from-picture.tsx
+++ b/components/from-picture.tsx
@@ -3,7 +3,7 @@
import { useEffect, useState } from "react";
import { canvasSize, CUSTOM_SIZE, MAX_SEED, MOTION_SIZES, RESOLUTIONS } from "../lib/simple.ts";
import { MAX_LORAS, STEPS, STRENGTH } from "../lib/model.ts";
-import { DEFAULT_PICTURE_VIDEO, parsePictureVideoSettings, pictureFrames, solarLoraFiles, type PictureVideoSettings } from "../lib/picture.ts";
+import { DEFAULT_PICTURE_VIDEO, LIGHT_LOOKS, parsePictureVideoSettings, pictureFrames, RELIGHT_PROMPT, solarLoraFiles, type PictureVideoSettings } from "../lib/picture.ts";
const PROMPT = "change <image 1> character pose follow the <image 2> pose. keep the final image as same body cut as <image 2> . do not change the background of <image 1> and do not add anything extra in <image 1>.";
const STORAGE = "openjutsu.picture-preview";
@@ -19,6 +19,9 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
const [start, setStart] = useState(initialStart);
const [seed, setSeed] = useState(0);
const [prompt, setPrompt] = useState(PROMPT);
+ const [look, setLook] = useState(-1);
+ const [ownLight, setOwnLight] = useState("Light the new photo with ");
+ const [relightPrompt, setRelightPrompt] = useState(RELIGHT_PROMPT);
const [job, setJob] = useState<Job | null>(null);
const [prepared, setPrepared] = useState("");
const [preview, setPreview] = useState<{ url: string; frame: string } | null>(null);
@@ -31,6 +34,8 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
const [submitting, setSubmitting] = useState(false);
const [error, setError] = useState("");
const busy = submitting || !!job;
+ const relight = look >= 0;
+ const light = look === LIGHT_LOOKS.length ? ownLight : LIGHT_LOOKS[look]?.light ?? "";
useEffect(() => {
try {
@@ -113,7 +118,8 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
try {
const form = new FormData();
form.set("video", video); form.set("picture", picture); form.set("start", String(start)); form.set("seed", String(nextSeed));
- form.set("width", String(size[0])); form.set("height", String(size[1])); form.set("prompt", prompt);
+ form.set("width", String(size[0])); form.set("height", String(size[1])); form.set("prompt", relight ? relightPrompt : prompt);
+ if (relight) form.set("light", light);
const response = await fetch("/api/picture", { method: "POST", body: form });
const data = await response.json();
if (!response.ok) throw new Error(data.error);
@@ -158,9 +164,19 @@ export function FromPicture({ initialVideo, initialPicture, start: initialStart
<p className="hint">Image 1 supplies the character and background. Image 2 supplies the pose and body cut.</p>
<p className="hint">Missing the FLUX model files? <a href="https://huggingface.co/black-forest-labs/flux-3-image-klein-comfy" target="_blank" rel="noopener noreferrer">Download FLUX 3 Image Klein files</a>.</p>
<p className="hint"><strong>Prompt upsampler required.</strong> This workflow needs the <a href="https://huggingface.co/black-forest-labs/flux-3-image-klein-comfy/blob/main/text_encoders/qwen3.5_9b_flux3_prompt_upsampler_int8_convrot.safetensors" target="_blank" rel="noopener noreferrer">Qwen3.5 9B FLUX prompt upsampler</a> to expand the edit prompt before image generation. Download it into <code>ComfyUI/models/text_encoders</code>.</p>
- <label>FLUX edit prompt<textarea className="report" rows={6} value={prompt} disabled={busy} onChange={(event) => { invalidate(); setPrompt(event.target.value); }} /></label>
+ <label>Relight<select className="file-name" value={look} disabled={busy} onChange={(event) => { invalidate(); setLook(Number(event.target.value)); }}>
+ <option value={-1}>Off · keep the picture's light</option>
+ {LIGHT_LOOKS.map((item, index) => <option key={item.name} value={index}>{item.name}</option>)}
+ <option value={LIGHT_LOOKS.length}>My own light</option>
+ </select></label>
+ {LIGHT_LOOKS[look] && <figure style={{ margin: 0, display: "grid", gap: 6 }}><img className="picture-reference" src={LIGHT_LOOKS[look].image} alt={`${LIGHT_LOOKS[look].name} example`} style={{ width: 200 }} /><small>Example on one portrait. Your picture and seed will differ.</small></figure>}
+ {relight && <p className="hint">Poses and relights the character in one FLUX pass at 3 MP, about 2 minutes. FLUX sees only image 1; image 2 reaches it as a written pose, so nothing of the video's person leaks in. The light usually changes the background too. Results vary a lot by seed; try two or three.</p>}
+ {look === LIGHT_LOOKS.length && <label>Your light<textarea className="report" rows={3} value={ownLight} disabled={busy} onChange={(event) => { invalidate(); setOwnLight(event.target.value); }} /></label>}
+ {relight
+ ? <label>FLUX relight prompt · {"{look}"} is the light<textarea className="report" rows={6} value={relightPrompt} disabled={busy} onChange={(event) => { invalidate(); setRelightPrompt(event.target.value); }} /></label>
+ : <label>FLUX edit prompt<textarea className="report" rows={6} value={prompt} disabled={busy} onChange={(event) => { invalidate(); setPrompt(event.target.value); }} /></label>}
<label>Seed<input className="target-input" type="number" min={0} max={MAX_SEED} step={1} value={seed} disabled={busy} onChange={(event) => { invalidate(); setSeed(Number(event.target.value)); }} /></label>
- <button className="primary" disabled={busy || !video || !picture || !size} onClick={() => prepare(seed)}>Generate preview image</button>
+ <button className="primary" disabled={busy || !video || !picture || !size || (relight && !light.trim())} onClick={() => prepare(seed)}>Generate preview image</button>
<p className="hint">This step only generates an image. After reviewing it, choose the video settings and press Render video.</p>
</section>
</div>}
diff --git a/lib/picture.ts b/lib/picture.ts
index 681b0f4..e3f4080 100644
--- a/lib/picture.ts
+++ b/lib/picture.ts
@@ -93,3 +93,38 @@ export function applyPictureVideoSettings(graph: ApiGraph, settings: PictureVide
});
graph["49"].inputs.model = model;
}
+
+export const LIGHT_LOOKS = [
+ { name: "Cold beam", image: "/relight/cold_beam.webp", light: "Light the new photo as a dark horror scene: a single narrow cold white spotlight shines straight down from directly above, with a visible beam of light cutting through dusty air above the head; it lights the top of the hair, forehead, cheekbones, nose and shoulders, leaves deep shadows in the eye sockets and under the nose and chin, and the background falls to black. Cold, desaturated blue-grey colour grade." },
+ { name: "Golden halo", image: "/relight/golden_halo.webp", light: "Light the new photo with a warm golden glow: a strong amber backlight behind the head turns the background into glowing orange-gold haze and puts a bright golden rim on the hair and shoulders, while soft warm gold light falls on the face from the front. Rich honey and orange tones, glowing highlights, warm shadows, like sunset in a cathedral." },
+ { name: "Red and blue gels", image: "/relight/red_blue_gels.webp", light: "Light the new photo with two coloured gel lights in a dark room: saturated red light from the left and deep electric blue light from the right, each colour wrapping around its side of the face, hair and body, purple where they meet, and a dark red-blue background. High-contrast cinematic colour." },
+ { name: "Magenta club · mostly the background", image: "/relight/magenta_club.webp", light: "Light the new photo as a night club scene: the face, hair, skin and clothes are washed in hot magenta and purple stage light from above and the left, with pink highlights and deep violet shadows on the person, pink haze in the air and a blurry purple background with sparkling lights." },
+ { name: "Teal lasers · mostly the background", image: "/relight/teal_lasers.webp", light: "Light the new photo as a dark club scene in teal and blue: the face, hair, skin and clothes are lit by cold cyan light from the side, with blue-tinted skin, cyan highlights and deep navy shadows on the person; thin bright cyan laser beams cut diagonally through the haze behind, and the background is dark blue-black." },
+ { name: "Toxic green", image: "/relight/toxic_green.webp", light: "Light the new photo with eerie toxic green light: a vivid green light from below and the side lights the face, hair and clothes, green laser beams shine through dark haze behind, deep green-black shadows, the whole scene tinted acid green." },
+ { name: "Lime and purple", image: "/relight/lime_purple.webp", light: "Light the new photo with two bold gel colours: acid yellow-lime light from the front left floods the face and hair, deep purple light fills the shadows on the right and the background, high saturation and strong contrast, like a music video." },
+ { name: "Cyan orb", image: "/relight/cyan_orb.webp", light: "Light the new photo as a dark, moody scene lit only by a small glowing cold-blue light orb floating close to the face on the right side of the frame; the person is in darkness except where the orb's cool cyan-white light falls on the near side of the face, falling off quickly into blue-black shadow, with blue highlights in the eyes. No other light in the scene." },
+ { name: "Red room", image: "/relight/red_room.webp", light: "Light the new photo with deep crimson red light filling the whole scene, dark red shadows and a red-black background, and one small bright white light at the left edge of the frame that puts a cold white highlight on the near cheek and in the eyes." },
+];
+
+export const RELIGHT_PROMPT = "Create a photograph of the person from <image 1> in exactly the pose, head angle, hand and arm positions, camera angle, framing and body cut of the person in <image 2>; describe that pose fully. The person from <image 1> also makes exactly the facial expression of the person in <image 2>, with the same mouth shape, eyebrows and gaze; describe that expression instead of their usual one. Take only the pose, expression and framing from <image 2>, never its person, face shape, hair or clothing. Keep the face, identity, eye colour, hair and clothing of <image 1>. {look}";
+
+const KLEIN_UPSAMPLER = "You are the prompt upsampler of an image-editing model. You see the reference image(s), attached in order (the first image, the second image, ...), and the user's short request. Expand it into ONE detailed, self-contained instruction: the caption the editing model receives together with the same images. Output only the instruction as one continuous English prose block, with no greetings, explanations, Markdown, JSON, coordinates, ids or angle-bracket tokens. Read a vague or conversational request as the most plausible edit of the supplied images.\n\nPick the voice by task shape. When one reference stays and gets bounded changes, write an imperative editing instruction on it (\"Remove the ...\", \"Change the ... to ...\", \"Zoom in on ...\"); a whole-image transformation opens with its scope (\"Convert the entire image into ...\", \"Apply a ... filter over the entire scene\", \"Horizontally flip the entire image so that ...\", \"Shift the camera viewpoint to ...\", then where the major elements should now be). When the references are drawn into a newly composed frame (combining subjects, a new scene, a garment, object or style donor), write a generation request (\"Create a photograph of ...\", \"Create an anime-style illustration of ...\", \"Generate a new image featuring ...\") that says what to take from each image, by ordinal, and what is newly generated.\n\nKeep every explicit edit, constraint, count, position, style cue and piece of wording, and decide what is left open as one concrete plan. Ground everything in what is actually visible: identify each referent unambiguously by its real appearance, never mention content that is not there, and use every image the user attached. State every end state concretely (which color, how large, which pose, which material) and give new content enough appearance to be generated. Cover the knock-on effects of the change (shadows, reflections, occlusion, perspective, scale, lighting) and say what fills a removed region. A zoom-in names what stays in frame and which neighbors are cropped out; an outpainting continues the scene in the same medium. Describe the change, not the whole scene, and close an editing instruction with one compact clause naming what stays unchanged; transferred people keep their identity, face and accessories unless asked otherwise. Transcribe on-image text verbatim in its original language inside double quotes, with line breaks as \\n, and match the original font, size and color when replacing it. Name recognizable public figures, brands and landmarks you can see. Directions are frame-relative; use no evaluative or quality words. Typical length is 50 to 150 words.\n\nOutput only the instruction.";
+
+// Pose and light in one FLUX pass, because stacked edits degrade the picture. FLUX sees only the character picture at 3 MP;
+// the raw frame reaches it through the upsampler's written pose and sets the canvas shape.
+export function applyRelight(graph: ApiGraph, light: string) {
+ const request = String(graph["6"].inputs.value);
+ graph["6"].inputs.value = request.includes("{look}") ? request.replaceAll("{look}", light) : `${request} ${light}`;
+ delete graph["5"];
+ graph["7"] = { class_type: "BatchImagesNode", inputs: { "images.image0": ["4", 0], "images.image1": ["3", 0] }, _meta: { title: "Character picture + raw frame for the prompt upsampler" } };
+ graph["25"].inputs.value = KLEIN_UPSAMPLER;
+ Object.assign(graph["23"].inputs, { max_length: 3584, sampling_mode: "on", "sampling_mode.temperature": 1, "sampling_mode.top_k": 1, "sampling_mode.top_p": 0.95,
+ "sampling_mode.min_p": 0, "sampling_mode.repetition_penalty": 1.05, "sampling_mode.seed": 0 });
+ const { positive, negative, vae } = graph["15"].inputs;
+ graph["15"] = { class_type: "ReferenceMultiLatent", inputs: { positive, negative, vae, resolution: "resize", "resolution.megapixels": 3, "images.image0": ["4", 0] }, _meta: { title: "Character picture only, 3 MP" } };
+ graph["relight_canvas"] = { class_type: "ImageScaleToTotalPixels", inputs: { image: ["3", 0], upscale_method: "lanczos", megapixels: 3, resolution_steps: 16 }, _meta: { title: "Frame shape at 3 MP" } };
+ graph["relight_size"] = { class_type: "GetImageSize", inputs: { image: ["relight_canvas", 0] } };
+ graph["relight_latent"] = { class_type: "EmptyFlux2LatentImage", inputs: { width: ["relight_size", 0], height: ["relight_size", 1], batch_size: 1 }, _meta: { title: "Relit canvas" } };
+ graph["19"].inputs.latent_image = ["relight_latent", 0];
+ graph["21"].inputs.filename_prefix = "openjutsu_picture/flux3/relit_frame";
+}
diff --git a/public/relight/cold_beam.webp b/public/relight/cold_beam.webp
new file mode 100644
index 0000000..8a3bb3d
Binary files /dev/null and b/public/relight/cold_beam.webp differ
diff --git a/public/relight/cyan_orb.webp b/public/relight/cyan_orb.webp
new file mode 100644
index 0000000..8bfa5bb
Binary files /dev/null and b/public/relight/cyan_orb.webp differ
diff --git a/public/relight/golden_halo.webp b/public/relight/golden_halo.webp
new file mode 100644
index 0000000..30386ea
Binary files /dev/null and b/public/relight/golden_halo.webp differ
diff --git a/public/relight/lime_purple.webp b/public/relight/lime_purple.webp
new file mode 100644
index 0000000..5476661
Binary files /dev/null and b/public/relight/lime_purple.webp differ
diff --git a/public/relight/magenta_club.webp b/public/relight/magenta_club.webp
new file mode 100644
index 0000000..ac93692
Binary files /dev/null and b/public/relight/magenta_club.webp differ
diff --git a/public/relight/red_blue_gels.webp b/public/relight/red_blue_gels.webp
new file mode 100644
index 0000000..6b7db7f
Binary files /dev/null and b/public/relight/red_blue_gels.webp differ
diff --git a/public/relight/red_room.webp b/public/relight/red_room.webp
new file mode 100644
index 0000000..8458618
Binary files /dev/null and b/public/relight/red_room.webp differ
diff --git a/public/relight/teal_lasers.webp b/public/relight/teal_lasers.webp
new file mode 100644
index 0000000..4056068
Binary files /dev/null and b/public/relight/teal_lasers.webp differ
diff --git a/public/relight/toxic_green.webp b/public/relight/toxic_green.webp
new file mode 100644
index 0000000..cdf9831
Binary files /dev/null and b/public/relight/toxic_green.webp differ
diff --git a/test/picture-pipeline.test.ts b/test/picture-pipeline.test.ts
index e7458dc..aed40b8 100644
--- a/test/picture-pipeline.test.ts
+++ b/test/picture-pipeline.test.ts
@@ -2,6 +2,7 @@ import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import test from "node:test";
import { isLink, type ApiGraph } from "../lib/simple.ts";
+import { applyRelight, LIGHT_LOOKS, RELIGHT_PROMPT } from "../lib/picture.ts";
const load = (stage: string): ApiGraph => JSON.parse(readFileSync(new URL(`../workflows/picture/from_picture_${stage}_flux3.api.json`, import.meta.url), "utf8"));
const prompt = "change <image 1> character pose follow the <image 2> pose. keep the final image as same body cut as <image 2> . do not change the background of <image 1> and do not add anything extra in <image 1>.";
@@ -41,9 +42,26 @@ test("the motion workflow consumes the prepared picture directly and resizes nor
assert.ok(!Object.values(graph).some((node) => /ControlNet|DWPreprocessor|ChainLoop/.test(node.class_type)));
});
+test("relight gives FLUX only the character picture at 3 MP on the frame's shape and writes the light into the request", () => {
+ const graph = load("prepare");
+ graph["6"].inputs.value = RELIGHT_PROMPT;
+ applyRelight(graph, LIGHT_LOOKS[1].light);
+ assert.ok(String(graph["6"].inputs.value).endsWith(LIGHT_LOOKS[1].light)); assert.ok(!String(graph["6"].inputs.value).includes("{look}"));
+ assert.equal(graph["15"].inputs["images.image1"], undefined);
+ assert.deepEqual(graph["15"].inputs["images.image0"], ["4", 0]);
+ assert.equal(graph["15"].inputs["resolution.megapixels"], 3);
+ assert.deepEqual(graph["7"].inputs, { "images.image0": ["4", 0], "images.image1": ["3", 0] });
+ assert.deepEqual(graph["19"].inputs.latent_image, ["relight_latent", 0]);
+ assert.deepEqual(graph["relight_canvas"].inputs.image, ["3", 0]);
+ const custom = load("prepare");
+ applyRelight(custom, "Light the new photo with candles.");
+ assert.ok(String(custom["6"].inputs.value).endsWith(" Light the new photo with candles."));
+});
+
test("picture pipelines have valid links and no disconnected processing nodes", () => {
- for (const stage of ["prepare", "motion", "solar"]) {
- const graph = load(stage);
+ const relit = load("prepare");
+ applyRelight(relit, LIGHT_LOOKS[0].light);
+ for (const [stage, graph] of [...["prepare", "motion", "solar"].map((stage) => [stage, load(stage)] as const), ["relight", relit] as const]) {
const reached = new Set<string>();
function visit(id: string) {
if (reached.has(id)) return;
diff --git a/workflows/picture/README.md b/workflows/picture/README.md
index 206c41f..40c4b38 100644
--- a/workflows/picture/README.md
+++ b/workflows/picture/README.md
@@ -44,6 +44,8 @@ change <image 1> character pose follow the <image 2> pose. keep the final image
The image-aware prompt upsampler sees both references and expands that request into an editing instruction. Both references also reach FLUX directly, in the same order. The expansion rules use gender-neutral wording. This first version uses FLUX; the Qwen Image trial copied the source setting into the result.
+**Relight** (off by default) poses and relights the character in the same FLUX pass; a second edit on top of the prepared picture degrades it. Pick one of nine light looks or write your own. The app then sends the prepare workflow with these changes (`applyRelight` in `lib/picture.ts`): FLUX gets only image 1, resized to 3 MP; image 2 reaches it only through the upsampler's written pose; the canvas is image 2's shape at 3 MP; the request's `{look}` is replaced by the light. The light usually changes the background too, and results vary a lot by seed. Relit pictures save as `relit_frame_*.png`.
+
Preparation defaults to a 768×1376 canvas and saves under `ComfyUI/output/openjutsu_picture/flux3/prepared_frame_*.png`. Compare the prepared picture against the raw first-frame preview before using it for video; a successful execution does not guarantee that the requested pose was followed.
The **motion** and **solar** variants wire the decoded prepared picture directly into H3. No manual re-upload is needed. They default to **141 frames / 5.875 seconds at 24 fps**, 480×864, four SolarRenderer steps or six H3/DMAD steps, and the original soundtrack (or silence if absent). Both renderers let you select up to 15 seconds, with no context loop. It rounds down to the H3 17k+5 frame grid: 6 seconds becomes 141 frames / 5.875 seconds; 15 seconds becomes 345 frames / 14.375 seconds. The duration shown in the settings is the actual render length. Models, LoRAs (including individual switches and strengths), H3 text encoder, VAEs, steps and seed can be changed before rendering. These settings persist across refreshes. Change `frame_load_cap` and H3's `length` together to select another valid H3 clip length.