One-click character swap for videos: an open-source, local Higgsfield Genjutsu alternative built on ComfyUI and MiniMax H3, with FLUX.3 Klein relighting.
Add Part to use, No lip-sync and motion reference size
D3voz0924065168d1
diff --git a/README.md b/README.mdindex ae0e5b7..0381eec 100644--- a/README.md+++ b/README.md@@ -78,10 +78,14 @@ A 6 s clip at 480p takes about 6–13 minutes on a 4060 Ti, most of it in H3 sam## Tips- Only the main person is regenerated; anyone else in the video stays as they are.+- **Part to use** (under the source video) swaps only part of the video: **Whole**, or a **Length** placed at the **Start**, **Middle** or **End**, fine-tuned with **Starts at**. The sound is cut to the same part, the preview loops it, and Check mask uses it too. A shorter part renders faster (fewer clips).+- **Mirrors and reflections do not work yet**, in either mode. The reflection keeps the original person, and H3 copies their look from it: hair, outfit and build come back on your character. Use videos without a mirror, window or other reflection of the person.- **Who do they replace?** names the person for the prompt by role (`the singer`), never by clothes: clothing words leak into the result. Leave it blank and the describer names them.- **What to replace** (Simple mode and Check mask) is the text SAM tracks: `person` by default, an animal such as `cat`, or a short description (`black and white cat`) when the biggest match is the wrong one. The biggest match in the frame is replaced.- **Hat fix** also masks the hat, so it is replaced too (about 45 s more per clip).+- **No lip-sync** (off by default) is for songs the people in the video do not sing or talk along to. Otherwise H3 makes the new people lip-sync any vocals. It gives H3 silent audio and a prompt saying nobody sings, and shows it the source faces for head turns (about 30 s more per clip); the final video still has the original soundtrack. Mouths can still smile more widely than the source.- **Check mask** runs only SAM and the mask (about a minute per 3 s of video) and shows a preview you can scrub frame by frame: cyan is the area H3 regenerates, orange the tracked person, and on the right the grey mannequin H3 copies the motion from. Anything outside the cyan area stays as it is. If the character is cut off or too much background changes, set **Mask margin** (Tight, Normal, Loose) and check again; a re-check takes seconds.+- **Motion reference** (under Advanced) is the size of the grey figures H3 copies the motion from: 512 px by default, or 768 or 1376 px. For general, simple videos 512 px is probably enough, but a bigger size does seem to improve quality a bit: H3 follows the source poses and placement more closely. Sampling takes about a third longer at 768 px and about 2.4× as long at 1376 px (H3 uses at most 768×1344 for references). Each size keeps its own saved clips, so switching size renders again instead of resuming.- Change the **seed** (under Advanced) for a new take. Running the same inputs again resumes from the last finished clip.- **Steps**, **Base model** and **LoRAs** (under Advanced) default to the H3 pruned INT8 model with the DMAD speed LoRA at 6 steps. LoRAs apply in order; without a speed LoRA, use 20 steps or more. Check mask ignores them.- **Fix faces:** drop the finished video and the photo the person was made from. **Match the photo** redraws the face from the photo: the closest likeness, but H3 makes up the expressions and mouth movements. **Follow the mouth** keeps the video's expressions and mouth movements, with a little less likeness. Only that person's face changes (matched by face identity); faces taller than a fifth of the frame are left alone, and stretches with only such faces are skipped. A 17 s video at 768p (4 chunks) took about 28 minutes on a 4060 Ti, including model loading. The video is read at 24 fps.@@ -91,9 +95,10 @@ A 6 s clip at 480p takes about 6–13 minutes on a 4060 Ti, most of it in H3 samEach clip, SAM 3.1 tracks everyone at 1008 px (its own working size), and H3 People Mask keeps the largest person. The edit mask adds 10% headroom above the head, holds each pose for 2 frames and grows by 40 px, so a differently built character has room; it does not affect the motion. The motion reference H3 follows is the source clip with that person turned into a sharp grey mannequin of their exact silhouette, so the pose, arms and fingers stay readable.-`workflows/simple.api.json` is one API export of that workflow with the hat fix enabled. For each request, `lib/simple.ts`:+`workflows/simple.api.json` is one API export of that workflow with the hat fix and No lip-sync enabled. For each request, `lib/simple.ts`:- without the hat fix, removes the hat nodes and connects Grow Mask to the People Mask output;+- without No lip-sync, removes the face and mute nodes, so H3 hears the soundtrack and the plan keeps its lip-sync line;- sets the Swap Plan's resolution, seed, clip length and role.`lib/model.ts` then sets the Swap Plan's steps and the H3 Diffusion Model file, and replaces the DMAD LoRA node with the chosen LoRAs in a chain (or links the model straight through when there are none). Advanced mode does the same.@@ -102,7 +107,7 @@ Each clip, SAM 3.1 tracks everyone at 1008 px (its own working size), and H3 Peo`workflows/mask.api.json` is the mask check: the same SAM, People Mask and grow settings over the whole video, with the mask tinted over the frames next to the mannequin, saved as an MP4.-Advanced mode labels the people before it masks them. Per clip, H3 Track Crops shows each SAM track once per shot (the person alone, and the frame with them boxed) to Qwen3-VL 8B, which names the subject or "other"; H3 Track Roles reads that answer and settles a subject named twice in one shot by colour. H3 Role Mask builds the edit mask from the labelled tracks only, and the shot-list describer sees each labelled person outlined in their subject's colour, so the prompt names the same people the masks do. `lib/advanced.ts` finds nodes by title, drops unused picture branches and, without the hat fix, the hat nodes; the exports are made from `user/default/oneclick_advanced_build.js` with every optional group on.+Advanced mode labels the people before it masks them. Per clip, H3 Track Crops shows each SAM track once per shot (the person alone, and the frame with them boxed) to Qwen3-VL 8B, which names the subject or "other"; H3 Track Roles reads that answer and settles a subject named twice in one shot by colour. H3 Role Mask builds the edit mask from the labelled tracks only, and the shot-list describer sees each labelled person outlined in their subject's colour, so the prompt names the same people the masks do. `lib/advanced.ts` finds nodes by title, drops unused picture branches and, without the hat fix or No lip-sync, their nodes; the exports are made from `user/default/oneclick_advanced_build.js` with every optional group on.Fix faces works on the face, not the frame. H3 Face Track + Crop follows the face of the person in the photo over the whole video (InsightFace picks them out when there are several) and crops it to 512×512, so a face a few dozen pixels tall fills the canvas. H3 Face Fix Chunks splits the crops into independent chunks (6.5 s by default, overlapping by at least 1 s) and outputs them as lists, so ComfyUI runs the H3 nodes once per chunk. Each chunk is a masked edit: only the SAM face mask is regenerated with the photo as reference, while the hair, ears, collar and background around it stay, which keeps the new face from sliding against the head. For Follow the mouth, a blurred copy of the crop is the motion reference. H3 Face Fix Join crossfades the chunk overlaps and lowers the per-frame paste weight where the face is large, and H3 Face Stitch pastes the face back through a slightly smaller mask with the original soundtrack. The chunks are not chained through the context loop: each one starts from the swap's own frames, so colours do not drift. `lib/face.ts` removes the blurred face for Match the photo and sets the seed and chunk length.diff --git a/app/api/jobs/route.ts b/app/api/jobs/route.tsindex 23370f5..723a954 100644--- a/app/api/jobs/route.ts+++ b/app/api/jobs/route.ts@@ -3,7 +3,7 @@ import { listModels, submitAdvanced, submitFace, submitMask, submitSimple } fromimport { CHUNK_SECONDS, FACE_STYLES, type FaceSettings, type FaceStyle } from "../../../lib/face.ts";import { MAX_CHARACTERS } from "../../../lib/advanced.ts";import { MAX_LORAS, STEPS, STRENGTH, type ModelSettings } from "../../../lib/model.ts";-import { CLIP_SECONDS, DEFAULT_TARGET, MARGINS, MAX_SEED, RESOLUTIONS, type Margin, type Resolution, type SimpleSettings } from "../../../lib/simple.ts";+import { CLIP_SECONDS, DEFAULT_TARGET, MARGINS, MAX_SEED, MIN_SECTION_SECONDS, MOTION_SIZES, RESOLUTIONS, type Margin, type MotionSize, type Resolution, type Section, type SimpleSettings } from "../../../lib/simple.ts";export const runtime = "nodejs";@@ -14,6 +14,15 @@ function parseVideo(form: FormData) {return video;}+// Without start and length the whole video is used.+function parseSection(form: FormData): Section {+ const start = Number(form.get("start") ?? 0);+ const length = Number(form.get("length") ?? 0);+ if (!(start >= 0)) throw new Error("The start of the part to use must be 0 s or later.");+ if (!(length === 0 || length >= MIN_SECTION_SECONDS)) throw new Error(`The part to use must be at least ${MIN_SECTION_SECONDS} s long.`);+ return { start, length };+}+function parseMask(form: FormData) {const margin = String(form.get("margin") ?? "normal");if (!Object.hasOwn(MARGINS, margin)) throw new Error("Choose a supported mask margin.");@@ -30,6 +39,8 @@ function parsePicture(picture: FormDataEntryValue | null) {function parseRun(form: FormData) {const resolution = String(form.get("resolution"));if (!RESOLUTIONS.includes(resolution as Resolution)) throw new Error("Choose a supported resolution.");+ const motion = Number(form.get("motion"));+ if (!MOTION_SIZES.includes(motion as MotionSize)) throw new Error("Choose a supported motion reference size.");const seed = Number(form.get("seed"));if (!Number.isInteger(seed) || seed < 0 || seed > MAX_SEED) throw new Error(`Seed must be a whole number from 0 to ${MAX_SEED}.`);const clipSeconds = Number(form.get("clip_seconds"));@@ -37,7 +48,7 @@ function parseRun(form: FormData) {throw new Error(`Clip length must be ${CLIP_SECONDS.min}-${CLIP_SECONDS.max} s.`);}- return { resolution: resolution as Resolution, ...parseMask(form), seed, clipSeconds };+ return { resolution: resolution as Resolution, motion: motion as MotionSize, ...parseMask(form), seed, clipSeconds };}function parseModel(form: FormData, available: { models: string[]; loras: string[] } | null): ModelSettings {@@ -70,7 +81,7 @@ function parseFace(form: FormData) {function parseSimple(form: FormData) {const picture = parsePicture(form.get("picture"));- const settings: SimpleSettings = { role: String(form.get("role") ?? "").slice(0, 200), ...parseRun(form) };+ const settings: SimpleSettings = { role: String(form.get("role") ?? "").slice(0, 200), noLipSync: form.get("no_lip_sync") === "true", ...parseRun(form) };return { picture, settings };}@@ -80,7 +91,7 @@ function parseAdvanced(form: FormData) {const maxQuality = form.get("max_quality") === "true";if (maxQuality && pictures.length !== 2) throw new Error("One pass per person needs exactly two characters.");const roles = form.getAll("role").map((role) => String(role).slice(0, 200));- return { pictures, maxQuality, settings: { roles, ...parseRun(form) } };+ return { pictures, maxQuality, settings: { roles, noLipSync: form.get("no_lip_sync") === "true", ...parseRun(form) } };}export async function POST(request: Request) {@@ -89,21 +100,22 @@ export async function POST(request: Request) {const form = await request.formData();const mode = form.get("mode");const video = parseVideo(form);+ const section = parseSection(form);if (mode === "mask") {const { hatFix, margin, target } = parseMask(form);- submit = () => submitMask(video, hatFix, margin, target);+ submit = () => submitMask(video, section, hatFix, margin, target);} else if (mode === "simple") {const { picture, settings } = parseSimple(form);const model = parseModel(form, await listModels());- submit = () => submitSimple(video, picture, settings, model);+ submit = () => submitSimple(video, section, picture, settings, model);} else if (mode === "advanced") {const { pictures, maxQuality, settings } = parseAdvanced(form);const model = parseModel(form, await listModels());- submit = () => submitAdvanced(video, pictures, settings, maxQuality, model);+ submit = () => submitAdvanced(video, section, pictures, settings, maxQuality, model);} else if (mode === "face") {const { picture, settings } = parseFace(form);const model = parseModel(form, await listModels());- submit = () => submitFace(video, picture, settings, model);+ submit = () => submitFace(video, section, picture, settings, model);} else {throw new Error("Unknown mode.");}diff --git a/components/openjutsu.tsx b/components/openjutsu.tsxindex f2e94d2..01ea92b 100644--- a/components/openjutsu.tsx+++ b/components/openjutsu.tsx@@ -4,7 +4,7 @@ import { useEffect, useRef, useState, type DragEvent, type ReactNode } from "reaimport type { PublicJob } from "../lib/comfy.ts";import type { NodeInfo } from "../lib/progress.ts";import { clipCount, clipStarts } from "../lib/clips.ts";-import { CLIP_SECONDS, DEFAULT_TARGET, MARGINS, MAX_SEED, RESOLUTIONS, type Margin, type Resolution } from "../lib/simple.ts";+import { CLIP_SECONDS, DEFAULT_TARGET, MARGINS, MAX_SEED, MIN_SECTION_SECONDS, MOTION_SIZES, RESOLUTIONS, type Margin, type MotionSize, type Resolution, type Section } from "../lib/simple.ts";import { MAX_CHARACTERS } from "../lib/advanced.ts";import { DEFAULT_MODEL, MAX_LORAS, STEPS, STRENGTH, type Lora } from "../lib/model.ts";import { CHUNK_SECONDS, faceChunkCount, type FaceStyle } from "../lib/face.ts";@@ -51,6 +51,17 @@ function droppedFiles(event: DragEvent, kind: "video" | "image") {return [...event.dataTransfer.files].filter((file) => file.type.startsWith(`${kind}/`) || (kind === "video" && /\.(mkv|mov|avi)$/i.test(file.name)));}+const PLACES = ["Whole", "Start", "Middle", "End"] as const;++// Where a part of the given length sits for each place button.+function placeStart(place: (typeof PLACES)[number], duration: number, length: number) {+ return place === "Middle" ? (duration - length) / 2 : place === "End" ? duration - length : 0;+}++function timecode(seconds: number) {+ return `${Math.floor(seconds / 60)}:${(seconds % 60).toFixed(1).padStart(4, "0")}`;+}+function elapsed(ms: number) {const s = Math.max(0, Math.floor(ms / 1000));return `${Math.floor(s / 60)}:${String(s % 60).padStart(2, "0")}`;@@ -108,16 +119,20 @@ function DropZone({ accept, onFiles, multiple, children, className }: {export function Openjutsu() {const [mode, setMode] = useState<Mode>("simple");const [video, setVideo] = useState<{ file: File; url: string; duration: number } | null>(null);+ const [part, setPart] = useState<Section>({ start: 0, length: 0 });+ const player = useRef<HTMLVideoElement>(null);const [characters, setCharacters] = useState<Character[]>([]);const [maxQuality, setMaxQuality] = useState(false);const [resolution, setResolution] = useState<Resolution>("736p");const [hatFix, setHatFix] = useState(false);+ const [noLipSync, setNoLipSync] = useState(false);const [margin, setMargin] = useState<Margin>("normal");const [target, setTarget] = useState(DEFAULT_TARGET);const [faceStyle, setFaceStyle] = useState<FaceStyle>("photo");const [chunkSeconds, setChunkSeconds] = useState(CHUNK_SECONDS.default);const [seed, setSeed] = useState(0);const [clipSeconds, setClipSeconds] = useState(CLIP_SECONDS.default);+ const [motion, setMotion] = useState<MotionSize>(MOTION_SIZES[0]);const [steps, setSteps] = useState(DEFAULT_MODEL.steps);const [model, setModel] = useState(DEFAULT_MODEL.model);const [loras, setLoras] = useState<Lora[]>(DEFAULT_MODEL.loras);@@ -176,12 +191,26 @@ export function Openjutsu() {if (video) URL.revokeObjectURL(video.url);const url = URL.createObjectURL(file);setVideo({ file, url, duration: 0 });+ setPart({ start: 0, length: 0 });const probe = document.createElement("video");probe.preload = "metadata";- probe.onloadedmetadata = () => setVideo((current) => (current?.url === url ? { ...current, duration: probe.duration } : current));+ probe.onloadedmetadata = () => {+ setVideo((current) => (current?.url === url ? { ...current, duration: probe.duration } : current));+ setPart({ start: 0, length: probe.duration });+ };probe.src = url;}+ function choosePart(start: number, length: number) {+ const duration = video?.duration ?? 0;+ const next = { start: Math.min(Math.max(0, start), duration - length), length };+ setPart(next);+ if (player.current) player.current.currentTime = next.start;+ }++ const whole = !video?.duration || (part.start === 0 && part.length >= video.duration);+ const place = whole ? "Whole" : PLACES.slice(1).find((p) => Math.abs(placeStart(p, video!.duration, part.length) - part.start) < 0.05);+// Simple mode and Fix faces use only the first character; the others are kept for switching back.const single = mode === "simple" || mode === "face";const limit = single ? 1 : MAX_CHARACTERS;@@ -209,7 +238,7 @@ export function Openjutsu() {setCharacters(characters.map((c, i) => (i === index ? { ...c, role } : c)));}- const clips = !video?.duration ? 0 : mode === "face" ? faceChunkCount(video.duration, chunkSeconds) : clipCount(video.duration, clipSeconds);+ const clips = !video?.duration ? 0 : mode === "face" ? faceChunkCount(part.length || video.duration, chunkSeconds) : clipCount(part.length || video.duration, clipSeconds);async function submit(kind: Kind) {if (!video || (kind !== "mask" && !character)) return;@@ -218,7 +247,12 @@ export function Openjutsu() {const form = new FormData();form.set("mode", kind);form.set("video", video.file);+ if (!whole) {+ form.set("start", part.start.toFixed(2));+ form.set("length", part.length.toFixed(2));+ }form.set("hat_fix", String(hatFix));+ form.set("no_lip_sync", String(noLipSync));form.set("margin", margin);form.set("target", target);if (kind === "simple" && character) {@@ -246,6 +280,7 @@ export function Openjutsu() {form.set("clip_seconds", String(chunkSeconds));}if (kind !== "mask") {+ form.set("motion", String(motion));form.set("steps", String(steps));form.set("model", model);for (const l of loras) {@@ -324,11 +359,34 @@ export function Openjutsu() {<h2><span>1</span> Source video</h2>{video ? (<div className="video-pick">- <video src={video.url} controls muted playsInline />+ <video+ ref={player} src={video.url} controls muted playsInline+ onTimeUpdate={(event) => { if (!whole && event.currentTarget.currentTime >= part.start + part.length) event.currentTarget.currentTime = part.start; }}+ /><div className="row"><small>{video.file.name}{video.duration ? ` · ${video.duration.toFixed(1)} s` : ""}</small><DropZone accept="video/*,.mkv" onFiles={chooseVideo} className="link">Replace</DropZone></div>+ {video.duration > MIN_SECTION_SECONDS && (+ <>+ <div className="setting">+ <span>Part to use <small>{timecode(part.start)}–{timecode(part.start + part.length)} ({part.length.toFixed(1)} s). Only this part is swapped.</small></span>+ <div className="segmented">+ {PLACES.map((p) => (+ <button key={p} className={p === place ? "on" : ""} onClick={() => p === "Whole" ? choosePart(0, video.duration) : choosePart(placeStart(p, video.duration, part.length), part.length)}>{p}</button>+ ))}+ </div>+ </div>+ <label className="setting">+ <span>Length <small>{part.length.toFixed(1)} s</small></span>+ <input type="range" min={MIN_SECTION_SECONDS} max={video.duration} step={0.5} value={part.length} onChange={(event) => choosePart(part.start, Number(event.target.value))} />+ </label>+ <label className="setting">+ <span>Starts at <small>{timecode(part.start)}</small></span>+ <input type="range" min={0} max={video.duration - part.length} step={0.1} value={part.start} disabled={whole} onChange={(event) => choosePart(Number(event.target.value), part.length)} />+ </label>+ </>+ )}</div>) : (<DropZone accept="video/*,.mkv" onFiles={chooseVideo} className="big">@@ -376,12 +434,13 @@ export function Openjutsu() {</>) : mode === "simple" ? (<p className="hint">The character replaces the main person in the video, the one who fills the most of the frame. Everyone else stays.+ Mirrors and reflections do not work yet.To name them for the prompt, use a role (“the singer”), not clothes.</p>) : (<p className="hint">Up to {MAX_CHARACTERS} characters. In every clip each tracked person is matched to a role; people without a- character (extras, passers-by) stay as they are. Name roles (“the drummer”), not clothes, and add how they look+ character (extras, passers-by) stay as they are. Mirrors and reflections do not work yet. Name roles (“the drummer”), not clothes, and add how they lookafter “ - ” to help the matching (“the drummer - man in a cap”). Blank roles go to the people with themost screen time.</p>@@ -410,9 +469,14 @@ export function Openjutsu() {<input type="checkbox" checked={hatFix} onChange={(event) => setHatFix(event.target.checked)} /><i aria-hidden /></label>}+ {mode !== "face" && <label className="setting toggle">+ <span>No lip-sync <small>For songs the people in the video do not sing or talk along to, so the new {mode === "advanced" ? "people" : "person"} do{mode === "advanced" ? "" : "es"} not lip-sync them (about 30 s more per clip).</small></span>+ <input type="checkbox" checked={noLipSync} onChange={(event) => setNoLipSync(event.target.checked)} />+ <i aria-hidden />+ </label>}{mode === "advanced" && (<label className="setting toggle">- <span>One pass per person <small>{shown.length === 2 ? "Each character gets their own H3 pass with only their photo. About twice the time." : "Needs exactly two characters."}</small></span>+ <span>One pass per person <small>Each character gets their own H3 pass with only their photo, so about twice the generation time.{shown.length === 2 ? "" : " Needs exactly two characters."}</small></span><input type="checkbox" checked={onePassEach} disabled={shown.length !== 2} onChange={(event) => setMaxQuality(event.target.checked)} /><i aria-hidden /></label>@@ -441,6 +505,14 @@ export function Openjutsu() {<input type="range" min={CLIP_SECONDS.min} max={CLIP_SECONDS.max} step={0.5} value={clipSeconds} onChange={(event) => setClipSeconds(Number(event.target.value))} /></div>)}+ {mode !== "face" && <div className="setting">+ <span>Motion reference <small>Size of the grey figures H3 copies the motion from. Bigger follows the source poses a bit more closely; sampling takes about a third longer at 768 px and about 2.4× as long at 1376 px.</small></span>+ <div className="segmented">+ {MOTION_SIZES.map((m) => (+ <button key={m} className={m === motion ? "on" : ""} onClick={() => setMotion(m)}>{m} px</button>+ ))}+ </div>+ </div>}<div className="setting"><span>Steps <small>6 with the DMAD speed LoRA; without a speed LoRA, 20 or more.</small></span><inputdiff --git a/lib/advanced.ts b/lib/advanced.tsindex 58ef58d..2db35b7 100644--- a/lib/advanced.ts+++ b/lib/advanced.ts@@ -1,15 +1,17 @@// Advanced motion transfer: 1-4 characters, SAM tracks labelled by role, only the labelled people regenerated.// workflows/advanced.api.json (one pass) and advanced_maxq.api.json (one pass per person, 2 characters) are exported-// with every picture and the hat fix enabled; each request prunes what it does not use. Nodes are found by title, as+// with every picture, the hat fix and No lip-sync enabled; each request prunes what it does not use. Nodes are found by title, as// the builder (ComfyUI user/default/oneclick_advanced_build.js) names them.-import { MARGINS, isLink, type ApiGraph, type Margin, type Resolution } from "./simple.ts";+import { MARGINS, dropNoLipSync, isLink, setMotionSize, type ApiGraph, type Margin, type MotionSize, type Resolution } from "./simple.ts";export const MAX_CHARACTERS = 4;export type AdvancedSettings = {roles: string[]; // one per picture, "" to let the describer name themresolution: Resolution;+ motion: MotionSize;hatFix: boolean;+ noLipSync: boolean;margin: Margin;seed: number;clipSeconds: number;@@ -65,6 +67,8 @@ export function pruneAdvanced(source: ApiGraph, pictures: number, settings: Advaelse throw new Error(`Node ${id}.${name} still links to removed node ${value[0]}.`);}}+ if (!settings.noLipSync) dropNoLipSync(graph);+ setMotionSize(graph, settings.motion);for (const node of Object.values(graph)) {if (node.class_type === "GrowMaskWithBlur" && / grow mask /.test(node._meta?.title ?? "")) node.inputs.expand = MARGINS[settings.margin];diff --git a/lib/comfy.ts b/lib/comfy.tsindex af5d464..a5e6ac0 100644--- a/lib/comfy.ts+++ b/lib/comfy.ts@@ -2,7 +2,7 @@ import { readFile } from "node:fs/promises";import { join } from "node:path";import { Comfy, COMFY_CLOUD_BASE_URL, resolveCredentials, type Job } from "@comfyorg/sdk";import { ComfyLow } from "@comfyorg/sdk/low";-import { MASK_NODES, NODES, pruneMask, pruneSimple, type ApiGraph, type Margin, type SimpleSettings } from "./simple.ts";+import { MASK_NODES, NODES, pruneMask, pruneSimple, setSection, type ApiGraph, type Margin, type Section, type SimpleSettings } from "./simple.ts";import { VIDEO, nodeByTitle, pruneAdvanced, type AdvancedSettings } from "./advanced.ts";import { applyModel, type ModelSettings } from "./model.ts";import { FACE_PICTURE, FACE_VIDEO, pruneFace, type FaceSettings } from "./face.ts";@@ -70,39 +70,44 @@ function upload(comfy: Comfy, file: File) {);}-export async function submitSimple(video: File, picture: File, settings: SimpleSettings, model: ModelSettings) {+export async function submitSimple(video: File, section: Section, picture: File, settings: SimpleSettings, model: ModelSettings) {const comfy = client();const graph = pruneSimple(await loadGraph("simple"), settings);applyModel(graph, model);+ setSection(graph, NODES.video, section);const workflow = comfy.workflows.fromJson(graph);workflow.setInput(NODES.video, "video", await upload(comfy, video));workflow.setInput(NODES.picture, "image", await upload(comfy, picture));return publicJob(await comfy.submit(workflow));}-export async function submitAdvanced(video: File, pictures: File[], settings: AdvancedSettings, maxQuality: boolean, model: ModelSettings) {+export async function submitAdvanced(video: File, section: Section, pictures: File[], settings: AdvancedSettings, maxQuality: boolean, model: ModelSettings) {const comfy = client();const graph = pruneAdvanced(await loadGraph(maxQuality ? "advanced_maxq" : "advanced"), pictures.length, settings);applyModel(graph, model);+ setSection(graph, nodeByTitle(graph, VIDEO), section);const workflow = comfy.workflows.fromJson(graph);workflow.setInput(nodeByTitle(graph, VIDEO), "video", await upload(comfy, video));for (const [i, picture] of pictures.entries()) workflow.setInput(nodeByTitle(graph, `Picture ${i + 1}`), "image", await upload(comfy, picture));return publicJob(await comfy.submit(workflow));}-export async function submitFace(video: File, picture: File, settings: FaceSettings, model: ModelSettings) {+export async function submitFace(video: File, section: Section, picture: File, settings: FaceSettings, model: ModelSettings) {const comfy = client();const graph = pruneFace(await loadGraph("face"), settings);applyModel(graph, model);+ setSection(graph, nodeByTitle(graph, FACE_VIDEO), section);const workflow = comfy.workflows.fromJson(graph);workflow.setInput(nodeByTitle(graph, FACE_VIDEO), "video", await upload(comfy, video));workflow.setInput(nodeByTitle(graph, FACE_PICTURE), "image", await upload(comfy, picture));return publicJob(await comfy.submit(workflow));}-export async function submitMask(video: File, hatFix: boolean, margin: Margin, target: string) {+export async function submitMask(video: File, section: Section, hatFix: boolean, margin: Margin, target: string) {const comfy = client();- const workflow = comfy.workflows.fromJson(pruneMask(await loadGraph("mask"), hatFix, margin, target));+ const graph = pruneMask(await loadGraph("mask"), hatFix, margin, target);+ setSection(graph, MASK_NODES.video, section);+ const workflow = comfy.workflows.fromJson(graph);workflow.setInput(MASK_NODES.video, "video", await upload(comfy, video));return publicJob(await comfy.submit(workflow));}diff --git a/lib/simple.ts b/lib/simple.tsindex 2aa074e..96fdf58 100644--- a/lib/simple.ts+++ b/lib/simple.ts@@ -1,6 +1,6 @@// Simple motion transfer: one character replaces the main person in the video.// workflows/simple.api.json (the swap) and workflows/mask.api.json (the mask check) are exported-// with the hat fix enabled; each request prunes what it does not use.+// with the hat fix (and the swap with No lip-sync) enabled; each request prunes what it does not use.export type ApiNode = { class_type: string; inputs: Record<string, unknown>; _meta?: { title?: string } };export type ApiGraph = Record<string, ApiNode>;@@ -15,12 +15,28 @@ export const MAX_SEED = 0xffffffff;export const MARGINS = { tight: 20, normal: 40, loose: 60 } as const;export type Margin = keyof typeof MARGINS;+// Long edge of the grey-mannequin motion reference H3 follows. H3 itself caps reference videos at 768x1344.+export const MOTION_SIZES = [512, 768, 1376] as const;+export type MotionSize = (typeof MOTION_SIZES)[number];++// Part of the source video to use, in seconds; length 0 runs to the end. VHS cuts the soundtrack to the same part.+export type Section = { start: number; length: number };+export const MIN_SECTION_SECONDS = 2;++export function setSection(graph: ApiGraph, video: string, section: Section) {+ // every export loads the video at force_rate 24+ graph[video].inputs.skip_first_frames = Math.round(section.start * 24);+ graph[video].inputs.frame_load_cap = Math.round(section.length * 24);+}+export const DEFAULT_TARGET = "person";export type SimpleSettings = {role: string; // who the character replaces, "" to let the describer name the main personresolution: Resolution;+ motion: MotionSize;hatFix: boolean;+ noLipSync: boolean; // the people do not sing along: H3 hears silence, sees their faces and is told nobody singsmargin: Margin;target: string; // what SAM tracks; the biggest one is replaced ("person", "cat", ...)seed: number;@@ -72,8 +88,46 @@ function maskSettings(source: ApiGraph, hatFix: boolean, margin: Margin, target:return graph;}+// The No lip-sync nodes, titled the same in every swap export (ComfyUI user/default/oneclick_*_build.js).+const MUTE = /^Mute the (clip|reference) audio H3 hears$/;+const NO_LIP_SYNC = /^(No lip-sync in the prompt|SAM: faces|SAM 3\.1 track faces|Faces \(shown on the mannequins?\))$/;++// No lip-sync off: the nodes go, H3 hears the soundtrack again and the mannequins get no faces.+export function dropNoLipSync(graph: ApiGraph) {+ const through = new Map<string, unknown>();+ for (const [id, node] of Object.entries(graph)) {+ const title = node._meta?.title ?? "";+ if (MUTE.test(title)) through.set(id, node.inputs.audio);+ else if (NO_LIP_SYNC.test(title)) through.set(id, undefined);+ }+ for (const id of through.keys()) delete graph[id];+ for (const [id, node] of Object.entries(graph)) {+ for (const [name, value] of Object.entries(node.inputs)) {+ if (!isLink(value) || !through.has(value[0])) continue;+ if (through.get(value[0])) node.inputs[name] = through.get(value[0]);+ else if (["no_lip_sync", "faces", "face_size"].includes(name)) delete node.inputs[name];+ else throw new Error(`Node ${id}.${name} still links to removed node ${value[0]}.`);+ }+ }+}++// Every motion reference resize in a swap export, including One pass per person's pass 2. The Swap Plan's run name+// (which picks the saved clips a rerun resumes from) does not know the size, so other sizes get their own run.+export function setMotionSize(graph: ApiGraph, size: MotionSize) {+ const nodes = Object.values(graph).filter((node) => node.class_type === "ImageScaleToMaxDimension" && /motion reference/i.test(node._meta?.title ?? ""));+ if (!nodes.length) throw new Error("Expected a motion reference resize in the workflow.");+ for (const node of nodes) node.inputs.largest_size = size;+ if (size === MOTION_SIZES[0]) return;+ const plans = Object.values(graph).filter((node) => node.class_type === "MiniMaxH3ChainPlanModern");+ if (plans.length !== 1) throw new Error(`Expected one Production Plan node in the workflow, found ${plans.length}.`);+ graph.motion_run_name = { class_type: "StringConcatenate", inputs: { string_a: plans[0].inputs.run_name, string_b: `_motion${size}`, delimiter: "" }, _meta: { title: "Run name for this motion reference size" } };+ plans[0].inputs.run_name = ["motion_run_name", 0];+}+export function pruneSimple(source: ApiGraph, settings: SimpleSettings): ApiGraph {const graph = maskSettings(source, settings.hatFix, settings.margin, settings.target, NODES);+ if (!settings.noLipSync) dropNoLipSync(graph);+ setMotionSize(graph, settings.motion);const plan = graph[NODES.plan].inputs;plan.resolution = settings.resolution;plan.seed = settings.seed;diff --git a/test/advanced.test.ts b/test/advanced.test.tsindex 1f66f4f..9e7566c 100644--- a/test/advanced.test.ts+++ b/test/advanced.test.ts@@ -8,7 +8,7 @@ import { isLink, type ApiGraph } from "../lib/simple.ts";const load = (name: string): ApiGraph => JSON.parse(readFileSync(new URL(`../workflows/${name}.api.json`, import.meta.url), "utf8"));const graph = load("advanced");const maxq = load("advanced_maxq");-const settings = (hatFix: boolean, roles: string[] = []): AdvancedSettings => ({ roles, resolution: "480p", hatFix, margin: "tight", seed: 3, clipSeconds: 6 });+const settings = (hatFix: boolean, roles: string[] = [], noLipSync = true): AdvancedSettings => ({ roles, resolution: "480p", motion: 512, hatFix, noLipSync, margin: "tight", seed: 3, clipSeconds: 6 });const titles = (g: ApiGraph) => Object.values(g).map((node) => node._meta?.title ?? "");const dangling = (g: ApiGraph) => Object.entries(g).flatMap(([id, node]) =>Object.entries(node.inputs).filter(([, v]) => isLink(v) && !g[v[0]]).map(([name]) => `${id}.${name}`));@@ -49,6 +49,31 @@ test("without hat fix the grow masks read the role masks", () => {}});+test("without No lip-sync every pass hears the soundtrack and sees no faces", () => {+ for (const [g, n] of [[graph, 4], [maxq, 2]] as const) {+ const pruned = pruneAdvanced(g, n, settings(true, [], false));+ assert.deepEqual(dangling(pruned), []);+ assert.equal(Object.keys(pruned).length, Object.keys(g).length - 6);+ for (const node of Object.values(pruned)) {+ if (node.class_type === "MiniMaxH3ReferenceToVideo") assert.equal(pruned[(node.inputs["ref_video_audios.ref_video_audio_0"] as [string, number])[0]].class_type, "MiniMaxH3ChainCurrent");+ if (node.class_type === "MiniMaxH3ContexLoopSourceSlice") assert.equal(pruned[(node.inputs.source_audio as [string, number])[0]].class_type, "TrimAudioDuration");+ if (node.class_type === "H3GreyMannequins") assert.ok(!("faces" in node.inputs));+ }+ assert.ok(!("no_lip_sync" in pruned[nodeByTitle(pruned, "Swap Plan (roles, seed, size)")].inputs));+ }+});++test("with No lip-sync every pass sees the faces and hears silence", () => {+ const pruned = pruneAdvanced(maxq, 2, settings(false, [], true));+ assert.deepEqual(dangling(pruned), []);+ const mannequins = Object.values(pruned).filter((node) => node.class_type === "H3GreyMannequins");+ assert.equal(mannequins.length, 2);+ for (const m of mannequins) assert.ok(isLink(m.inputs.faces));+ for (const node of Object.values(pruned)) {+ if (node.class_type === "MiniMaxH3ReferenceToVideo") assert.equal(pruned[(node.inputs["ref_video_audios.ref_video_audio_0"] as [string, number])[0]].inputs.volume, -100);+ }+});+test("roles fill in for blank ones only when some are given", () => {const plan = (roles: string[]) => {const pruned = pruneAdvanced(graph, 3, settings(false, roles));@@ -61,6 +86,14 @@ test("roles fill in for blank ones only when some are given", () => {assert.deepEqual([inputs.resolution, inputs.seed, inputs.clip_seconds], ["480p", 3, 6]);});+test("the motion reference size reaches both passes", () => {+ const sizes = (g: typeof graph) => Object.values(g)+ .filter((node) => node.class_type === "ImageScaleToMaxDimension" && /motion reference/i.test(node._meta?.title ?? ""))+ .map((node) => node.inputs.largest_size);+ assert.deepEqual(sizes(pruneAdvanced(graph, 2, { ...settings(false), motion: 1376 })), [1376]);+ assert.deepEqual(sizes(pruneAdvanced(maxq, 2, { ...settings(false), motion: 1376 })), [1376, 1376]);+});+test("picture counts outside the export are refused", () => {assert.throws(() => pruneAdvanced(graph, 0, settings(true)));assert.throws(() => pruneAdvanced(maxq, 3, settings(true)));diff --git a/test/simple.test.ts b/test/simple.test.tsindex 2501ee6..e2bee17 100644--- a/test/simple.test.ts+++ b/test/simple.test.ts@@ -3,13 +3,15 @@ import { readFileSync } from "node:fs";import test from "node:test";import { clipCount, clipStarts } from "../lib/clips.ts";import { nodeDescriber } from "../lib/progress.ts";-import { MASK_NODES, NODES, pruneMask, pruneSimple, type ApiGraph, type SimpleSettings } from "../lib/simple.ts";+import { MASK_NODES, NODES, pruneMask, pruneSimple, setSection, type ApiGraph, type SimpleSettings } from "../lib/simple.ts";+const isLinkTo = (g: ApiGraph, value: unknown, type: string) => Array.isArray(value) && g[value[0] as string]?.class_type === type;const load = (name: string): ApiGraph => JSON.parse(readFileSync(new URL(`../workflows/${name}.api.json`, import.meta.url), "utf8"));const graph = load("simple");const mask = load("mask");const HAT_TYPES = ["CLIPTextEncode", "SAM3_VideoTrack", "SAM3_TrackToMask", "GrowMaskWithBlur", "MaskComposite", "MaskComposite"];-const settings = (hatFix: boolean): SimpleSettings => ({ role: " the singer ", resolution: "480p", hatFix, margin: "loose", target: "cat", seed: 7, clipSeconds: 6 });+const settings = (hatFix: boolean, noLipSync = true): SimpleSettings => ({ role: " the singer ", resolution: "480p", motion: 512, hatFix, noLipSync, margin: "loose", target: "cat", seed: 7, clipSeconds: 6 });+const byClass = (g: ApiGraph, type: string) => Object.values(g).filter((node) => node.class_type === type);test("the node ids match the export", () => {const types = { picture: "LoadImage", video: "VHS_LoadVideo", plan: "H3OneClickSwapPlan", loopStart: "MiniMaxH3ChainLoopStart", peopleMask: "H3PeopleMask",@@ -32,12 +34,55 @@ test("without hat fix the hat nodes go and the grow mask reads the people mask",assert.ok(graph[NODES.hat[0]], "the source graph is not modified");});+test("a part of the video becomes VHS's skip and frame cap; the whole video keeps 0", () => {+ const pruned = pruneSimple(graph, settings(false));+ setSection(pruned, NODES.video, { start: 3.5, length: 4 });+ assert.deepEqual([pruned[NODES.video].inputs.skip_first_frames, pruned[NODES.video].inputs.frame_load_cap], [84, 96]);+ setSection(pruned, NODES.video, { start: 0, length: 0 });+ assert.deepEqual([pruned[NODES.video].inputs.skip_first_frames, pruned[NODES.video].inputs.frame_load_cap], [0, 0]);+});++test("a bigger motion reference gets its own run name, 512 keeps the plan's", () => {+ const pruned = pruneSimple(graph, { ...settings(false), motion: 768 });+ assert.deepEqual(byClass(pruned, "ImageScaleToMaxDimension").filter((n) => /motion reference/i.test(n._meta?.title ?? "")).map((n) => n.inputs.largest_size), [768]);+ const production = byClass(pruned, "MiniMaxH3ChainPlanModern")[0].inputs.run_name as [string, number];+ assert.deepEqual(pruned[production[0]].inputs, { string_a: [NODES.plan, 3], string_b: "_motion768", delimiter: "" });+ const plain = pruneSimple(graph, settings(false));+ assert.deepEqual(byClass(plain, "MiniMaxH3ChainPlanModern")[0].inputs.run_name, [NODES.plan, 3]);+});+test("with hat fix the full graph is kept", () => {const pruned = pruneSimple(graph, settings(true));assert.equal(Object.keys(pruned).length, Object.keys(graph).length);assert.deepEqual(pruned[NODES.growMask].inputs.mask, [NODES.hat[5], 0]);});+test("without No lip-sync H3 hears the soundtrack and the mannequin gets no faces", () => {+ const pruned = pruneSimple(graph, settings(true, false));+ assert.equal(Object.keys(pruned).length, Object.keys(graph).length - 6);+ assert.deepEqual(byClass(pruned, "AudioAdjustVolume"), []);+ assert.ok(!Object.values(pruned).some((node) => /faces/.test(node._meta?.title ?? "")));+ const [ref] = byClass(pruned, "MiniMaxH3ReferenceToVideo");+ const [src] = byClass(pruned, "MiniMaxH3ContexLoopSourceSlice");+ const [mannequin] = byClass(pruned, "H3GreyMannequins");+ assert.equal(pruned[(ref.inputs["ref_video_audios.ref_video_audio_0"] as [string, number])[0]].class_type, "MiniMaxH3ChainCurrent");+ assert.equal(pruned[(src.inputs.source_audio as [string, number])[0]].class_type, "TrimAudioDuration");+ assert.ok(!("faces" in mannequin.inputs) && !("face_size" in mannequin.inputs));+ assert.ok(!("no_lip_sync" in pruned[NODES.plan].inputs));+});++test("with No lip-sync H3 hears silence and the plan drops the lip-sync line", () => {+ const pruned = pruneSimple(graph, settings(false, true));+ const [ref] = byClass(pruned, "MiniMaxH3ReferenceToVideo");+ const [src] = byClass(pruned, "MiniMaxH3ContexLoopSourceSlice");+ for (const link of [ref.inputs["ref_video_audios.ref_video_audio_0"], src.inputs.source_audio]) {+ assert.deepEqual(pruned[(link as [string, number])[0]].inputs.volume, -100);+ }+ const flag = pruned[(pruned[NODES.plan].inputs.no_lip_sync as [string, number])[0]];+ assert.deepEqual([flag.class_type, flag.inputs.value], ["PrimitiveBoolean", true]);+ assert.ok(isLinkTo(pruned, byClass(pruned, "H3GreyMannequins")[0].inputs.faces, "H3PeopleMask"));+});+test("node ids map to titles and clip numbers", () => {const describe = nodeDescriber(graph);assert.deepEqual(describe("4"), { title: "Describe picture 1", classType: "TextGenerate", clip: null });diff --git a/workflows/advanced.api.json b/workflows/advanced.api.jsonindex a153d83..7627bbb 100644--- a/workflows/advanced.api.json+++ b/workflows/advanced.api.json@@ -281,6 +281,10 @@"steps": 6,"seed": 0,"roles": "",+ "no_lip_sync": [+ "83",+ 0+ ],"source_frames": ["14",0@@ -869,7 +873,7 @@0],"source_audio": [- "15",+ "87",0]},@@ -950,7 +954,8 @@},"61": {"inputs": {- "blur": 0.01,+ "blur": 0.04,+ "face_blur": 0.05,"images": ["41",0@@ -962,6 +967,14 @@"size": ["59",1+ ],+ "faces": [+ "86",+ 0+ ],+ "face_size": [+ "86",+ 1]},"class_type": "H3GreyMannequins",@@ -1079,8 +1092,8 @@0],"ref_video_audios.ref_video_audio_0": [- "38",- 12+ "88",+ 0]},"class_type": "MiniMaxH3ReferenceToVideo",@@ -1097,6 +1110,10 @@"shots": ["52",0+ ],+ "role_map": [+ "50",+ 2]},"class_type": "H3FillShots",@@ -1457,5 +1474,91 @@"_meta": {"title": "Final Video"}+ },+ "83": {+ "inputs": {+ "value": true+ },+ "class_type": "PrimitiveBoolean",+ "_meta": {+ "title": "No lip-sync in the prompt"+ }+ },+ "84": {+ "inputs": {+ "text": "face",+ "clip": [+ "32",+ 1+ ]+ },+ "class_type": "CLIPTextEncode",+ "_meta": {+ "title": "SAM: faces"+ }+ },+ "85": {+ "inputs": {+ "detection_threshold": 0.2,+ "max_objects": 0,+ "detect_interval": 1,+ "images": [+ "43",+ 0+ ],+ "model": [+ "32",+ 0+ ],+ "conditioning": [+ "84",+ 0+ ]+ },+ "class_type": "SAM3_VideoTrack",+ "_meta": {+ "title": "SAM 3.1 track faces"+ }+ },+ "86": {+ "inputs": {+ "headroom": 0,+ "hold_frames": 0,+ "max_people": 0,+ "track_data": [+ "85",+ 0+ ]+ },+ "class_type": "H3PeopleMask",+ "_meta": {+ "title": "Faces (shown on the mannequins)"+ }+ },+ "87": {+ "inputs": {+ "volume": -100,+ "audio": [+ "15",+ 0+ ]+ },+ "class_type": "AudioAdjustVolume",+ "_meta": {+ "title": "Mute the clip audio H3 hears"+ }+ },+ "88": {+ "inputs": {+ "volume": -100,+ "audio": [+ "38",+ 12+ ]+ },+ "class_type": "AudioAdjustVolume",+ "_meta": {+ "title": "Mute the reference audio H3 hears"+ }}}\ No newline at end of filediff --git a/workflows/advanced_maxq.api.json b/workflows/advanced_maxq.api.jsonindex 882566c..7bb2a09 100644--- a/workflows/advanced_maxq.api.json+++ b/workflows/advanced_maxq.api.json@@ -191,6 +191,10 @@"steps": 6,"seed": 0,"roles": "",+ "no_lip_sync": [+ "97",+ 0+ ],"source_frames": ["8",0@@ -763,7 +767,7 @@0],"source_audio": [- "9",+ "101",0]},@@ -844,7 +848,8 @@},"55": {"inputs": {- "blur": 0.01,+ "blur": 0.04,+ "face_blur": 0.05,"images": ["35",0@@ -856,6 +861,14 @@"size": ["53",1+ ],+ "faces": [+ "100",+ 0+ ],+ "face_size": [+ "100",+ 1]},"class_type": "H3GreyMannequins",@@ -961,8 +974,8 @@0],"ref_video_audios.ref_video_audio_0": [- "32",- 12+ "102",+ 0]},"class_type": "MiniMaxH3ReferenceToVideo",@@ -1245,7 +1258,8 @@},"74": {"inputs": {- "blur": 0.01,+ "blur": 0.04,+ "face_blur": 0.05,"images": ["70",0@@ -1257,6 +1271,14 @@"size": ["72",1+ ],+ "faces": [+ "100",+ 0+ ],+ "face_size": [+ "100",+ 1]},"class_type": "H3GreyMannequins",@@ -1362,8 +1384,8 @@0],"ref_video_audios.ref_video_audio_0": [- "32",- 12+ "102",+ 0]},"class_type": "MiniMaxH3ReferenceToVideo",@@ -1757,5 +1779,91 @@"_meta": {"title": "Final Video"}+ },+ "97": {+ "inputs": {+ "value": true+ },+ "class_type": "PrimitiveBoolean",+ "_meta": {+ "title": "No lip-sync in the prompt"+ }+ },+ "98": {+ "inputs": {+ "text": "face",+ "clip": [+ "26",+ 1+ ]+ },+ "class_type": "CLIPTextEncode",+ "_meta": {+ "title": "SAM: faces"+ }+ },+ "99": {+ "inputs": {+ "detection_threshold": 0.2,+ "max_objects": 0,+ "detect_interval": 1,+ "images": [+ "37",+ 0+ ],+ "model": [+ "26",+ 0+ ],+ "conditioning": [+ "98",+ 0+ ]+ },+ "class_type": "SAM3_VideoTrack",+ "_meta": {+ "title": "SAM 3.1 track faces"+ }+ },+ "100": {+ "inputs": {+ "headroom": 0,+ "hold_frames": 0,+ "max_people": 0,+ "track_data": [+ "99",+ 0+ ]+ },+ "class_type": "H3PeopleMask",+ "_meta": {+ "title": "Faces (shown on the mannequins)"+ }+ },+ "101": {+ "inputs": {+ "volume": -100,+ "audio": [+ "9",+ 0+ ]+ },+ "class_type": "AudioAdjustVolume",+ "_meta": {+ "title": "Mute the clip audio H3 hears"+ }+ },+ "102": {+ "inputs": {+ "volume": -100,+ "audio": [+ "32",+ 12+ ]+ },+ "class_type": "AudioAdjustVolume",+ "_meta": {+ "title": "Mute the reference audio H3 hears"+ }}}\ No newline at end of filediff --git a/workflows/mask.api.json b/workflows/mask.api.jsonindex 433b51e..02b5c74 100644--- a/workflows/mask.api.json+++ b/workflows/mask.api.json@@ -12,7 +12,7 @@"11": {"class_type": "SAM3_VideoTrack", "_meta": {"title": "SAM 3.1 track hats"}, "inputs": {"detection_threshold": 0.35, "max_objects": 0, "detect_interval": 1, "images": ["5", 0], "model": ["2", 0], "conditioning": ["10", 0]}},"12": {"class_type": "SAM3_TrackToMask", "_meta": {"title": "All hats"}, "inputs": {"object_indices": "", "track_data": ["11", 0]}},"13": {"class_type": "MaskComposite", "_meta": {"title": "People + hats"}, "inputs": {"x": 0, "y": 0, "operation": "add", "destination": ["7", 0], "source": ["20", 0]}},- "14": {"class_type": "H3GreyMannequins", "_meta": {"title": "Grey mannequin (what H3 follows)"}, "inputs": {"blur": 0.01, "images": ["4", 0], "mask": ["8", 0], "size": ["8", 1]}},+ "14": {"class_type": "H3GreyMannequins", "_meta": {"title": "Grey mannequin (what H3 follows)"}, "inputs": {"blur": 0.04, "images": ["4", 0], "mask": ["8", 0], "size": ["8", 1]}},"15": {"class_type": "ImageAndMaskPreview", "_meta": {"title": "Regenerated area (cyan)"}, "inputs": {"mask_opacity": 0.45, "mask_color": "0, 200, 255", "pass_through": true, "image": ["4", 0], "mask": ["9", 0]}},"16": {"class_type": "ImageAndMaskPreview", "_meta": {"title": "Tracked person (orange)"}, "inputs": {"mask_opacity": 0.35, "mask_color": "255, 140, 0", "pass_through": true, "image": ["15", 0], "mask": ["8", 0]}},"17": {"class_type": "ImageStitch", "_meta": {"title": "Mask | mannequin"}, "inputs": {"direction": "right", "match_image_size": true, "spacing_width": 0, "spacing_color": "white", "image1": ["16", 0], "image2": ["14", 0]}},diff --git a/workflows/simple.api.json b/workflows/simple.api.jsonindex 5e42d0c..4103da7 100644--- a/workflows/simple.api.json+++ b/workflows/simple.api.json@@ -146,6 +146,10 @@"steps": 6,"seed": 0,"roles": "",+ "no_lip_sync": [+ "68",+ 0+ ],"source_frames": ["5",0@@ -425,8 +429,8 @@0],"ref_video_audios.ref_video_audio_0": [- "29",- 12+ "73",+ 0]},"class_type": "MiniMaxH3ReferenceToVideo",@@ -455,7 +459,7 @@0],"source_audio": [- "6",+ "72",0]},@@ -612,7 +616,8 @@},"41": {"inputs": {- "blur": 0.01,+ "blur": 0.04,+ "face_blur": 0.05,"images": ["34",0@@ -624,6 +629,14 @@"size": ["40",1+ ],+ "faces": [+ "71",+ 0+ ],+ "face_size": [+ "71",+ 1]},"class_type": "H3GreyMannequins",@@ -1169,5 +1182,91 @@"_meta": {"title": "Final Video"}+ },+ "68": {+ "inputs": {+ "value": true+ },+ "class_type": "PrimitiveBoolean",+ "_meta": {+ "title": "No lip-sync in the prompt"+ }+ },+ "69": {+ "inputs": {+ "text": "face",+ "clip": [+ "23",+ 1+ ]+ },+ "class_type": "CLIPTextEncode",+ "_meta": {+ "title": "SAM: faces"+ }+ },+ "70": {+ "inputs": {+ "detection_threshold": 0.2,+ "max_objects": 0,+ "detect_interval": 1,+ "images": [+ "36",+ 0+ ],+ "model": [+ "23",+ 0+ ],+ "conditioning": [+ "69",+ 0+ ]+ },+ "class_type": "SAM3_VideoTrack",+ "_meta": {+ "title": "SAM 3.1 track faces"+ }+ },+ "71": {+ "inputs": {+ "headroom": 0,+ "hold_frames": 0,+ "max_people": 0,+ "track_data": [+ "70",+ 0+ ]+ },+ "class_type": "H3PeopleMask",+ "_meta": {+ "title": "Faces (shown on the mannequin)"+ }+ },+ "72": {+ "inputs": {+ "volume": -100,+ "audio": [+ "6",+ 0+ ]+ },+ "class_type": "AudioAdjustVolume",+ "_meta": {+ "title": "Mute the clip audio H3 hears"+ }+ },+ "73": {+ "inputs": {+ "volume": -100,+ "audio": [+ "29",+ 12+ ]+ },+ "class_type": "AudioAdjustVolume",+ "_meta": {+ "title": "Mute the reference audio H3 hears"+ }}}\ No newline at end of file