Python
Mixed Media: anime faces that keep the person, and a new example on a cherry blossom street
neuregexafeae66743b6
diff --git a/README.md b/README.mdindex 4310653..15a3085 100644--- a/README.md+++ b/README.md@@ -22,9 +22,9 @@ browser ──▶ web/ + server/ (Node, Comfy SDK) ──▶ gateway :8000 ─| Apps | 120: 119 *ready* (39 apps with a workflow of their own, the 5 IC-LoRA apps (`genjutsu`, `reframe`, `sdr-hdr`, `mixed-media`, `dub`), `morph`, 72 template apps (a fixed prompt over one of those pipelines, written by `tools/template_apps.py`) and 2 model-free smoke apps) and `moss-image` as a draft. The 118 that use models each have a lab example made on a GPU || Open weights | No API or partner nodes. Every model is a file you download, listed in its app's `app.json` with its Hugging Face repo, size and license; the cloud text-encoder branch of Lightricks' examples is removed || Workflows | Built from official sources by `tools/build_apps.py` (ComfyUI's templates, Lightricks' ComfyUI-LTXVideo examples, Comfy's `comfy-reshoot`); the template apps reuse those bases and the two smoke apps are hand-written. `tools/import_workflow.py` drafts a new app from any other ComfyUI workflow. All accepted by ComfyUI's validator: ComfyUI `e3bee7f` (0.42.0), ComfyUI-LTXVideo `44c8a36`, ComfyUI-CrossViewWarp `fbb1e93` |-| GPU runs | 44 lab runs on Modal from 20 to 26 Oct (L40S and H100), 700 of 723 cases ok. Every example in the web UI comes from one of them, with its time and peak VRAM. Run by run, and what each one changed: [`docs/lab-runs.md`](docs/lab-runs.md) |+| GPU runs | 47 lab runs on Modal from 20 to 26 Oct (L40S and H100), 706 of 729 cases ok. Every example in the web UI comes from one of them, with its time and peak VRAM. Run by run, and what each one changed: [`docs/lab-runs.md`](docs/lab-runs.md) || End to end | Run from the browser on 23 Oct, web UI → server → gateway on Modal → ComfyUI on an H100: `morph` 160.8 s from the button to the video (128.7 s of it on the GPU), `mixed-media` 149 s, `dub` 72 s. On 24 Oct through the hosted demo (`infra/web_app.py`, the same web app on Modal behind a passcode): `face-swap` 25.2 s on the GPU. The model-free smoke apps cover the same path in CI |-| LoRA lab | 328 cases: one showcase per app (117), one per effect (24), one per look of `mixed-media` (16), one per transformation of `morph` (15) and one per character type of `ai-influencer` (8 more), 41 references the others start from, 6 candidate portraits for those references (`refx-*`), 12 style-drift experiments (`mmx-*`), 36 template experiments (`tplx-*`), 2 runs of `reshoot` without its Turbo LoRA, 30 image-model comparisons (`cmp-*`, below), 4 soundtrack candidates for the demo video and 4 checks of new controls (Create Image's looks and 2K, Cinema Studio's genre and palette, Relight's direction), 3 Cinema Studio video tests (two camera moves in order, a cut on Dynamic tempo, the enhancer off) 1 check of several picks per group in AI Influencer and 1 clip for the demo video; each experiment has its verdict in the note |+| LoRA lab | 330 cases: one showcase per app (117), one per effect (24), one per look of `mixed-media` (16), one per transformation of `morph` (15) and one per character type of `ai-influencer` (8 more), 42 references the others start from, 6 candidate portraits for those references (`refx-*`), 12 style-drift experiments (`mmx-*`), 36 template experiments (`tplx-*`), 2 runs of `reshoot` without its Turbo LoRA, 30 image-model comparisons (`cmp-*`, below), 4 soundtrack candidates for the demo video and 4 checks of new controls (Create Image's looks and 2K, Cinema Studio's genre and palette, Relight's direction), 3 Cinema Studio video tests (two camera moves in order, a cut on Dynamic tempo, the enhancer off) 1 check of several picks per group in AI Influencer, 1 clip for the demo video and 1 more anime clip for Mixed Media; each experiment has its verdict in the note || CI | `.github/workflows/validate.yml` on every push: the manifest validator, the tests, the type check, the generated packs and VRAM map, and a CPU-only ComfyUI that accepts every workflow and the prompt of every lab case (about 3 minutes) |## Appsdiff --git a/apps/mixed-media/app.json b/apps/mixed-media/app.jsonindex aa540a0..e08eb50 100644--- a/apps/mixed-media/app.json+++ b/apps/mixed-media/app.json@@ -54,7 +54,7 @@{"node": "ff_style","field": "value",- "value": "Japanese anime style, cel shading, clean line art, vibrant flat colors"+ "value": "Japanese TV anime style, cel shading, clean line art, vibrant flat colors; people drawn as anime characters with large expressive eyes and small noses, each keeping their own skin tone and hair"}]},diff --git a/apps/mixed-media/workflow.api.json b/apps/mixed-media/workflow.api.jsonindex 8d7278c..712f3dc 100644--- a/apps/mixed-media/workflow.api.json+++ b/apps/mixed-media/workflow.api.json@@ -806,7 +806,7 @@0],"text": [- "ff_edit_prompt",+ "ff_edit_full",0]},@@ -915,7 +915,7 @@"ff_edit_prompt": {"class_type": "StringConcatenate","inputs": {- "string_a": "Redraw this exact photo in this look, keeping the composition, the people and the place:",+ "string_a": "Redraw this exact photo in this look, keeping the composition, the people, their expressions and the place:","string_b": ["ff_style",0@@ -925,5 +925,22 @@"_meta": {"title": "Edit instruction"}+ },+ "ff_edit_full": {+ "class_type": "StringConcatenate",+ "inputs": {+ "string_a": [+ "ff_edit_prompt",+ 0+ ],+ "string_b": [+ "5508",+ 0+ ],+ "delimiter": ". "+ },+ "_meta": {+ "title": "Edit instruction + what is in the shot"+ }}}diff --git a/docs/lab-runs.md b/docs/lab-runs.mdindex 29d1fc3..75a16c6 100644--- a/docs/lab-runs.md+++ b/docs/lab-runs.md@@ -3,7 +3,7 @@Every GPU run of the lab (`lab/lab_modal.py`) from 20 to 26 Oct 2026, in order. Each run is a folder`lab/results/<run id>` with `results.jsonl` (one row per case: status, seconds, peak VRAM), `run.json` (GPU, starttime) and `review.html`. Peaks are the highest of the run; on an 80 GB H100 ComfyUI can keep the models of earlier-steps loaded, so a peak there is not the least an app needs. 44 runs, 700 of 723 cases ok: 22 of the 23 failures are+steps loaded, so a peak there is not the least an app needs. 47 runs, 706 of 729 cases ok: 22 of the 23 failures arethe first IC-LoRA run, before the gated licenses were accepted, and the other is `recast` in the first run.| Run | GPU | What ran | Cases ok | Longest case | Peak |@@ -52,6 +52,9 @@ the first IC-LoRA run, before the gated licenses were accepted, and the other is| `20261026-021507-l40s` | L40S | Queue job 22: the photo mode with plain shoes asked for | 2 / 2 | 18 s | 12.6 GB || `20261026-021906-l40s` | L40S | Queue job 23: the photo mode example waist-up at a night food stall | 2 / 2 | 19 s | 13.2 GB || `20261026-023750-l40s` | L40S | Queue job 24: for the demo video, AI Influencer's example animated by Image to Video, as *Use in…* does | 2 / 2 | 91 s | 43.9 GB |+| `20261026-034805-l40s` | L40S | Queue job 25: a second Mixed Media clip (a Japanese woman on a cherry blossom street) and its anime look | 2 / 2 | 121 s | 44.2 GB |+| `20261026-040431-l40s` | L40S | Queue job 26: the Anime look asking for anime eyes and each person's own skin tone | 2 / 2 | 117 s | 44.2 GB |+| `20261026-041209-l40s` | L40S | Queue job 27: the shot description also reaching Klein's first frame; the new Mixed Media example | 2 / 2 | 110 s | 44.2 GB |A run with several GPUs (`--shards N`) splits the cases into N groups that run at the same time and comes back asone folder. The cases a run lists include the reference images, clips and songs that the others read.@@ -123,6 +126,10 @@ one folder. The cases a run lists include the reference images, clips and songsseveral picks showed (face gems, glasses, headphones); its first runs drew a "G/O" print on a vest and brand-likemarks on sneakers until the prompt asked for plain clothes and shoes, and the photo mode cut the hair short until itwas told to keep the hairstyle unless a new one is picked.+- **Mixed Media's anime, for the demo video (26 Oct).** On a new clip of a Japanese woman on a cherry blossom street the+ anime first frame came out tan, narrow-eyed and heavy-faced: Klein kept her real proportions under warm light. Asking+ the look for anime eyes and each person's own skin tone gave anime eyes but a frown; passing the shot description to+ Klein as well (who is in it: a smiling woman in her twenties) gave the smile back. That clip is now the app's example.## End to end from the browser (23 Oct)diff --git a/lab/cases.json b/lab/cases.jsonindex f1cb8ea..74bbbd7 100644--- a/lab/cases.json+++ b/lab/cases.json@@ -209,8 +209,7 @@"note": "The walking woman's clip widened: new sidewalk on both sides, consistent with the prompt and with her motion."},{"id": "sdr-hdr-lake", "showcase": true, "app": "sdr-hdr", "values": {"clip": "@case:ref-pan", "size": 720, "exposure": 0},"note": "Two outputs: the HLG master (10-bit) and the tone-mapped preview. The sunrise and mist should gain highlight range; colours must not shift."},- {"id": "mm-anime", "app": "mixed-media", "showcase": true, "values": {"clip": "@case:ref-stroll", "style": "anime", "details": "A young woman in a yellow summer dress walking toward the camera along a park path", "size": 544},- "note": "Anime: the walk and the street layout must stay (edges), the look must change all over the frame."},+ {"id": "mm-anime", "app": "mixed-media", "values": {"clip": "@case:ref-stroll", "style": "anime", "details": "A young woman in a yellow summer dress walking toward the camera along a park path", "size": 544}, "note": "Anime: the walk and the street layout must stay (edges), the look must change all over the frame. Showcase until 26 Oct; the cherry blossom case (mm-anime-tokyo) replaced it."},{"id": "mm-claymation", "app": "mixed-media", "values": {"clip": "@case:ref-stroll", "style": "claymation", "details": "A young woman in a yellow summer dress walking toward the camera along a park path", "size": 544},"note": "Claymation: the walk and the street layout must stay (edges), the look must change all over the frame."},{"id": "mm-watercolor", "app": "mixed-media", "values": {"clip": "@case:ref-stroll", "style": "watercolor", "details": "A young woman in a yellow summer dress walking toward the camera along a park path", "size": 544},@@ -578,6 +577,9 @@{"id": "demo-influencer-video", "app": "video", "values": {"image": "@case:tpl-ai-influencer", "prompt": "She smiles at the camera, waves with one hand and shifts her weight, her curls moving; a quiet photo studio, soft even light", "duration": 4, "aspect": "9:16 (Portrait Widescreen)", "enhance": true}, "note": "For the demo video: AI Influencer's example sent to Image to Video with Use in…, as a user would."},+ {"id": "ref-stroll-tokyo", "app": "video-t2v", "values": {"prompt": "Medium shot of a Japanese woman in her mid-twenties with long straight black hair in a ponytail, a white blouse and a light blue cardigan, walking toward the camera along a quiet Tokyo street lined with cherry blossom trees in full bloom, petals drifting, spring afternoon light, gentle smile, cinematic, realistic", "duration": 4, "enhance": false}, "note": "Second reference clip for Mixed Media (26 Oct, asked for the demo video): a Japanese woman on a cherry blossom street, for a look closer to Japanese anime."},+ {"id": "mm-anime-tokyo", "showcase": true, "app": "mixed-media", "values": {"clip": "@case:ref-stroll-tokyo", "style": "anime", "details": "A smiling Japanese woman in her twenties with a black ponytail and fair skin, in a white blouse and a light blue cardigan, walks toward the camera along a street lined with cherry blossom trees, petals in the air", "size": 544}, "note": "Anime on the cherry blossom clip (asked for the demo video, 26 Oct). Run 20261026-034805-l40s (121.0 s, 44.2 GB): a Japanese anime street, but the woman came out tan with narrow eyes and a heavy face, because Klein kept her real proportions under warm light; the Anime look now asks for anime eyes and noses and for each person to keep their own skin tone and hair. Run 20261026-040431-l40s (117.4 s): anime eyes now, but still tan and frowning; the shot description now reaches Klein's first frame too, with the smile and her fair skin, and the instruction keeps expressions. Run 20261026-041209-l40s (109.6 s, 44.2 GB): smiling, anime eyes and rosy cheeks, the walk and the street hold. Now the app's example."},+{"id": "ref-tpl-sketch", "app": "z-image", "values": {"prompt": "A rough pencil sketch on white paper of a cat sitting on a windowsill next to a potted plant, loose lines, no color", "width": 1024, "height": 1024},"note": "Reference sketch for Sketch to Real."},{"id": "ref-tpl-sign", "app": "z-image", "values": {"prompt": "Photo of a small bakery storefront with a wooden sign above the door that reads OPEN in bold white letters, daylight, photorealistic", "width": 1280, "height": 768},diff --git a/tools/build_apps.py b/tools/build_apps.pyindex 7dd96b5..8eb51c8 100644--- a/tools/build_apps.py+++ b/tools/build_apps.py@@ -426,10 +426,15 @@ def ltx_union_control(api: Api, edit: Api) -> None:rewire(api, ["edit:11", 0], ["ff_first_frame", 0])drop(api, "edit:11")api["ff_edit_prompt"] = {"class_type": "StringConcatenate",- "inputs": {"string_a": "Redraw this exact photo in this look, keeping the composition, the people and the place:",+ "inputs": {"string_a": "Redraw this exact photo in this look, keeping the composition, the people, their expressions and the place:","string_b": ["ff_style", 0], "delimiter": " "},"_meta": {"title": "Edit instruction"}}- put(api, "edit:4", "text", ["ff_edit_prompt", 0])+ # the shot description reaches Klein too, so the first frame keeps who is in it (a smile, an age) as well as the layout+ # (26 Oct: without it the anime first frame frowned and turned a smiling woman's face heavy)+ api["ff_edit_full"] = {"class_type": "StringConcatenate",+ "inputs": {"string_a": ["ff_edit_prompt", 0], "string_b": ["5508", 0], "delimiter": ". "},+ "_meta": {"title": "Edit instruction + what is in the shot"}}+ put(api, "edit:4", "text", ["ff_edit_full", 0])styled = find(api, "VAEDecode", vae=["edit:3", 0])load_image = find(api, "LoadImage")rewire(api, [load_image, 0], [styled, 0])diff --git a/web/examples/index.json b/web/examples/index.jsonindex f53940c..a7543cb 100644--- a/web/examples/index.json+++ b/web/examples/index.json@@ -3069,7 +3069,7 @@"run": "20261022-181155-h100"},"mixed-media": {- "case": "mm-anime",+ "case": "mm-anime-tokyo","type": "video","card": "examples/mixed-media/card.mp4","poster": "examples/mixed-media/poster.jpg",@@ -3081,40 +3081,40 @@"type": "video","src": "examples/mixed-media/input-clip.jpg","full": "examples/mixed-media/input-clip-full.mp4",- "from_case": "ref-stroll"+ "from_case": "ref-stroll-tokyo"}],"values": {"style": "anime",- "details": "A young woman in a yellow summer dress walking toward the camera along a park path",+ "details": "A smiling Japanese woman in her twenties with a black ponytail and fair skin, in a white blouse and a light blue cardigan, walks toward the camera along a street lined with cherry blossom trees, petals in the air","negative": "photorealistic, photo, blurry, distorted","size": 544,"strength": 0.7},"seed": 42,- "seconds": 109.4,- "peak_vram_gb": 51.25,- "gpu": "NVIDIA H100 80GB HBM3, 81559 MiB",- "run": "20261023-143551-h100",+ "seconds": 109.6,+ "peak_vram_gb": 44.16,+ "gpu": "NVIDIA L40S, 46068 MiB",+ "run": "20261026-041209-l40s","options": {"style": {"anime": {- "case": "mm-anime",+ "case": "mm-anime-tokyo","type": "video","card": "examples/mixed-media/options/style/anime/card.mp4","poster": "examples/mixed-media/options/style/anime/poster.jpg","values": {"style": "anime",- "details": "A young woman in a yellow summer dress walking toward the camera along a park path",+ "details": "A smiling Japanese woman in her twenties with a black ponytail and fair skin, in a white blouse and a light blue cardigan, walks toward the camera along a street lined with cherry blossom trees, petals in the air","negative": "photorealistic, photo, blurry, distorted","size": 544,"strength": 0.7},"seed": 42,- "seconds": 109.4,- "peak_vram_gb": 51.25,- "gpu": "NVIDIA H100 80GB HBM3, 81559 MiB",- "run": "20261023-143551-h100"+ "seconds": 109.6,+ "peak_vram_gb": 44.16,+ "gpu": "NVIDIA L40S, 46068 MiB",+ "run": "20261026-041209-l40s"},"claymation": {"case": "mm-claymation",@@ -4620,5 +4620,5 @@"run": "20261026-021906-l40s"}},- "updated_from": "20261026-021906-l40s"+ "updated_from": "20261026-041209-l40s"}diff --git a/web/examples/mixed-media/card.mp4 b/web/examples/mixed-media/card.mp4index 3f192cf..d92d75f 100644Binary files a/web/examples/mixed-media/card.mp4 and b/web/examples/mixed-media/card.mp4 differdiff --git a/web/examples/mixed-media/full.mp4 b/web/examples/mixed-media/full.mp4index 272cd34..7fc30d5 100644Binary files a/web/examples/mixed-media/full.mp4 and b/web/examples/mixed-media/full.mp4 differdiff --git a/web/examples/mixed-media/input-clip-full.mp4 b/web/examples/mixed-media/input-clip-full.mp4index 0635a3c..3af4a08 100644Binary files a/web/examples/mixed-media/input-clip-full.mp4 and b/web/examples/mixed-media/input-clip-full.mp4 differdiff --git a/web/examples/mixed-media/input-clip.jpg b/web/examples/mixed-media/input-clip.jpgindex d1cc7d7..3fafce7 100644Binary files a/web/examples/mixed-media/input-clip.jpg and b/web/examples/mixed-media/input-clip.jpg differdiff --git a/web/examples/mixed-media/options/style/anime/card.mp4 b/web/examples/mixed-media/options/style/anime/card.mp4index 3f192cf..d92d75f 100644Binary files a/web/examples/mixed-media/options/style/anime/card.mp4 and b/web/examples/mixed-media/options/style/anime/card.mp4 differdiff --git a/web/examples/mixed-media/options/style/anime/poster.jpg b/web/examples/mixed-media/options/style/anime/poster.jpgindex a4f91b8..c80f988 100644Binary files a/web/examples/mixed-media/options/style/anime/poster.jpg and b/web/examples/mixed-media/options/style/anime/poster.jpg differdiff --git a/web/examples/mixed-media/poster.jpg b/web/examples/mixed-media/poster.jpgindex a4f91b8..c80f988 100644Binary files a/web/examples/mixed-media/poster.jpg and b/web/examples/mixed-media/poster.jpg differ