Photoreal Blender scenes from a one-paragraph brief, built by AI agents: image→3D props on fal, scripted layout, lighting, camera and grade.
TOGGLE.md: an unreal option on /ref2scene, and /scene2unreal
Design note. Nothing here is implemented yet. It builds on unreal/export_blend.py, unreal/build_level.py,
unreal/render_mrq.ps1, unreal/perf.ps1 and unreal/NIAGARA.md, and on builder/app.py
(SceneBuilderApp, BuildRequest, _build_and_shoot).
1. API
1a. Toggle on /ref2scene (and its /build alias)
Add one optional field to BuildRequest. Like shots, it stays out of the build fingerprint (fp in _build),
so one Blender build serves any number of Unreal conversions.
class UnrealOptions(BaseModel):
enabled: bool = Field(True, description="Convert the passed build into an Unreal 5.8 level.")
cameras: list[str] = Field(default_factory=list, description="Cameras to render MRQ stills from (default: every scouting camera in shots.json).")
fx: bool = Field(True, description="Port effects (flames, steam, rain, ...) per NIAGARA.md. False: static fog only.")
look_match_rounds: int = Field(2, description="Agent look-match passes against the Cycles frames (0 = deterministic conversion only).")
package: str = Field("plugin", description="plugin: a UE content plugin zip (drop into any 5.8 project). uproject: a full project zip. both.")
still_res: list[int] = Field([1920, 1080], description="MRQ still resolution (W, H).")
clip_frames: int = Field(0, description="Also render N-frame MRQ clips per camera (0 = stills only).")
perf: bool = Field(True, description="Run the real-time perf capture (CSV profiler, GPU memory) at perf_res.")
perf_res: list[int] = Field([2560, 1440])
gpu_budget_ms: float = Field(16.6, description="Real-time frame budget the perf report grades against.")
class BuildRequest(BaseModel):
...
unreal: UnrealOptions | None = Field(None, description="Optional: once the build passes its critic, convert it to Unreal.")
Add to BuildResponse: unreal_job: dict | None, holding status, request_id and an error if any, with the same
shape as shots_job. Add unreal: UnrealResult | None (§3).
1b. Standalone /scene2unreal
SceneUnrealRequest = {build_id: str, **UnrealOptions, fresh: bool = False, effort: str = "medium"}.
- It runs on any finished build in
/data/anyscene/builder/<build_id>. - It is retry-safe the same way
_buildis: an index keyed bysha256(build_id + options)and an.aliveheartbeat. /ref2scenewithunrealset calls it after_build, exactly like_anyshot.- Run it as a separate fal app (
anyscene-unreal) on a UE container. The builder image stays Blender-only, and the UE image (about 40 GB with DDC) only starts when someone asks for Unreal. The builder forwards the order withfal_clientand relays the stage lines, ascall_anyshotdoes.
2. What runs after the critic passes
The gate is the same as for shots: res.critiques[-1]["passed"] and res.scene_blend_url.
| # | Stage | Where | Deterministic? |
|---|---|---|---|
| 1 | Export: blender -b <copy> --python export_blend.py -- --out exp/ --gpu --grade-sh ... --frame <hero frame>, sharded (§5), then a merge pass writes scene.json incl. fx (NIAGARA.md §4) and the flipbook / VDB bakes the FX plan asks for |
builder GPU worker (has Blender + OptiX cache) or the UE worker if Blender is installed there | yes |
| 2 | FX plan: fx_plan.py fills technique / template / params per fx entry from the NIAGARA.md decision table |
either | yes (rules); entries with confidence < 0.6 are queued for the agent |
| 3 | UE build: UnrealEditor-Cmd AnySceneUE.uproject -run=pythonscript -script=build_level.py (stages textures, meshes, materials, level, fx, sequences) on a Linux GPU container with the AnySceneFX plugin preinstalled and a persistent DDC on /data |
UE worker | yes |
| 4 | Shader warm: one -game -RenderOffscreen run of the level with fx.Niagara.ForceWaitForCompilationOnActivate=1 and Editor.AsyncAssetCompilation=0 to fill the DDC (skipped when DDC is warm) |
UE worker | yes |
| 5 | MRQ stills per camera (MRQ_Still + MRQ_Linear), plus clips if requested |
UE worker | yes, given a fixed seed and warm-up |
| 6 | Look-match pass (agent): compare against the Cycles stills (side_by_side.py, crops per fx bounds, exposure, fog), adjust parameters only (exposure offset, fog density, FX user params, light scale) with the NIAGARA.md §6 rules, rebuild only the level/fx stages, re-render; look_match_rounds times |
UE worker + Claude | agent-judged, logged as a param diff |
| 7 | Perf: perf.ps1-equivalent -game -RenderOffscreen with csvprofile, nvidia-smi sampling |
UE worker | measured |
| 8 | Package: Plugins/AnyScene_<build>/Content/... (+ .uplugin) zip; optionally the whole .uproject minus Intermediate/, Saved/, DerivedDataCache/ |
UE worker | yes |
Linux notes.
- The Linux editor renders through Vulkan (SM6).
-RenderOffscreenworks without a display. Install the Vulkan ICD from the NVIDIA container toolkit (NVIDIA_DRIVER_CAPABILITIES=all, as the builder Dockerfile already sets). - Lumen with hardware ray tracing on Vulkan needs a recent driver. Fall back to
r.Lumen.HardwareRayTracing 0and record which mode the render used. - Swap the Windows
.ps1wrappers for Pythonsubprocesscallers.render_mrq.ps1andperf.ps1are thin. - Validate on Linux once against the Windows reference box before trusting look-match numbers. Vulkan vs D3D12 shading differences are small but non-zero. [untested]
3. Outputs (UnrealResult)
{
"package_url": "...AnyScene_b_xxxx_plugin.zip",
"uproject_url": "... (optional)",
"stills": [{"camera": "cam_street", "ue": "...png", "ue_linear": "...exr", "cycles": "...png", "side_by_side": "...png"}],
"clips": [],
"look_match": {"rounds": 2, "changes": [{"round": 1, "target": "fx:steam_wok_0", "param": "Opacity", "from": 0.12, "to": 0.18, "why": "transmission 0.71 vs 0.58"}]},
"fx": [{"id": "steam_wok_0", "type": "steam", "technique": "niagara", "template": "/AnySceneFX/Systems/NS_Steam", "status": "ok"}],
"perf": {"res": [2560, 1440], "gpu_ms_p50": 0, "gpu_ms_p95": 0, "frame_ms_p95": 0, "gpu_mem_delta_mb": 0, "draw_calls": 0, "verdict": "within budget|over budget"},
"build_report": "...build_report.json",
"warnings": [],
"minutes": {"export": 0, "ue_build": 0, "mrq": 0, "look_match": 0, "total": 0}
}
4. Time and cost (from the HK run: 627 meshes)
| Stage | HK today (serial) | With §5 sharding | Notes |
|---|---|---|---|
| Blender export (bake + GLB) | ~35 min | ~6-9 min | about 3.3 s per unique mesh; scales with unique meshes, not instances |
| FX bakes (flipbooks / VDB) | n/a | +2-6 min | only for niagara_flipbook / svt entries; runs in parallel with the mesh shards |
| UE build (import + level) | ~15 min | ~10-15 min | import-bound; Interchange is already batched by 16; FX stage adds < 1 min |
| Shader warm (cold DDC) | ~3-5 min | 0-5 min | ~0 once /data DDC is warm for the templates and master materials |
| MRQ stills | ~1 min/still | ~1 min/still | 3 cameras x (1 + rounds) with rounds = 2 → 9 stills ≈ 9 min (or ~45 s each when batched in one process instead of one per camera) |
| Look-match agent | n/a | ~3-5 min/round of thinking + rebuild (level,fx stages ≈ 1-2 min) |
|
| Perf capture | n/a | ~2 min | 900 frames + startup |
| Total | ~55 min + stills | ~30-40 min |
Cost = GPU-minutes x the fal machine rate, plus agent tokens.
- GPU time is about 35 GPU-minutes on the parallel path.
- The export shards are CPU-heavy. Bakes can use
--gpu(OptiX) on the builder's own GPU. Run the export on the builder worker that already holds the scene, and hand onlyexp/(GLBs + PNGs, roughly 1-3 GB for HK) to the UE app through/data. - Agent tokens apply only to look-match rounds and to low-confidence FX entries: roughly one Claude session of about 15-30 tool calls per round.
- With
look_match_rounds = 0the job is purely deterministic and costs only GPU time.
5. Parallelizing the slow export
The export is embarrassingly parallel per unique mesh. Bakes are independent, and existing outputs are skipped (rerunnable).
- Plan pass. One quick Blender process classifies the objects and writes
export_plan.json: the list of asset keys with estimated bake pixels (area x density). It does no baking. - Shards. Run N Blender processes on copies of the
.blend, each started with--shard i/N. This is a new flag: keepkeys[k]wherek % N == i, after sorting by estimated cost and dealing round-robin, so shards balance. Each shard bakes into the sharedtex/andmesh/directories.- Today's
--reverseand--onlyalready show that two processes can share the folder safely. - Pick N = min(CPU cores ÷ 4, 8) for CPU bakes. With
--gpu, run 2-3 processes per GPU, since OptiX bakes are short and the time goes into Blender overhead.
- Today's
- Merge pass. One process with
--no-bake. Every texture and GLB already exists, so it only writesscene.json, lights, cameras, world, grade andfx. This also removes the partial-scene.jsonmerge (ONLY/LIMITbranch) from the hot path. - Cache across builds. Key
tex/andmesh/by a hash of (evaluated mesh data + material node trees + bake settings). The same prop library recurs across scenes, and a retry of/scene2unrealre-bakes nothing. - FX bakes (flipbook renders, VDB sampling) are extra shards, scheduled first because they are the longest single jobs.
The UE side can overlap with the export. Start texture import of finished shards while the others are still baking:
the textures stage is rerunnable and skips assets that exist. Mesh import order does not matter.
6. Deterministic vs agent-judged
Deterministic. Same input gives the same output, so these steps are cacheable and testable.
- Everything in
export_blend.py, and the fx classification scores. fx_plan.pytechnique and parameter mapping (NIAGARA.md §3-5).build_level.py.- Seeds:
RandomSeedOffset = crc32(fx id). - Warm-up counts.
- MRQ settings.
- Packaging.
- Perf numbers (measured, so noisy within ±5%).
Agent-judged. Logged as explicit parameter diffs, bounded by look_match_rounds.
- Low-confidence fx classification (< 0.6), including
unknownentries. - Switching technique when a parameter-only match fails after three rounds (NIAGARA.md §6.5).
- Exposure and fog calibration offsets.
- FX user-parameter tuning.
- Light scale tweaks.
- The final "does it read like the Cycles frame" verdict, which goes into the response next to the side-by-sides.
Rule. The agent never edits build_level.py, export_blend.py or templates during a job. It writes overrides to
exp/overrides.json, which build_level.py applies on top of scene.json. A rerun with the same overrides
reproduces the agent's result exactly.