Skip to content

Photoreal Blender scenes from a one-paragraph brief, built by AI agents: image→3D props on fal, scripted layout, lighting, camera and grade.

Python
8.6 KB
Raw Blame

reference-to-scene

A fal Serverless app that turns one or more reference images into a lit, dressed Blender scene and returns it as a package. Two calls, with an optional approval stop between them:

POST /plan    reference images (+ optional floor plan, prompt)  ->  plan_id, floor plan, grey-box renders, prop list
POST /plan    plan_id + edit (text or JSON)                     ->  revised plan (repeat as needed)
POST /build   plan_id (or reference images + auto_approve)       ->  package zip + scene.blend + props + renders

Both run on the fal queue and stream their progress as stage events (see Progress). The app lives in endpoint/; final renders go to the blender-farm app (farm/app.py, synced from AnyShot). A third call, /scene-to-scene, changes a built scene's dials (see Dials) and re-renders it.

To shoot a finished build (stills or camera moves described in words), use AnyShot: the builder's /scene2image and /scene2video forward to it.

Dials

Ten optional controls, each changing one aspect of the scene and meaningful for any kind of place (a diner, a forest camp, a spaceship bridge). Unset dials are read from the references and prompt. The plan records the value each dial resolved to and what it means for this scene, so a caller can see "wear 0.62" and nudge it.

Group Dial Range Changes (and only this) Cost to change
Stuff density 0 - 1 how much is in the space: items per surface and per m² placement pass + new props
tidiness 0 messy - 1 tidy how things are arranged, not how many: scattered, tipped, piled vs aligned, stacked, put away placement pass
Surfaces wear 0 pristine - 1 decayed age and condition: dirt, scratches, stains, rust, peeling, moss re-render
wetness 0 dry - 1 soaked water on surfaces, puddles re-render
Environment time_of_day dawn, day, dusk, night sky, sun or moon, daylight through openings re-render
weather clear, overcast, rain, fog, snow sky and falling particles re-render
atmosphere 0 clear - 1 thick particles in the air: dust, haze, smoke, steam re-render
Energy power 0 dormant - 1 live whether the place is switched on: screens, fires, machines, practical lights, steam re-render
Look warmth -1 cold - +1 warm colour temperature of the lights and grade re-render
look clean, cinematic, gritty, vintage grade, grain, lens effects re-render

The dials are independent. Two couplings are defaults only: weather: rain raises wetness unless it is set, and time_of_day: night lowers ambient light while power decides what is lit. "Peak service" is roughly density 0.8, tidiness 0.3, power 1; the same place "after close" is tidiness 0.9, power 0.15.

In the plan each dial carries its value and its meaning for this scene:

"dials": {
  "wear":     { "value": 0.65, "means": "grease film on steel, chipped green paint on the cart, soot above the burners" },
  "power":    { "value": 0.9,  "means": "wok burners lit, bare bulbs and neon on, steam from the stock pots" },
  "tidiness": { "value": 0.4,  "means": "stools scattered off-square, newspapers left on tables, crates stacked unevenly" }
}

The build agent builds every scene as a rig: one control per dial (a strength input on the shared grime group, a wetness mask, an atmosphere density, named light and emitter groups for power, and so on), recorded in the package's dials.json with an apply_dials.py that sets them. /scene-to-scene diffs the requested dials against the scene's: re-render-only dials skip planning and generation; density and tidiness run a placement pass first.

POST /scene-to-scene
{ "scene_id": "s_…", "dials": { "wetness": 0.0, "time_of_day": "dawn" },   // the dry morning version
  "preview": "wear",           // optional: one still at three values of this dial before committing
  "renders": ["stills"] }      // or the shots too

Coordinates

Metres, Z up. The origin is the floor at the room's front-left corner, seen from the street/entrance: X runs right along the front wall, Y runs into the room, Z up. Walls are named front (y = 0), back (y = depth), left (x = 0) and right (x = width). A rotation of 0 means the object's front faces -Y (towards the entrance).

Plan

{
  "plan_id": "p_7f3a2c",
  "name": "hk-dai-pai-dong",
  "summary": "Open-air street-food stall under a tarp awning, late at night after rain",
  "setting": "semi_outdoor",                     // interior | semi_outdoor | exterior
  "time_of_day": "night", "mood": ["humid", "busy", "neon"], "palette": ["#2b3a2f", "#c0392b", "#f5b041"],
  "room": { "width": 6.0, "depth": 8.0, "height": 3.2 },
  "openings": [ { "wall": "front", "kind": "shopfront", "center": 3.0,   // center = X on front/back walls, Y on left/right
                  "width": 5.0, "height": 2.6, "sill": 0.0 } ],
  "objects": [
    { "id": "kitchen_cart", "label": "green steel kitchen cart", "category": "furniture",
      "x": 1.1, "y": 5.5, "z": 0.0, "w": 2.2, "d": 0.8, "h": 1.0, "rot": 90,
      "source": "reference", "generate": true,
      "prompt": "battered green-painted steel street-food kitchen cart with two wok burners",
      "crop": [0.04, 0.18, 0.46, 0.66] }          // where it is in reference image 0, normalised x0,y0,x1,y1
  ],
  "lights": [ { "id": "bulb_1", "kind": "practical", "x": 1.0, "y": 5.0, "z": 2.4, "color": "#ffb46b", "power": "medium" } ],
  "reference_camera": { "position": [3.0, -2.5, 1.6], "look_at": [2.6, 5.0, 1.0], "lens_mm": 28 },
  "notes": ["The area behind the reference camera is invented: a second row of tables and a drinks fridge."]
}

Every object and light carries source: reference (visible in a reference image) or invented (added to complete the space). The floor plan draws invented items dashed, so a reviewer sees at a glance what was extrapolated.

/plan

Input

field type
reference_image_urls string[] (1-4) the look: mood, materials, props
floor_plan_url string, optional a sketch, photo or drawing of the layout; overrides inferred room shape and placement
prompt string, optional anything the images can't show ("rain outside", "it's 1 AM", "make it twice as deep")
plan_id + edit string + string revise an existing plan from a text instruction
plan_id + plan string + object replace an existing plan with an edited JSON version

Output

field
plan_id id for /build or another /plan revision
plan the Plan JSON above
floor_plan top-down drawing (PNG): walls, openings, every footprint labelled, lights, reference camera
blockout grey-box renders (PNG): one from the reference camera, one high three-quarter overview
comparison reference and blockout side by side (PNG)

/build

Input: plan_id, or the /plan inputs plus auto_approve: true; tier (draft | full); num_stills (default 3); walkthrough (bool); resolution (default [1920, 804]).

Output: package (zip), scene_blend, props (id, mesh, preview, reference crop, model), stills, walkthrough, manifest, timings, cost.

Package layout

scene_package.zip
  scene.blend              packed textures, lights, cameras; opens standalone
  props/<id>.glb           cleaned, real-scale, front -Y, bottom-centre origin
  props/<id>_ref.png       the reference each prop was generated from
  renders/still_*.png      Cycles stills, including one from the reference camera
  renders/walkthrough.mp4  handheld walk-through (if requested)
  plan.json, manifest.json props + picks + reasons, lights, shots, costs, timings
  README.md, LICENSE

Progress

Each request streams stage events through the fal queue logs, one JSON object per line prefixed ANYSCENE_STAGE :

{"stage": "generating_meshes", "status": "running", "message": "41 of 80 meshes done", "progress": 0.51,
 "preview_url": "https://..."}

Stages, in order:

/plan /build
received received
reading_reference prop_references (crop + clean each prop from the reference)
planning generating_meshes (all submitted at once)
drawing_floor_plan layout (shell + stand-ins, runs during mesh generation)
blockout dressing (meshes swapped in, collision check)
done materials (textures, grime, decals)
lighting (practicals, atmosphere, critic pass vs the reference)
rendering (farm: stills + walkthrough)
packaging
done

A failed stage emits "status": "failed" with a message, and the request fails with the same message.