Skip to content

Free, open-source 3D sculpting for Linux, macOS and Windows: dense meshes, remeshing, painting, built-in AI (image/text to 3D), Send to Unreal, game-ready LODs, Gaussian splats and OpenUSD.

Merge feat/sculptable

islandmed501cc6d2024a
diff --cc CHANGELOG.md
index ad584129,06263379..f205cee4
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -6,33 -6,29 +6,55 @@@ All notable changes to SculptForge are
## [Unreleased]
+ ### Make sculptable (splat capture -> sculptable, animation-ready mesh)
+ - Splats palette > **From scan > Make sculptable**: one click turns the capture into a new SubTool next to the
+ splats (which are hidden, not deleted): a watertight, manifold surface (opacity-weighted Gaussian density on a
+ voxel grid, built in parallel z slabs with the same result on any thread count; gaps closed; floater shells
+ dropped; space boxed in on 5 of 6 sides filled, which seals open bottoms; largest piece kept), QRemesh quads at
+ a **Game** (4,000), **Film** (12,000) or **Sculpt** (2,000 + 2 subdivision levels with the scan projected onto
+ the top) budget, and the capture's colours as polypaint or an Auto UV texture (splats inside the subject are
+ faded out). A hidden "<name> Scan" SubTool keeps the dense surface for ProjectAll.
+ - **Face loops**: a face is looked for on the scan from the current view (depth and colour, no ML model, no
+ download): eye corners, nose tip, mouth corners and chin become QRemesh Guides (rings round the eyes and the
+ orbits, the lips, the nasolabial loop), mirrored, with QRemesh symmetry. **Review** stops after the scan to
+ drag the landmarks (the mirror follows) or **Place by hand**, then **Finish**.
+ - **Crop Box** with draggable face handles in the viewport; options for Detail, Min Opacity, Close Gaps, Fill
+ holes, All pieces, Keep scan. Progress bar and Cancel; Ctrl-hover help; F3 entries.
+ - Commands: `splat_to_mesh` now runs the whole pipeline (`detail`, `preset`, `target`, `levels`, `face_guides`,
+ `landmarks`, `front`, `up`, `color`, `texture_size`, `crop`, `fill_holes`, `keep_parts`, `keep_scan`, `stage`
+ scan/finish, ...; `resolution` still works) and adds SubTools instead of replacing the sculpt; `face_guides`;
+ `splat_test_capture` (a built-in test head or mug capture, no download).
+ - 5 million splats to Film quads with face loops in about 9 s, 0.9 GB (M1 Pro).
+ - Guide chapter "From scan to sculptable mesh"; UI test `make_sculptable.sfui`; UI tests: `drag_landmark` verb,
+ `hover` scrolls a panel to the item as `click` does.
+
### Added
+- Facial shapes (Tool > Facial Shapes, `face_shapes_generate`): the 52 Apple ARKit blend shapes, exactly named, as
+ sculpt layers on the active SubTool, from a built-in template head (ICT-FaceKit's ICT Face Model Light, MIT):
+ automatic or landmark-guided fit (similarity + non-rigid ICP), deformation transfer (Sumner and Popovic 2004) so
+ the shapes follow the sculpt's proportions, mirrored Left/Right on symmetric sculpts, made on a low subdivision
+ level and carried up with the detail. Eye SubTools get eyeLook rotations, teeth and tongue the lower jaw's motion,
+ brows and lashes follow the skin. Face test presets and a slider per shape; `face_landmarks_detect`,
+ `face_preset`.
+- Live face: `face_weights json={...}` sets ARKit-named shape weights on every SubTool, re-evaluating only the
+ vertices they move (no undo step, partial GPU upload; about 9 ms for all 52 on a 700k-vertex head on an M1 Pro).
+- Face takes: play recorded face feeds (JSON frames, `face_take_play`), bake them (`face_take_bake`) into a glTF
+ animation of the morph target weights (GLB) and USD blend shape weight time samples; saved with the project.
+- `new_head`: a new scene with a life-size human head (with mouth and eye sockets) and its eyes.
+- Blender bridge: sculpt layers arrive as shape keys with their names and weights (a `.shapes` file beside each
+ .GoZ, which now holds the base shape when the mesh has layers).
+- UI tests: `zoom` and `look` steps; `run` expands `$SCRIPT_DIR`.
+- Splat training from captures (Splats > Train from capture): a phone video, a folder of photos, a phone-link
+ capture or a COLMAP / nerfstudio dataset becomes a Gaussian splat that appears in the viewport after the first
+ checkpoint and sharpens while it trains (worker thread; orbit freely). Progress over the canvas: step, PSNR,
+ elapsed and remaining time, Good enough - stop, Cancel. Presets Fast preview and Quality. Uses Brush 0.3.0
+ (Apache-2.0, WebGPU: NVIDIA, AMD, Intel, Apple), downloaded on request from its GitHub release with a pinned
+ SHA-256, or a path / `SCULPTFORGE_BRUSH`; COLMAP (BSD; downloaded on request on Windows) for camera positions,
+ skipped for captures with AR poses in `capture.json`; ffmpeg for video, keeping the sharpest frame of each
+ stretch. The result is centred and levelled from the camera positions, saved next to the capture and copied to
+ the LightBox Splats tab. Dropping a video onto the window trains it. Commands `splat_train` and
+ `splat_train_cancel` (blocking in sculptforge-cli, Python and MCP). Guide chapter "Scan an object into a splat";
+ a generated CC0 test capture (tools/splat/make_test_capture.py, tests/data/splat_capture).
- "Fixes & suggestions" in the LightBox header (shown at startup) and the Help menu: opens the bug report and
feature request forms on GitHub.
diff --cc CMakeLists.txt
index 4ac7b0c3,44e185df..a93de2ac
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@@ -52,7 -52,7 +52,8 @@@ add_library(sculptforge_core STATI
src/core/Image.cpp
src/core/ImageTo3D.cpp
src/core/Splats.cpp
+ src/core/SplatCapture.cpp
+ src/core/Sculptable.cpp
src/core/Voxel.cpp
src/core/EnginePresets.cpp
src/core/Masking.cpp
@@@ -202,7 -194,7 +203,8 @@@ if(SCULPTFORGE_BUILD_APP
src/app/Gizmo.cpp
src/app/PanelBake.cpp
src/app/PanelRetopo.cpp
+ src/app/PanelSplatTrain.cpp
+ src/app/PanelSculptable.cpp
src/app/PanelPerformance.cpp
src/app/PanelGoz.cpp
src/app/PanelBlender.cpp
diff --cc docs/guide/index.md
index 53e61c7a,cdcc66ce..b9c6f9a6
--- a/docs/guide/index.md
+++ b/docs/guide/index.md
@@@ -39,25 -38,25 +39,26 @@@ You can read it inside SculptForge (**H
## AI and capture
-20. [Image and text to 3D](generate-3d.md)
-21. [Gaussian splats](splats.md)
-22. [From scan to sculptable mesh](scan-to-sculptable.md)
+21. [Image and text to 3D](generate-3d.md)
+22. [Gaussian splats](splats.md)
+23. [Scan an object into a splat](scan-to-splat.md)
++24. [From scan to sculptable mesh](scan-to-sculptable.md)
## Files and pipelines
- 24. [Baking normal and displacement maps](baking.md)
- 25. [Exporting, Unreal and ZBrush](exporting.md)
- 26. [Working with Blender](blender.md)
- 27. [Projects, autosave and QuickSave](projects.md)
- 28. [Scripting and automation](scripting.md)
- 29. [SculptForge for studios](studios.md)
-23. [Baking normal and displacement maps](baking.md)
-24. [Exporting, Unreal and ZBrush](exporting.md)
-25. [Working with Blender](blender.md)
-26. [Projects, autosave and QuickSave](projects.md)
-27. [Scripting and automation](scripting.md)
-28. [SculptForge for studios](studios.md)
++25. [Baking normal and displacement maps](baking.md)
++26. [Exporting, Unreal and ZBrush](exporting.md)
++27. [Working with Blender](blender.md)
++28. [Projects, autosave and QuickSave](projects.md)
++29. [Scripting and automation](scripting.md)
++30. [SculptForge for studios](studios.md)
## Reference
- 30. [Keyboard and mouse](shortcuts.md)
- 31. [Performance and big meshes](performance.md)
- 32. [Troubleshooting and FAQ](troubleshooting.md)
- 33. [Glossary](glossary.md)
-29. [Keyboard and mouse](shortcuts.md)
-30. [Performance and big meshes](performance.md)
-31. [Troubleshooting and FAQ](troubleshooting.md)
-32. [Glossary](glossary.md)
++31. [Keyboard and mouse](shortcuts.md)
++32. [Performance and big meshes](performance.md)
++33. [Troubleshooting and FAQ](troubleshooting.md)
++34. [Glossary](glossary.md)
## Getting help
diff --cc docs/guide/whats-new.md
index 2f41b637,9555d5cb..0a23833d
--- a/docs/guide/whats-new.md
+++ b/docs/guide/whats-new.md
@@@ -7,22 -7,14 +7,30 @@@ chapter opens by itself the first time
## Next version (in development)
+### Facial shapes and live face
+
+**Tool > Facial Shapes > Generate Facial Shapes** gives a sculpted head the 52 Apple ARKit blend shapes
+(`eyeBlinkLeft`, `jawOpen`, `mouthSmileRight`, ...) as sculpt layers, fitted to its proportions from a template head
+with those shapes (ICT-FaceKit, MIT licence). Eyes, teeth and tongue SubTools get theirs too. Fix any shape by
+sculpting into its layer, check them with the **Face test** presets and sliders, and export them with their ARKit
+names to GLB, USD, FBX and Blender (as shape keys). `face_weights` drives them live from a face feed, and recorded
+takes play in the viewport and bake into a glTF animation or USD time samples. `new_head` starts a scene with a
+human head to sculpt on. See [Facial shapes and live face](facial-shapes.md).
+### Scan an object into a splat
+
+Film an object with your phone, or take photos all around it, and SculptForge trains a Gaussian splat from it on
+your own graphics card. The splat appears in the viewport after a few hundred steps and sharpens while it trains;
+you can orbit around it, stop when it's good enough, or cancel. Pick **Video...** or **Photos Folder...** under
+**Splats > Train from capture**, or drop a video onto the window. The free Brush trainer is offered as a one-click
+download the first time. See [Scan an object into a splat](scan-to-splat.md).
+ ### Make sculptable: from a scan to a production asset
+
+ One click in the **Splats** palette turns a Gaussian-splat capture into a clean SubTool you can sculpt, rig and
+ animate: a watertight surface with the floaters and background gone and open bottoms sealed, quad topology at a
+ Game, Film or Sculpt budget, edge loops round the eyes and mouth and along the nasolabial folds when the capture
+ is a head (mirrored across the face), and the capture's colours as polypaint or a texture. A crop box cuts away
+ the table or the room; Review shows the face landmarks to nudge before remeshing. The splats stay, hidden. Five
+ million splats take about ten seconds on a laptop. See [From scan to sculptable mesh](scan-to-sculptable.md).
## Version 0.8.1
diff --cc src/app/App.cpp
index e3695794,187424c1..16ecbe44
--- a/src/app/App.cpp
+++ b/src/app/App.cpp
@@@ -344,7 -344,7 +344,8 @@@ int App::runUiTest(int w, int h, int fp
pollAutoUv();
pollBake();
pollQRemesh();
+ pollSplatTrain();
+ pollSculptable();
updateCameraAspect();
Demo::beginFrame();
@@@ -502,7 -502,7 +503,8 @@@ int App::runRecording()
pollAutoUv();
pollBake();
pollQRemesh();
+ pollSplatTrain();
+ pollSculptable();
updateCameraAspect();
Demo::beginFrame();
@@@ -570,7 -570,7 +572,8 @@@ int App::run()
pollAutoUv();
pollBake();
pollQRemesh();
+ pollSplatTrain();
+ pollSculptable();
int fbW = 0, fbH = 0, winW = 0, winH = 0;
SDL_GetWindowSizeInPixels(window_, &fbW, &fbH);
@@@ -692,7 -692,7 +695,8 @@@ void App::onMouseDown(float x, float y
if (maskSelectMouseDown(x, y, mod)) return;
if (gizmoMouseDown(x, y, mod)) return; // Gizmo 3D handles, pivot clicks (Gizmo.cpp)
if (guideMouseDown(x, y, mod)) return; // the QRemesh Guides brush (PanelRetopo.cpp)
+ if (faceMouseDown(x, y, mod)) return; // placing facial landmarks (PanelFace.cpp)
+ if (sculptableMouseDown(x, y)) return; // Make sculptable: landmarks, crop box (PanelSculptable.cpp)
// Alt+click on another SubTool selects it (ZBrush); Alt+drag stays an inverted stroke or a pan.
// Shift+Alt+click selects and frames it (ZBrush), the active one too.
const bool frameClick = (mod & SDL_KMOD_ALT) && (mod & SDL_KMOD_SHIFT) && !ctrlDown(mod);
@@@ -1313,10 -1313,9 +1320,11 @@@ void App::drawUi()
const ImVec2 mouse(mouseX_, mouseY_);
drawGizmo(); // Gizmo 3D handles and panel (Gizmo.cpp)
drawQRemeshGuides(); // QRemesh Guides curves (PanelRetopo.cpp)
+ drawFaceOverlay(); // facial landmarks (PanelFace.cpp)
+ drawSplatTrainOverlay(); // splat training progress (PanelSplatTrain.cpp)
+ drawSculptableOverlay(); // Make sculptable: crop box, face landmarks (PanelSculptable.cpp)
// In Gizmo mode (and with the QRemesh Guides brush) the brush ring shows only while Ctrl masks or Shift smooths.
- const bool ringOff = gizmoActive() || (guidesMode_ && !(inputMods() & SDL_KMOD_SHIFT));
+ const bool ringOff = gizmoActive() || (guidesMode_ && !(inputMods() & SDL_KMOD_SHIFT)) || sculptableHot();
if (((!ImGui::GetIO().WantCaptureMouse && canvasRect().contains(mouse.x, mouse.y)) || drag_ == Drag::Stroke) &&
!hideBrushCursor_ && drag_ != Drag::Select && drag_ != Drag::Guide &&
(!ringOff || drag_ == Drag::Stroke || ctrlDown(inputMods())))
@@@ -2228,7 -2224,7 +2235,8 @@@ void App::drawPalette_Splats()
if (ImGui::Button("Save splats...", ImVec2(half, 0))) saveFileDialog(FileAction::SaveSplats);
ImGui::SameLine();
if (ImGui::Button("Clear##splats", ImVec2(half, 0))) runCommand("splat_clear");
+ drawSplatTrainSection();
+ drawSculptableSection(); // Make sculptable (PanelSculptable.cpp)
}
void App::drawBrushCursor() {
diff --cc src/app/Config.cpp
index a3b73a97,223d57ef..4c836a77
--- a/src/app/Config.cpp
+++ b/src/app/Config.cpp
@@@ -365,9 -365,8 +365,10 @@@ void App::initActions()
registerGizmoActions(); // Q W E R: Draw, and the Gizmo 3D's Move, Scale, Rotate (Gizmo.cpp)
registerBakeUiActions(); // Tool > Bake (PanelBake.cpp)
registerRetopoUiActions(); // Geometry > QRemesh, SubTool > ProjectAll (PanelRetopo.cpp)
+ registerSplatTrainUi(); // Splats > Train from capture (PanelSplatTrain.cpp)
+ registerSculptableUiActions(); // Splats > Make sculptable (PanelSculptable.cpp)
registerVdmActions(); // Brush > Vector Displacement (PanelVdm.cpp)
+ registerFaceUiActions(); // Tool > Facial Shapes, face_take_play / face_take_stop (PanelFace.cpp)
registerPerformanceUiActions(); // Preferences > Performance (PanelPerformance.cpp)
registerInputActions(); // ZBrush keys: U I O, Space, L, V, Ctrl+A / I / H / W..., the input style (Input.cpp)
diff --cc src/app/UiTest.cpp
index 252780a4,f755ef0c..316d3bdf
--- a/src/app/UiTest.cpp
+++ b/src/app/UiTest.cpp
@@@ -24,10 -24,10 +24,11 @@@
// move_view rotate_x rotate_y rotate_z rotate_view scale_x scale_y scale_z scale_uniform;
// keep: leave the button down (a shot mid-drag), then `release`
// gizmo_hover HANDLE move the mouse onto a Gizmo 3D handle (it lights up, for screenshots)
+ // drag_landmark N DX DY [S] left-drag Make sculptable's face landmark N (0-7, FaceLandmarks order) by DX DY pixels
// orbit DX DY right-drag on the background, in pixels (with the held modifiers)
// camera YAW PITCH set the view's angles in degrees (0 0 = the front view), e.g. for guide screenshots
-// run COMMANDS sculptforge-cli commands, e.g. run new_sphere subdivisions=4
+// zoom FACTOR / look X Y Z move the camera FACTOR times closer / turn it about (and look at) the point X Y Z
+// run COMMANDS sculptforge-cli commands, e.g. run new_sphere subdivisions=4 ($SCRIPT_DIR: the script's folder)
// drop FILE as if FILE were dropped onto the window (relative to the script's folder)
// env NAME VALUE set an environment variable; $SCRIPT_DIR is the script's folder
// copy SRC DST copy a file or folder (SRC relative to the script's folder, DST to the working
@@@ -86,8 -85,9 +87,11 @@@
// center_x / center_y / center_z and size_x / size_y / size_z (active mesh bounds)
// bake_running, baked_maps (maps in the last bake), bake_size, bake_files (files it wrote),
// normal_map_preview (1 when the active SubTool shows its baked normal map)
+// splat_training (1 while a splat trains), splat_previews (checkpoints shown), splat_train_step,
+// splat_train_stage (4 done, 5 failed, 6 cancelled), splat_train_psnr, splat_train_prompt
+ // sculptable_running, sculptable_review, sculptable_landmarks (8 when placed),
+ // sculptable_placing (next landmark to click, -1), sculptable_crop, sculptable_crop_height,
+ // show_splats, landmark_x_N / landmark_y_N / landmark_z_N (face landmark N, mesh space)
// (mesh values are the active SubTool's)
// snapshot remember the active mesh's vertices (for the moved* values)
// remember VALUE NAME keep VALUE's current value as $NAME
@@@ -264,7 -264,7 +268,8 @@@ double Demo::query(const std::string& n
if (name == "active_texture_layer") return hasTexture(m) && !m.texture->layers.empty() ? double(m.texture->activeLayer) : -1.0;
if (name == "auto_uv_running") return app_.autoUv_ ? 1.0 : 0.0;
if (const double v = app_.bakeTestValue(name); !std::isnan(v)) return v;
+ if (const double v = app_.splatTrainTestValue(name); !std::isnan(v)) return v; // splat_training, ...
+ if (const double v = app_.sculptableTestValue(name); !std::isnan(v)) return v;
if (name == "qremesh_running") return app_.qremesh_ ? 1.0 : 0.0;
if (name == "qremesh_guides") return double(app_.doc_.qremeshGuides.size());
if (name == "qremesh_guide_loops") {
@@@ -867,7 -882,7 +910,7 @@@ bool Demo::parseStep(int line, const st
!busy(app_.pendingUnreal_) && !busy(app_.pendingComfyCheck_) && !app_.galleryBusy() &&
!app_.thumbs_.busy() && !app_.libraryScanner_.busy() && !app_.autoUv_ &&
!app_.bakeBusy() && !app_.qremesh_ && !app_.voxelRemesh_ && !app_.updateJob_ &&
- !app_.splatTrainBusy();
- !app_.sculptableBusy();
++ !app_.splatTrainBusy() && !app_.sculptableBusy();
*idleFrames = idle ? *idleFrames + 1 : 0;
if (*idleFrames >= 4) return true;
if (idle) return false;