Skip to content

Free, open-source 3D sculpting for Linux, macOS and Windows: dense meshes, remeshing, painting, built-in AI (image/text to 3D), Send to Unreal, game-ready LODs, Gaussian splats and OpenUSD.

Merge feat/splat

islandmedff49acfeef49
diff --cc CHANGELOG.md
index 7155c90c,ac56734d..ad584129
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -7,21 -7,17 +7,32 @@@ All notable changes to SculptForge are
## [Unreleased]
### Added
+- Facial shapes (Tool > Facial Shapes, `face_shapes_generate`): the 52 Apple ARKit blend shapes, exactly named, as
+ sculpt layers on the active SubTool, from a built-in template head (ICT-FaceKit's ICT Face Model Light, MIT):
+ automatic or landmark-guided fit (similarity + non-rigid ICP), deformation transfer (Sumner and Popovic 2004) so
+ the shapes follow the sculpt's proportions, mirrored Left/Right on symmetric sculpts, made on a low subdivision
+ level and carried up with the detail. Eye SubTools get eyeLook rotations, teeth and tongue the lower jaw's motion,
+ brows and lashes follow the skin. Face test presets and a slider per shape; `face_landmarks_detect`,
+ `face_preset`.
+- Live face: `face_weights json={...}` sets ARKit-named shape weights on every SubTool, re-evaluating only the
+ vertices they move (no undo step, partial GPU upload; about 9 ms for all 52 on a 700k-vertex head on an M1 Pro).
+- Face takes: play recorded face feeds (JSON frames, `face_take_play`), bake them (`face_take_bake`) into a glTF
+ animation of the morph target weights (GLB) and USD blend shape weight time samples; saved with the project.
+- `new_head`: a new scene with a life-size human head (with mouth and eye sockets) and its eyes.
+- Blender bridge: sculpt layers arrive as shape keys with their names and weights (a `.shapes` file beside each
+ .GoZ, which now holds the base shape when the mesh has layers).
+- UI tests: `zoom` and `look` steps; `run` expands `$SCRIPT_DIR`.
+ - Splat training from captures (Splats > Train from capture): a phone video, a folder of photos, a phone-link
+ capture or a COLMAP / nerfstudio dataset becomes a Gaussian splat that appears in the viewport after the first
+ checkpoint and sharpens while it trains (worker thread; orbit freely). Progress over the canvas: step, PSNR,
+ elapsed and remaining time, Good enough - stop, Cancel. Presets Fast preview and Quality. Uses Brush 0.3.0
+ (Apache-2.0, WebGPU: NVIDIA, AMD, Intel, Apple), downloaded on request from its GitHub release with a pinned
+ SHA-256, or a path / `SCULPTFORGE_BRUSH`; COLMAP (BSD; downloaded on request on Windows) for camera positions,
+ skipped for captures with AR poses in `capture.json`; ffmpeg for video, keeping the sharpest frame of each
+ stretch. The result is centred and levelled from the camera positions, saved next to the capture and copied to
+ the LightBox Splats tab. Dropping a video onto the window trains it. Commands `splat_train` and
+ `splat_train_cancel` (blocking in sculptforge-cli, Python and MCP). Guide chapter "Scan an object into a splat";
+ a generated CC0 test capture (tools/splat/make_test_capture.py, tests/data/splat_capture).
- "Fixes & suggestions" in the LightBox header (shown at startup) and the Help menu: opens the bug report and
feature request forms on GitHub.
diff --cc ROADMAP.md
index f4e3e150,4fe66fd7..94f88e52
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@@ -147,14 -147,11 +147,17 @@@ Ranked by what sculptors cite most on t
Nostril, Create VDM from a VDM plane, Blender-compatible OpenEXR in and out
- ✅ Blender add-on: ZBrush-style keys in Sculpt Mode (a preference)
-## In development (v0.9) — maquette to performance
-
+## In development for v0.9 — maquette to performance
+
+- ✅ Facial shapes: the 52 ARKit blend shapes generated on any sculpted humanoid head (template fit + deformation
+ transfer, eyes / teeth / tongue SubTools, symmetry, subdivision levels), Face test, export with ARKit names to
+ GLB / USD / FBX / Blender shape keys
+- ✅ Live face: `face_weights` (partial updates, no undo), takes played and baked to glTF animation / USD time samples
+- 📋 Next for faces: FBX blend-shape animation, take animation over the Blender bridge, an image-based landmark
+ detector, corrective combination shapes, eye rotations as bones for rigs that want them
+ - ✅ Splat training from a phone video or photos with a live viewport preview (Brush, downloaded on request;
+ COLMAP camera positions, or the phone's AR poses; sharpest-frame selection; Good enough - stop / Cancel)
+ - 📋 Splat training: background masking of the capture, resume a stopped training, a Brush build for Linux ARM
## Phase 3 — AI assistant, beyond brushes
diff --cc THIRD_PARTY_NOTICES.md
index 7f01c6fd,07653280..30005743
--- a/THIRD_PARTY_NOTICES.md
+++ b/THIRD_PARTY_NOTICES.md
@@@ -17,9 -17,20 +17,21 @@@ licenses require
| [tinyexr](https://github.com/syoyo/tinyexr) | 1.0.13 | BSD 3-Clause License (includes OpenEXR code under the OpenEXR BSD licence) | OpenEXR reading and writing for vector displacement maps (Brush > Vector Displacement). Vendored in third_party/tinyexr; uses stb's zlib. |
| [Instant Meshes](https://github.com/wjakob/instant-meshes) | formulas only, nothing vendored | BSD 3-Clause License | QRemesh (Tool > Geometry): the field formulas in src/core/QRemesh.cpp (extrinsic 4-RoSy orientation / position compatibility, the middle point of two tangent planes, lattice rounding) follow its reference code. The rest of QRemesh is our own. |
| [Inter typeface](https://github.com/rsms/inter) | 4.1 | SIL Open Font License 1.1 | User interface font, embedded in the desktop app. Vendored in assets/fonts. |
+| [ICT-FaceKit](https://github.com/ICT-VGL/ICT-FaceKit) (ICT Face Model Light) | commit on master, 2020 | MIT License | The template head of Generate Facial Shapes (Tool > Facial Shapes): its neutral skin, mouth and eye sockets and its 53 ARKit-style expression shapes, converted by tools/face/build_ict_template.py into assets/face/ict-arkit.sfface (embedded). Not the separately licensed full ICT Face Model. |
| [libcurl](https://curl.se) | system library | curl License (MIT/X derivate) | HTTPS for optional AI features on Linux and macOS. Linked dynamically from the operating system; Windows uses the built-in WinHTTP instead. |
+ Tools that are **not** included but that SculptForge can download for you on request (it says what, from where
+ and how big, and checks a pinned SHA-256 before use) and runs as separate programs, for splat training
+ (Splats > Train from capture):
+
+ | Tool | Version | License | Downloaded from |
+ |---|---|---|---|
+ | [Brush](https://github.com/ArthurBrussee/brush) | 0.3.0 | Apache License 2.0 | Its GitHub release (Windows x64, Linux x64, macOS arm64) |
+ | [COLMAP](https://github.com/colmap/colmap) | 4.2.1 | BSD 3-Clause License | Its GitHub release (Windows x64, no CUDA); elsewhere the system's package |
+
+ ffmpeg, used to read video frames, is never downloaded: SculptForge uses the one on the system, if any. The test
+ capture in tests/data/splat_capture is generated by tools/splat/make_test_capture.py and is CC0.
+
SculptForge does not include, bundle or redistribute any AI model. The optional AI features connect
to services you choose (see [docs/ai-services.md](docs/ai-services.md)), which have their own terms.
diff --cc docs/guide/index.md
index 3dae46ec,7546a6b9..53e61c7a
--- a/docs/guide/index.md
+++ b/docs/guide/index.md
@@@ -39,24 -38,25 +39,25 @@@ You can read it inside SculptForge (**H
## AI and capture
-20. [Image and text to 3D](generate-3d.md)
-21. [Gaussian splats](splats.md)
-22. [Scan an object into a splat](scan-to-splat.md)
+21. [Image and text to 3D](generate-3d.md)
+22. [Gaussian splats](splats.md)
++23. [Scan an object into a splat](scan-to-splat.md)
## Files and pipelines
- 23. [Baking normal and displacement maps](baking.md)
- 24. [Exporting, Unreal and ZBrush](exporting.md)
- 25. [Working with Blender](blender.md)
- 26. [Projects, autosave and QuickSave](projects.md)
- 27. [Scripting and automation](scripting.md)
- 28. [SculptForge for studios](studios.md)
-22. [Baking normal and displacement maps](baking.md)
-23. [Exporting, Unreal and ZBrush](exporting.md)
-24. [Working with Blender](blender.md)
-25. [Projects, autosave and QuickSave](projects.md)
-26. [Scripting and automation](scripting.md)
-27. [SculptForge for studios](studios.md)
++24. [Baking normal and displacement maps](baking.md)
++25. [Exporting, Unreal and ZBrush](exporting.md)
++26. [Working with Blender](blender.md)
++27. [Projects, autosave and QuickSave](projects.md)
++28. [Scripting and automation](scripting.md)
++29. [SculptForge for studios](studios.md)
## Reference
- 29. [Keyboard and mouse](shortcuts.md)
- 30. [Performance and big meshes](performance.md)
- 31. [Troubleshooting and FAQ](troubleshooting.md)
- 32. [Glossary](glossary.md)
-28. [Keyboard and mouse](shortcuts.md)
-29. [Performance and big meshes](performance.md)
-30. [Troubleshooting and FAQ](troubleshooting.md)
-31. [Glossary](glossary.md)
++30. [Keyboard and mouse](shortcuts.md)
++31. [Performance and big meshes](performance.md)
++32. [Troubleshooting and FAQ](troubleshooting.md)
++33. [Glossary](glossary.md)
## Getting help
diff --cc docs/guide/whats-new.md
index 76b873d9,a9a12e5b..2f41b637
--- a/docs/guide/whats-new.md
+++ b/docs/guide/whats-new.md
@@@ -7,15 -7,13 +7,22 @@@ chapter opens by itself the first time
## Next version (in development)
+### Facial shapes and live face
+
+**Tool > Facial Shapes > Generate Facial Shapes** gives a sculpted head the 52 Apple ARKit blend shapes
+(`eyeBlinkLeft`, `jawOpen`, `mouthSmileRight`, ...) as sculpt layers, fitted to its proportions from a template head
+with those shapes (ICT-FaceKit, MIT licence). Eyes, teeth and tongue SubTools get theirs too. Fix any shape by
+sculpting into its layer, check them with the **Face test** presets and sliders, and export them with their ARKit
+names to GLB, USD, FBX and Blender (as shape keys). `face_weights` drives them live from a face feed, and recorded
+takes play in the viewport and bake into a glTF animation or USD time samples. `new_head` starts a scene with a
+human head to sculpt on. See [Facial shapes and live face](facial-shapes.md).
+ ### Scan an object into a splat
+
+ Film an object with your phone, or take photos all around it, and SculptForge trains a Gaussian splat from it on
+ your own graphics card. The splat appears in the viewport after a few hundred steps and sharpens while it trains;
+ you can orbit around it, stop when it's good enough, or cancel. Pick **Video...** or **Photos Folder...** under
+ **Splats > Train from capture**, or drop a video onto the window. The free Brush trainer is offered as a one-click
+ download the first time. See [Scan an object into a splat](scan-to-splat.md).
## Version 0.8.1
diff --cc src/app/App.cpp
index bd159c8b,29c40d43..e3695794
--- a/src/app/App.cpp
+++ b/src/app/App.cpp
@@@ -1310,7 -1309,7 +1313,8 @@@ void App::drawUi()
const ImVec2 mouse(mouseX_, mouseY_);
drawGizmo(); // Gizmo 3D handles and panel (Gizmo.cpp)
drawQRemeshGuides(); // QRemesh Guides curves (PanelRetopo.cpp)
+ drawFaceOverlay(); // facial landmarks (PanelFace.cpp)
+ drawSplatTrainOverlay(); // splat training progress (PanelSplatTrain.cpp)
// In Gizmo mode (and with the QRemesh Guides brush) the brush ring shows only while Ctrl masks or Shift smooths.
const bool ringOff = gizmoActive() || (guidesMode_ && !(inputMods() & SDL_KMOD_SHIFT));
if (((!ImGui::GetIO().WantCaptureMouse && canvasRect().contains(mouse.x, mouse.y)) || drag_ == Drag::Stroke) &&
diff --cc src/app/Config.cpp
index 9cc3638e,d1b08c8c..a3b73a97
--- a/src/app/Config.cpp
+++ b/src/app/Config.cpp
@@@ -365,8 -365,8 +365,9 @@@ void App::initActions()
registerGizmoActions(); // Q W E R: Draw, and the Gizmo 3D's Move, Scale, Rotate (Gizmo.cpp)
registerBakeUiActions(); // Tool > Bake (PanelBake.cpp)
registerRetopoUiActions(); // Geometry > QRemesh, SubTool > ProjectAll (PanelRetopo.cpp)
+ registerSplatTrainUi(); // Splats > Train from capture (PanelSplatTrain.cpp)
registerVdmActions(); // Brush > Vector Displacement (PanelVdm.cpp)
+ registerFaceUiActions(); // Tool > Facial Shapes, face_take_play / face_take_stop (PanelFace.cpp)
registerPerformanceUiActions(); // Preferences > Performance (PanelPerformance.cpp)
registerInputActions(); // ZBrush keys: U I O, Space, L, V, Ctrl+A / I / H / W..., the input style (Input.cpp)