Skip to content

Free, open-source 3D sculpting for Linux, macOS and Windows: dense meshes, remeshing, painting, built-in AI (image/text to 3D), Send to Unreal, game-ready LODs, Gaussian splats and OpenUSD.

39.5 KB
Raw Blame

Performance moonshots: sculpting that costs pixels, not polygons

Research branch research/perf-moonshots (2026-11-01). Goal: the cost of a stroke grows with the pixels on screen and the brush footprint, never with the total polygon count. This note surveys recent work, rates eight ideas, reports three prototypes measured with --benchmark standard-33m, and recommends what to ship.

Starting point (a clean 33M run, M1 Pro, Asahi, Mesa GL 4.6, unified memory): orbit 89 fps; Clay small p99 38 ms; Clay large p99 119 ms; 240 px brush p99 338 ms; Move 18 fps. The branch is built on perf/bigbrush-p99 (dabs of meshes from 2M vertices go to the solver thread without the preview), which already takes the stroke p99s to about 30 ms; what it leaves is the surface trailing the pen (each 240 px dab at 33M still costs 100-200 ms of CPU) and Move, which never used the solver path.

0. Hard requirement: the floor tier (11-03)

Weak GPUs must get great performance too, not only big cards. Every idea, prototype and default below is held to this; an experiment that misses it stays off for that tier or doesn't ship.

  1. Floor hardware. Integrated GPUs: Intel UHD 620 / Iris Xe, AMD Vega / 680M iGPU, Apple M1 base. OpenGL 3.3, 8 GB RAM, about 1-2 GB of shared video memory, 4 cores. Windows first (the biggest install base; real-hardware checks by testers).
  2. Costs bounded by screen pixels and brush footprint, never total polygon count. The cluster-LOD pixel budget and the draw budget come from the hardware profile (calibration: renderTriangleBudget(), motionProxyThreshold(), TierConfig::uploadBudgetMBPerFrame). Under load they degrade gracefully: fewer drawn triangles, a lower view resolution while the view moves, full quality on release. They never stutter: a budget is cut before a frame is late, not after.
  3. GPU work is optional. Sculpting, normals, picking and undo stay on the CPU, multithreaded; the GPU only draws a budgeted set. No feature may require compute shaders, SSBOs, mesh shaders or big VRAM; anything newer than GL 3.3 is an optional fast path with a GL 3.3 fallback that produces the same picture. A CPU raster path for sub-pixel clusters (and for the canvas, §7 E) is part of the plan.
  4. Memory. Compact per-vertex storage so 20M+ triangles fit in 8 GB with headroom. Today the scene is about 90 bytes per triangle (docs/performance.md): 20M = 1.8 GB plus undo, which fits; 50M = 4.5 GB, which does not leave headroom on 8 GB. Target: 60 bytes per triangle at the top level (quantized positions in the cluster grid, octahedral normals, implicit adjacency), and micro levels (1.25 bytes per triangle) above ~8M.
  5. Benchmarks. Every number is reported twice: this laptop and the iGPU-simulated floor profile (below), with p99 frame time and stroke latency at 5, 20 and 50M triangles. A beta must not regress the floor tier.
  6. The floor decides; a big GPU never does (11-03). Performance must never be gated by an expensive GPU: that is the value proposition. Every perf claim and every beta gate is judged on the floor tier first (the simulated profile and this laptop), and only then on the RTX 3060, as a sanity check. The 3060 PC is for test speed and Windows driver checks, not the target. A result that meets its target only on the 3060 has not met it.

The iGPU-simulated profile (SF_HW_SIMULATE=floor, core/HardwareProfile.h; applied where the hardware profile is applied, so it is the same code a real floor machine runs): Laptop tier; 4 worker threads; render triangle budget 2M (1M while the view moves); motion proxy from 3M visible triangles; upload budget 16 MB a frame; no GL 4.4 buffer storage (Fast Upload falls back); resident caches capped at 1 GB; the governor may lower the budget, never raise it. View resolution 75% while the view moves, full when it stops (SF_MOTION_RESOLUTION=25..100 sets it on any profile).

Limits of the simulation (11-04). It sets the floor's budgets and thread count, not its per-core speed. On our dab code one M1 Pro performance core is about 1.6x a Ryzen 5600X core (216 px dab on 1 thread: 82 vs 134 ms; a GCC build on the same Ryzen is no faster, so it isn't the compiler; n2-stroke). A floor laptop core (8th-gen i5 class) is slower again, so roughly 2.5-3x an M1 Pro P-core. The simulated floor on this laptop's P-cores is therefore optimistic for CPU-bound work: dab cost, solver, normals, CPU shading. So CPU-bound results are reported as a bracket:

  • floor (P-cores): SF_HW_SIMULATE=floor, as is. The optimistic bound.
  • floor (E-cores): the same under taskset -c 0,1, the two efficiency cores (Linux cpu_capacity 485 vs 1024). Each is ~0.47 of a P-core, near a floor core's speed. With two of them the process has ~1 P-core of compute against a floor laptop's ~1.4-1.6, so this is the pessimistic bound.

A CPU-bound claim counts on the floor only if it holds on the E-cores, or if it holds on the P-cores by a ~2.5x margin. GPU-side results (draw, fill, upload) are bounded by the budgets the simulation does set. The Windows PC can't stand in for a floor CPU either. Windows numbers taken while the PC ran other jobs read about 2x slow (n2-stroke found other sessions' WSL build jobs at 75-100% CPU during win.sh runs; win.sh now takes their lock), and the RTX 3060 runs in this doc (eb2f2aa) predate that fix: sanity checks only, possibly pessimistic.

Frame split. --benchmark now writes frame_split_ms per phase (p50 / p95 / p99 of the main thread's events, sculpt, UI, upload = ghost + scene sync, draw = GL calls, swap_wait = UI draw + swap, where the frame waits for the GPU), with gpu_ms from GL timer queries beside it; stroke phases add stroke_work_ms, the stroke's own totals: solver-thread kernel, normals, BVH, dynamic topology, and the main thread's uploads). It decides whether a native backend (D3D11/12, Metal) behind a thin render interface should move earlier: only if draw + swap_wait dominate on the floor profile at 20M while sculpt and upload are within budget. Run: --benchmark --benchmark-sizes 5,20,50, with and without SF_HW_SIMULATE=floor. For A/Bs add SF_GOVERNOR=0: otherwise the frame governor may lower the budget in some runs and not others (core-200m saw 0.3M vs 1.44M triangles drawn for the same setting), which swamps the effect being measured. The motion-resolution A/B below ran with the governor on, so treat it as indicative.

First measurement (11-03, M1 Pro, one run each at load < 2 at start; p99 frame / stroke latency p50):

20M laptop floor (simulated)
60 px stroke 30.0 ms / 89 ms 36.2 ms / 58 ms
240 px stroke 30.2 ms / 414 ms 30.6 ms / 311 ms
orbit 25.4 ms 21.8 ms
GL draw p99 (any phase) <= 4.7 ms <= 6.1 ms

The floor misses its stroke targets by far (latency 58-311 ms against < 16), while orbit at 20M meets < 33 ms. The GL calls are a few ms, so the time goes to CPU stroke throughput and scene-sync spikes (the motion proxy's build at the start of an orbit), not the driver: a native backend would not fix these. At 50M on the floor, orbit p99 is 44 ms with draw at 32 ms p99, the one place the draw side shows.

Motion resolution, measured (floor, 75% vs 100% while moving, interleaved twice, load < 2 at each start): 20M orbit p50 9.0-9.2 vs 7.6-8.0 ms, p99 21.5-25.9 vs 23.7-24.1 ms; 50M p50 14.5-15.1 vs 9.0-14.0 ms, p99 35.8-51.9 vs 33.0-40.0 ms. No gain at 20-50M: there the motion proxy keeps the pixel cost low, the scaled path's extra pass costs about 1 ms, and the worst frames are scene sync at the start of an orbit (upload p99 15-20 ms in every run). It stays env-only (SF_MOTION_RESOLUTION) until it helps somewhere: the 200M micro scene, where the per-pixel cost dominates (core-200m §8), is the case to test. The 20-50M orbit fix is building the motion proxy before the orbit starts.

Windows, real GPU: sanity check, not the target (RTX 3060, Ryzen 5600X, Win 11, GL 3.3 NVIDIA 616.56; this branch's CI package eb2f2aa; bench/windows/rtx3060-eb2f2aa-*.json). standard-15m, the three toggles off (3 runs) vs on (2 runs), interleaved:

15M frame p99 off / on visible stroke behind the pointer p50 / p99, off / on
Clay 25 px 15-16 / 7 ms 5 / 65-79 px vs 0 / 4-5 px
Clay 60 px 18-22 / 9-10 ms 9 / 104-125 px vs 0 / 6 px
Clay 120 px 22-32 / 23-30 ms 144-160 / 250-323 px vs 0 / 15-18 px
Smooth 19-21 / 9-10 ms 10-11 / 111-141 px vs 0 / 6 px
240 px brush 31-39 / 35 ms 389-496 / 639-824 px vs 0 / 27-28 px
Move 5-8 / 5 ms (p50 4.4-4.8 / 3.6-3.7) (no dabs)
orbit 4.4-4.6 / 4.5-5.7 ms

With the toggles on the stroke you see stays under the pen (the preview's head, preview_lag_px) and small-brush frame p99 halves; the committed mesh finishes as before (its lag is the same or up to ~20% later: the solver does the same work, and the frame no longer waits for it). This is Background Preview's case for on-by-default after the beta's tester reports; the floor A/B (§5) shows the same, which is what counts (item 6). GL draw stays 0.1-0.8 ms p50 even at 50M; the real-GPU and simulated-floor runs at 5/20/50M (governor pinned) agree on strokes (240 px at 20M: 536 / 562 ms committed-surface lag p50) and differ only in orbit, where the floor's motion proxy from 3M makes 5-20M orbit cheaper (20M 1.1 vs 4.6 ms p50); the 50M orbit upload spike (27.5 ms p99) is in both. A 3060 can't simulate an iGPU's fill rate: the floor numbers here test the budgets, and a real iGPU run is still needed. (These JSONs have ram_mb 0: resident memory was read only on Linux until the cherry-picked fix from research/core-200m. On the same PC core-200m orbits 200M at p50 0.9 / p99 2.2 ms, GPU 0.6 ms: the 200M p99 tail on Asahi was presentation on that desktop, not the engine.) It simulates the budgets, not the GPU's fill rate, so real iGPU runs by testers (Windows UHD 620 / Iris Xe first) confirm it.

Targets, floor profile (laptop targets in brackets):

5M 20M 50M
p99 frame, orbit < 16 ms [< 8] < 33 ms [< 11] < 33 ms, proxy [< 16]
p99 frame, 240 px stroke < 33 ms [< 16] < 33 ms [< 33] < 50 ms [< 33]
pen to visible stroke (the preview's head), p99 < 16 ms < 16 ms (canvas track, §7) < 33 ms
committed surface: solver + frame beyond the spacing floor, p50 < 16 ms < 16 ms < 33 ms
RSS after load + 20 strokes < 1 GB < 2.5 GB < 5 GB (needs item 4)

Two stroke latencies, reported side by side (agreed with core-200m and beta-boost, 11-04). Pen to visible is how far the stroke you see trails the pen. That is the preview's job (Background Preview: 3-6 px p50 on the floor, §5). Committed is when the real surface reaches the pointer. It has a floor that no solver can beat: dabs are placed every 0.15 x radius of path, so a dab is on average half a spacing old when it is placed (240 px at 973 px/s: a dab every ~37 ms of path, ~18 ms average age). Its target is solver + frame <= 16 ms beyond that floor, so at 240 px about 35 ms p50 raw. Every report gives the raw lag and the lag minus the spacing floor.

The three prototypes against the floor:

Prototype GL needed Floor tier
Live Move GL 3.3 (vertex shader, uniform arrays) works; takes per-frame grab and upload work off the CPU, so it helps most here. The commit at pen lift is one grab on the CPU (cost is the brush footprint)
Background Preview GL 3.3 works; draws only the clusters the dabs reach
Fast Upload GL 4.4 buffer storage (optional) an optional fast path: without the extension (GL 3.3 contexts, macOS GL 4.1) it is off and uploads go the old way, same picture; on Windows iGPUs (GL 4.6 drivers) it needs the floor-profile numbers before it may default on

1. What SculptForge already does (so the ideas below are measured against the right thing)

Piece Where State
Solver thread for dabs, two-part upload (pack on the thread, glBufferSubData on the GL thread) app/AsyncStroke.cpp, Renderer::prepareStrokeUpload on (bigbrush: from 2M vertices)
GPU preview of the dabs not solved yet ("ghost"), analytic per brush kernel in the vertex shader, normals by forward differentiation render/Ghost.h, kMeshVS GHOST only with Predictive Strokes, which is off by default
Pen prediction: alpha-beta-gamma filter, 25 ms ahead core/PenPredictor.h with the ghost
Draft strokes: positions only, normals at pen lift Sculptor::draftStrokes off; measured no gain on M1
Cluster culling + motion proxy render/Clusters.h on
Micro levels: 16-bit scalar height per point above the top level, evaluated in the vertex shader; stride-sampled brushes core/Micro.h, render/MicroRenderer.h on; 1.02B triangles in 1.7 GB RSS
Kelvinlet brushes (grab, twist, scale, snake) core/Kelvinlet.h on

So "GPU preview from a dab list" (idea 1) and "quantized scalar displacement" (idea 6) exist already; the question for them is where they are not used yet.

2. Research (2022-2026, plus the classics they build on)

Micro-meshes and displacement micro-maps. NVIDIA's micro-mesh primitive is a base mesh plus a compressed, UV-free displacement per base face (11-bit scalars, four watertight levels), with hardware support on recent GPUs for ray tracing [1]. Maggiordomo, Moreton and Tarini, Micro-Mesh Construction (SIGGRAPH 2023 / TOG 42(4)) build them automatically from dense meshes with a base simplification tuned for tessellation and displacement sampling, with an open reference implementation [2]. Our micro levels are the same idea (16-bit heights, barycentric grid per base triangle) for editing rather than for ray tracing; [2]'s base-mesh simplification is relevant to "Divide to micro" on scanned meshes.

Cluster LOD / virtual geometry. Nanite (Karis, Stubbe, Wihlidal, SIGGRAPH 2021 Advances [3]): ~128-triangle clusters, a DAG of cluster groups simplified with locked group borders, LOD chosen per group by projected error, visibility buffer and a software rasterizer for small triangles, streaming of pages. The 2021 design assumed rigid meshes; UE 5.4 added tessellation with programmable displacement and UE 5.5 skinned Nanite meshes (deformed on the GPU before culling) [4] (secondary sources; Epic's own notes should be checked before quoting). NVIDIA's RTX Mega Geometry (2025) builds cluster acceleration structures, cluster templates (topology reused, only vertices new), per-frame cluster tessellation of subdivision surfaces at about one triangle per pixel, and a continuous cluster-LOD library [5]. The common lesson for an editor: keep topology-derived data (clusters, LOD DAG, templates) fixed while vertices change, and rebuild only the cluster groups an edit touched.

Sculpting engines. Blender 4.3's sculpt refactor (2024) reports brushes about 8x faster and 30% less memory on a 16M-face mesh, from data-oriented attribute arrays, a face (not triangle) BVH, removing per-element call layers and writing deformation directly per symmetry pass [6]. That is the "make the CPU kernel lean" path; we are already there in structure (flat arrays, parallel kernels); our remaining dab cost is proportional to the vertices under the brush, which is the term this note attacks.

Analytic deformers. Regularized Kelvinlets (de Goes and James, SIGGRAPH 2017) give closed-form, volume-aware grab/twist/scale/pinch brushes from regularized fundamental solutions of linear elasticity [7]; Dynamic Kelvinlets (2018) and Sharp Kelvinlets (DigiPro 2019, multi-scale cusps) extend them [7]. Blender's Elastic Deform and Unreal's modeling tools ship them. Key property for us: a closed-form displacement of the start position is exactly what a vertex shader can evaluate; so are our Move and Standard kernels.

Progressive refinement. Hoppe's progressive meshes (SIGGRAPH 1996) and their GPU successors (cluster DAGs above) refine coarse-to-fine by screen error. For editing, the exact final geometry is the constraint: coarse first is useful as a display order, not as a different result.

Zero-copy buffers. GL_ARB_buffer_storage (core in 4.4) allows immutable storage that stays mapped while the GPU uses it (MAP_PERSISTENT), with writes visible without flushes (MAP_COHERENT); it is the core of the "Approaching Zero Driver Overhead" techniques (Everitt, McDonald et al., GDC 2014 [8]). It does not remove synchronisation: the application must fence what the GPU may still read [8].

Pen-latency prediction. Henze et al. (MobileHCI 2016/2017) extrapolate touch with small neural networks; predicting 33 ms ahead improved task performance, beyond that it did not [9]. A GRU-CNN stylus predictor (Samsung Research, HCII 2020) reports 0.07/0.24/0.47 mm error at ~16/33/50 ms, several times below LSTM baselines [10]. TurboTouch (UIST 2018) uses finite-time derivative estimation and predicts 2-3x further with few side effects [11]. Our alpha-beta-gamma filter is the classical baseline these improve on.

What the Asahi driver offers (checked on this machine). Mesa 26.1.8 reports OpenGL 4.6 core with GL_ARB_buffer_storage, GL_ARB_compute_shader, GL_ARB_shader_storage_buffer_object, GL_ARB_multi_draw_indirect, GL_ARB_indirect_parameters and GL_ARB_shader_draw_parameters (glxinfo). The Asahi drivers are conformant for OpenGL 4.6, OpenGL ES 3.2, OpenCL 3.0 and Vulkan 1.4 (Honeykrisp, in Mesa since 24.3); the hardware has no geometry or tessellation shaders, which the driver emulates with compute [12] (so our texture-painting geometry shader is a compute pass on this GPU). No source says Honeykrisp exposes VK_EXT_mesh_shader, and vulkaninfo is not installed here; treat mesh shaders as unavailable. GPU-driven culling therefore means compute + multi-draw-indirect, which the driver has.

3. How the established desktop sculpting app does it (publicly documented approaches only)

From its vendor's public reference manual only (no binaries inspected): (a) 2.5D canvas: the document is "a 2.5D canvas", not a 3D scene; each canvas pixel stores depth, orientation and material [13]. The heritage is a CPU renderer whose cost follows pixels. (b) Subdivision levels: the classic multires stack (ours: subdivision levels). (c) Local high-resolution levels: extra levels above the top subdivision level are sculpted only in a circular or square region around the cursor [14]; only that region is at the extra resolution while the rest stays at the regular top level. (d) Per-mesh caps: a memory preference limits polygons per mesh (in millions), with a separate cap for the local high-resolution preview [15]; dense work is split into many objects, hidden ones costing little.

What it implies: their interactivity comes from bounding the active point count (cap per object, a local HD region) on a CPU renderer. Our equivalents must bound the same thing with our own design: micro levels (bounded bytes per triangle), a focus region (idea 8) and GPU drawing whose cost follows the screen.

4. Ideas, rated

Gain is for the 33M standard scene unless said; effort in agent-days; fit = how well it suits our engine.

# Idea Expected gain Risk Effort Fit
1 GPU stroke preview from the dab list, async commit surface under the pen within a few px at any brush size (measured below); Move frame cost independent of the region low: preview only, the solver's mesh is unchanged 1-2 (exists; extend) high
2 Coarse-first progressive dab application none on the final mesh (same work); as a display order (visible clusters packed first) maybe 30-50% sooner visible surface on big symmetric brushes medium: order changes must stay bit-identical 2-3 low once (1) is on: the preview already shows the result
3 Strokes as data (analytic fields, replay, re-tune, cheap undo) undo of a 240 px stroke at 33M: tens of MB of positions -> ~10 KB of dabs; re-tune strength after the fact high: Clay, Smooth, Flatten read the surface, so replay is sequential and as costly as the stroke; determinism across versions 10+ medium (Sculpt Timeline is the natural home)
4 Zero-copy persistent mapped buffers on unified memory the GL thread's commit (2-10 ms per batch) -> a GPU copy command; measured below low-medium: fences; not on macOS GL 4.1 (fallback kept) 1 (prototyped) medium
5 Cluster LOD with a per-cluster pixel budget (Nanite-style DAG on an editable mesh) orbit cost bounded by screen (as micro levels already are: 3-4M triangles drawn at 299M, 8-9 ms); 33M orbit 89 -> ~110+ fps high: cracks at group borders, rebuild the edited groups after each stroke 10-15 medium (compute + MDI available, no mesh shaders)
6 Quantized scalar displacement + GPU normals memory 90 -> 1.25 B per triangle above the top level (micro levels, measured in docs/performance.md) low (exists) 2-3 to make it automatic ("Add Level" past ~8M goes to a micro level) high
7 Learned pen prediction at most 1-3 px of the 1-4 px preview lag left; literature: no gain beyond 33 ms ahead [9] medium: training data, per-device tuning 3-5 low
8 Focus Detail (below) billion-triangle assets in a few GB of RAM and ~0.3-0.5 GB of GPU, editable locally at the speed of a 30-60M mesh medium-high: streaming latency, seams between levels 8-12 (data side started on research/focus-detail) high

Idea 8: Focus Detail (our design)

The region under and near the brush is promoted to full micro resolution; everything else stays at a coarse micro level (or compressed on disk), and the GPU draws each tile at its own level.

  • Data (core/FocusDetail, branch research/focus-detail, in progress): micro tiles (one per base triangle) stored zstd-compressed in a tile file; a focus region (brush footprint plus a margin, plus what the view needs at one micro triangle per pixel) is decompressed to full level; tiles outside hold an upsampled coarse grid (level 3) so the MicroRenderer works unchanged today. Promotion and demotion markDirty(tri); budget the promoted tiles by bytes (e.g. 512 MB) and demote least recently used.
  • GPU (follow-up here): per-tile resident level. Coarse tiles keep 45 points (level 3) instead of 8385 (level 7), drawn with the same vertex-shader evaluation at that level; the data side exposes FocusDetail::tileLevel(tri). Edge stitching between a level-3 and a level-7 tile: snap the fine tile's edge points to the coarse edge (as micro levels already stitch shared edges).
  • What it enables, estimated for 1.0B triangles (micro level 7 on 61K base triangles; 8385 points a tile): all tiles at full level are 511M points: 1.0 GB of heights on the CPU, ~3 GB on the GPU (2 B height + 4 B shading per point). With Focus Detail: coarse everywhere 2.7M points (16 MB GPU); a focus of ~2% of the tiles (a 240 px brush plus margin on a full figure) 1.2K tiles x 8385 points = 10M points (60 MB GPU, 20 MB CPU); compressed tile file perhaps 0.4-0.6 GB on disk (16-bit heights of sculpted detail compress 2-3x; to be measured). Dab cost depends on the points under the brush at full level, the same as on a 30-60M mesh today (docs/performance.md: Standard 64 px on 1.02B Micro 5 is 39 ms a step, 300 px 283 ms); drawing stays bounded by the screen. A billion-triangle asset, editable locally at interactive rates on a 16 GB laptop, is the claim to test; nothing here is measured yet, so no public claim until it is.
  • Measured on the data side (research/focus-detail @ 6184559, docs/design/focus-detail.md §5.1; 1B triangles, shared machine, load ~4): promoting one tile 0.041 ms p50 / 0.051 ms p99, 65.5K tiles/s in parallel; RSS 1147 MB with the focus on against 1526 MB with every tile at full level; GPU memory (estimated) 3130 MB with today's renderer against 597 MB with per-tile resident levels (FocusDetail::tileLevel(tri) is in). So promotion is far below a frame (a few hundred tiles a frame while orbiting is ~10 ms on one thread, ~5 ms spread over the pool); the GPU side (per-tile levels in the MicroRenderer) is what turns the estimate into the 5x saving.
  • Difference from (c): the focus follows the brush automatically (no mode to enter), the coarse remainder is the same surface at a lower micro level (not a separate HD layer), and GPU memory follows focus + screen.

5. Prototypes (feature flags, off by default)

All three are Experimental toggles in Preferences > Performance (Live Move, Background Preview, Fast Upload), off by default, with environment overrides (=0 / =1) for benchmarks; they cost nothing when off. None changes what a stroke leaves on the mesh except Live Move's commit (one grab with the summed move instead of one per frame; tested equal to 1e-6).

  1. Background preview (SF_BG_GHOST=1, idea 1): the ghost (already written for predictive strokes) also draws the dabs of bigbrush's background path, and the frame no longer waits 12.5 ms for a batch. The solver's dabs are unchanged (same dabs, same order), so the final mesh is identical by construction.
  2. Live Move (SF_LIVE_MOVE=1, ideas 1 and 3): a Move drag is not applied frame by frame. The GPU draws it from the grab's start positions (GhostKind::Move: x0 + d w(|x0 - c|), normals by the rank-one inverse transpose), per symmetry mirror, and one Sculptor::grab with the whole move is applied when the stroke ends. Falls back to the old path with masks, hidden faces, sculpt layers, micro levels, connected-only or view falloff. Test live_move_matches_the_grab: the preview matches the grab to 1e-5, one grab matches the per-frame grabs to 1e-6 (X symmetry, falloff).
  3. Persistent upload (Fast Upload, SF_PERSISTENT_UPLOAD=1, idea 4): two persistently mapped, coherent staging buffers (48 MB each, glBufferStorage); the solver thread packs straight into one, the GL thread issues glCopyBufferSubData for each range and a fence; a buffer is reused only after its fence. Batches larger than 4M vertices pack into the old vector as before.

Measurements

Correctness (all in ctest): live_move_matches_the_grab (preview vs grab 1e-5, one grab vs per-frame grabs 1e-6); ui_live_move (the same drag with Live Move on and off moves the mesh by the same sum and maximum to 0.1%; mid-drag the mesh is untouched and the GPU draws the move: screenshot live_move_dragging matches the committed live_move; undo/redo); ui_stroke_background_preview and ui_stroke_persistent_upload (SF_MEASURE_VERIFY=1: the background path's mesh is bit-identical to the plain path's with the preview and the persistent upload on).

Speed, floor tier first (§0 item 6). Run on 11-04 on this laptop (M1 Pro, Asahi), local release build of 522aea3, standard-15m, SF_GOVERNOR=0, all three toggles off vs on. Floor (SF_HW_SIMULATE=floor, so Fast Upload is off and only Live Move and Background Preview act): 4 pairs, interleaved, order reversed for the second two. Laptop: 1 pair. Every run started at load < 2. bench/floor/m1pro-522aea3-std15m-*.json. Ranges are across runs; "lag" is when the committed surface reaches the pointer.

floor, 15M frame p50 / p99, off -> on visible stroke behind the pointer p50 / p99, off -> on committed lag p50, off -> on
Clay 25 px 7.9-12.5 / 26-36 -> 6.1-10.3 / 21-33 ms 8-14 / 79-152 px -> 3-5 / 14-30 px 14-19 -> 27-39 ms
Clay 60 px 12.4-20.6 / 28-38 -> 6.7-10.1 / 16-26 ms 18-55 / 95-137 px -> 3-5 / 12-27 px 24-78 -> 28-43 ms
Clay 120 px 16.3-24.4 / 32-34 -> 8.7-9.2 / 15-20 ms 91-112 / 149-176 px -> 5 / 12-14 px 123-137 -> 106-146 ms
Move 16.8-21.8 / 36-42 -> 6.5-12.5 / 17-32 ms (no dabs)
Smooth 18.0-23.3 / 30-44 -> 6.7-8.0 / 14-15 ms 60-71 / 107-124 px -> 3-4 / 11-13 px 71-85 -> 37-65 ms
240 px brush 10.4-11.1 / 31-33 -> 10.2-10.6 / 16-29 ms 189-212 / 302-356 px -> 6 / 17-31 px 232-267 -> 206-327 ms
orbit 8.6-8.9 / 23-33 -> 7.4-9.0 / 22-36 ms

With Fast Big Brushes (perf/beta-boost, on by default on main since 9b81767e; merged tree 2dd3fa0, floor, 2 pairs interleaved, load < 2 at each start; bench/floor/m1pro-2dd3fa0-fastbig-std15m-*.json), the three toggles off -> on:

floor, 15M frame p99 visible stroke p50 / p99 committed lag p50, raw (minus the ~18 ms spacing floor at 240 px)
Clay 25 px 31-34 -> 21-26 ms 14 / 151-165 -> 3-5 / 24-26 px 18 -> 29-32 ms
Clay 60 px 35-41 -> 21-24 ms 17-18 / 114-122 -> 5 / 18 px 22 -> 31-32 ms
Clay 120 px 32-33 -> 15-16 ms 54-56 / 103-114 -> 4-5 / 13-23 px 65-66 -> 45-46 ms
Move 35-36 -> 31 ms (no dabs)
Smooth 35-45 -> 13-14 ms 69-73 / 121-124 -> 4 / 12-13 px 80-83 -> 59-62 ms
240 px brush 32-33 -> 17-21 ms 68-70 / 165 -> 5 / 16-18 px 84-86 -> 66-76 ms (48-58 ms beyond the floor)

Fast Big Brushes takes the 240 px committed lag from 232-267 to 84-86 ms with the toggles off, and to 66-76 ms with them on, so the two stack. With both, every stroke phase meets p99 < 33 ms on the floor and the visible stroke stays within 5 px p50. Committed lag beyond the spacing floor (target <= 16 ms) is still about 3x over at 240 px.

Pessimistic bound: the same on the efficiency cores (taskset -c 0,1, §0; tree 6231d18 with Fast Big Brushes, 2 pairs interleaved, load < 2 at each start; bench/floor/m1pro-6231d18-ecores-std15m-*.json), toggles off -> on:

floor, E-cores, 15M frame p99 visible stroke p50 / p99 committed lag p50 pen up to done
Clay 25 px 34-37 -> 20-22 ms 14 / 179-216 -> 6 / 18-21 px 18-19 -> 32-33 ms 0.17 -> 0.12 s
Clay 60 px 32-37 -> 25-29 ms 18-39 / 565-826 -> 8 / 27-30 px 25-54 -> 39-55 ms 0.7 -> 0.8 s
Clay 120 px 41-44 -> 28-31 ms 715-722 / 1163-1274 -> 13 / 28-32 px 717-756 -> 594-644 ms 1.8 -> 1.6 s
Move 69-72 -> 27-30 ms (no dabs)
Smooth 39-45 -> 28 ms 39-40 / 603-604 -> 8 / 29-31 px 45-47 -> 57-64 ms 0.25 -> 0.28 s
240 px brush 41-43 -> 33-34 ms 1272-1745 / 3231-3521 -> 15-17 / 35-36 px 1294-1794 -> 1424-1496 ms 3.0 -> 3.0 s
orbit 165-166 -> 164-166 ms (p50 8.6-8.9 either way)

On slow cores, Background Preview and Live Move still do their job: the visible stroke stays within 6-17 px p50 of the pen, against 0.7-1.7 s behind with them off. Move's p99 frame halves. All stroke frames are p99 <= 31 ms except the 240 px brush at 33-34 ms (target < 33: just misses). What fails on the pessimistic floor, with or without the toggles, is the CPU:

  • committed lag of big brushes: 0.6-1.5 s, and the stroke finishes up to 3 s after pen up at 240 px;
  • a ~165 ms hitch at the start of every orbit: about 3 frames, in the upload phase (the motion proxy's build and upload on the main thread). On the P-cores it is 23-36 ms. These two are the floor's top problems: building the motion proxy ahead of the orbit and off the main thread, and CPU dab throughput (Fast Big Brushes, core-200m N2). They come before anything that needs a faster GPU.

Fixed: the orbit-start hitch (11-04). SF_FRAME_LOG + SF_SYNC_LOG on the E-cores showed three frames of 157-160 ms, all in Renderer::buildProxies. Its 8 ms budget was checked only after each fixed batch of 1024 clusters, and on two slow cores one batch took the whole 157 ms. Now each batch is sized from the measured cost per cluster to fit what is left of the frame's budget (16-1024 clusters). Still frames after a stroke also rebuild the stale proxies at 4 ms a frame (Renderer::setProxyPrebuild, only for meshes already orbited, never while sculpting), so the next orbit usually starts with them ready. Floor, standard-15m, orbit p50 / p95 / p99 / max (bench/floor/m1pro-proxyfix-*):

before after
E-cores 8.6-8.7 / 16 / 165-166 / 170-174 ms 8.2-9.1 / 21 / 26-31 / 50-64 ms
P-cores 9.0 / 12-14 / 26-27 / 36-42 ms 8.8-9.1 / 14-15 / 21-31 / 42-53 ms

On the E-cores the floor's orbit now meets p99 < 33 ms. p95 rises a little because the build is spread over more frames. On the P-cores nothing changes beyond noise. One frame per orbit still reaches 50-64 ms; that's the next one to find.

The laptop profile shows the same pattern (240 px: 208 -> 6 px p50, frame p99 33 -> 16 ms; Smooth p99 36 -> 12 ms). On the simulated floor tier (optimistic for CPU-bound work, §0), with the toggles on, every stroke phase's p99 frame meets the < 33 ms target (off: 4 of 6 miss) and the stroke you see stays within a few pixels of the pen. The cost: the committed surface on a small brush lands about 15-20 ms later (the frame no longer waits for the batch), while on big brushes and Smooth it lands as early or earlier. Orbit doesn't change (the toggles don't act there; the ranges overlap over 4 pairs). Its p99 still touches 33-36 ms in some runs either way: that is the orbit-start scene-sync spike (§0), not the toggles. The RTX 3060 runs (§0) agree, as a sanity check. Not yet met on the floor tier: committed stroke latency (< 16 ms target; 14-327 ms here), which is the canvas track's job (§7) and CPU dab throughput (core-200m N2).

6. Recommendation

  • Ship now, Experimental, off by default (0.11.0-beta.1): Live Move, Background Preview, Fast Upload. They are tested equal to the plain path, so the beta risk is in drawing (driver differences), which is what the #testers checklist asks about. Promote one to on-by-default only when the floor profile (§0, item 6) shows the gain, with clean tester reports on Windows (iGPU laptops first), Linux and macOS; the RTX 3060 numbers are a sanity check, not the case.
  • Build next: Focus Detail's GPU side (per-tile resident levels in the MicroRenderer, idea 8): the data side's measurements (promotion 0.05 ms a tile, GPU 3130 -> 597 MB at 1B) make it the biggest lever toward billion-triangle assets. Then idea 6 made automatic (Divide past ~8M goes to a micro level).
  • Later or not at all: cluster LOD (5) only if orbit at 33M+ stays below target after Focus Detail; strokes as data (3) belongs with the Sculpt Timeline; coarse-first (2) and learned prediction (7) don't pay once the preview is on.

7. Canvas track (the cached screen; 11-03)

Generic techniques, designed here from first principles (clean room: no decompiling, no copied code or docs). Goal: a dab costs about the brush's pixels, whatever the mesh size, so the floor tier sculpts 20M like a small mesh.

  • A. Cached screen canvas. Keep a G-buffer between frames: depth, normal (octahedral, 2x16 bit), material and colour per pixel. When nothing moves, nothing is redrawn (the canvas is composited as one quad). While sculpting, only the clusters overlapping the brush's dirty screen rectangle (the dab's screen bound, grown by the move) are re-rasterized into the canvas, scissored to that rectangle, and only that rectangle is re-shaded. The rectangle comes from the dab list we already have (the solver's dabs and the ghost's bounds); the clusters come from the existing cluster bounds. Cost per dab: brush pixels plus the clusters under them. Correctness: the depth test inside the scissor against the cached depth outside it; clusters straddling the edge are drawn whole and clipped.
  • B. Deferred Lit Sphere shading. Shade the canvas per pixel from its cached normals with the Lit Sphere lookup (one texture fetch, no lights, no shadows) in the interactive path; the full material (PBR, cavity, shadows) is applied when the view is still. Re-shading a rectangle is then a texture pass, not a geometry pass.
  • C. Motion. While orbiting or zooming: draw a lower level or budget (the motion proxy, micro levels' coarse level), or reproject the cached canvas for the first frames of a move; full detail returns on release. Input is never stalled: the frame takes what the budget allows, never waits for geometry.
  • D. Compact data, levels as offsets. Multires levels stored as offsets from the level below (the micro levels' 16-bit heights already are), so a large brush works at a coarse level and the detail rides along; with §0 item 4 this is how 50M fits the floor.
  • E. CPU rasterizer for the canvas. A tiled, multithreaded CPU rasterizer (binning by 64x64 tiles, SIMD edge functions, one thread per tile) that writes the same G-buffer, so a GL 3.3 iGPU only composites one textured quad. Also the path for sub-pixel clusters (§0 item 3). Compare CPU vs GPU fill per machine on the floor profile; calibration measures both and the hardware profile picks one (TierConfig, a new canvasRaster field).

Targets (floor profile, 20M): stroke latency < 16 ms, p99 frame < 33 ms; idle frames ~0 ms of GPU (composite only). Measured as in §0 (5/20/50M, laptop and floor).

Order of experiments, each behind a flag, off by default, with numbers before it ships: (1) A + B on the GPU (the canvas, dirty rectangles, Lit Sphere re-shade); (2) C reprojection of the canvas; (3) E the CPU rasterizer writing the same canvas, and the per-machine pick; (4) D with Focus Detail's per-tile levels. Risks: the canvas must match a full redraw (a screenshot test compares them pixel for pixel at rest); transparency, X-Ray and wireframe overlays draw on top of the canvas, not into it.

References

  1. NVIDIA Research, Micro-Mesh Construction project page and micro-mesh format: https://research.nvidia.com/publication/2023-08_micro-mesh-construction
  2. A. Maggiordomo, H. Moreton, M. Tarini, Micro-Mesh Construction, ACM TOG 42(4), SIGGRAPH 2023: https://micromesh.di.unimi.it/
  3. B. Karis, R. Stubbe, G. Wihlidal, Nanite: A Deep Dive, SIGGRAPH 2021 Advances: https://advances.realtimerendering.com/s2021/Karis_Nanite_SIGGRAPH_Advances_2021_final.pdf
  4. UE 5.4 Nanite tessellation (Puget Systems): https://www.pugetsystems.com/blog/2024/04/16/exploring-unreal-engine-5-4-whats-new-and-whats-next/ ; UE 5.5 skinned Nanite (secondary): https://wnhub.io/news/engines/item-45659
  5. NVIDIA, Fast Ray Tracing of Dynamic Scenes Using OptiX 9 and RTX Mega Geometry: https://developer.nvidia.com/blog/fast-ray-tracing-of-dynamic-scenes-using-nvidia-optix-9-and-nvidia-rtx-mega-geometry ; Vulkan samples: https://developer.nvidia.com/blog/nvidia-rtx-mega-geometry-now-available-with-new-vulkan-samples
  6. Blender developers, This Summer's Sculpt Mode Refactor (2024): https://code.blender.org/2024/11/this-summers-sculpt-mode-refactor/
  7. F. de Goes, D. L. James, Regularized Kelvinlets (SIGGRAPH 2017): https://graphics.pixar.com/library/Kelvinlets ; Sharp Kelvinlets (DigiPro 2019): https://graphics.pixar.com/library/SharpKelvinlets
  8. ARB_buffer_storage specification: https://developer.download.nvidia.com/opengl/specs/GL_ARB_buffer_storage.txt ; C. Everitt, J. McDonald, Beyond Porting (AZDO): https://www.ozone3d.net/dl/201401/NVIDIA_OpenGL_beyond_porting.pdf ; persistent mapping notes: https://cppstories.com/2015/01/persistent-mapped-buffers-in-opengl
  9. N. Henze, H. V. Le, S. Mayer, V. Schwind, Improving Software-Reduced Touchscreen Latency, MobileHCI 2017: https://www.medien.informatik.uni-muenchen.de/pubdb/publications/pub/henze2017improving/henze2017improving.pdf
  10. Samsung Research, Active Stylus Input Latency Compensation on Touch Screen Mobile Devices (HCII 2020): https://research.samsung.com/research-papers/Active-Stylus-Input-Latency-Compensation-on-Touch-Screen-Mobile-Devices
  11. TurboTouch predictor (UIST 2018): https://github.com/LokiResearch/TurboTouchPredictor ; https://hal.archives-ouvertes.fr/hal-01893310
  12. A. Rosenzweig, Asahi GPU driver status (OpenGL 4.6, Vulkan 1.4, geometry/tessellation emulation): https://rosenzweig.io/blog/asahi-gpu-part-n.html ; Honeykrisp in Mesa 24.3: https://www.phoronix.com/news/Apple-Vulkan-Driver-Mesa-24.3
  13. The vendor's public reference manual, chapters on the canvas document and its pixels.
  14. The same manual, the chapter on local high-resolution levels.
  15. The same manual, the memory preferences chapter.