Free, open-source 3D sculpting for Linux, macOS and Windows: dense meshes, remeshing, painting, built-in AI (image/text to 3D), Send to Unreal, game-ready LODs, Gaussian splats and OpenUSD.
Testing flow
Each assurance is paid for once, by the cheapest machine that can give it. Before this flow, one change was tested four times: by the agent, by the merger's full local ctest, by CI on main, and by CI again on the release bump.
| tier | when | where | what | target |
|---|---|---|---|---|
| T0 impact | every agent commit | the PC (laptop if the PC sleeps), or GitHub Actions while the PC queue is long | build + the tests the change can break | < 1 min |
| T1 batch | once per merge batch | the PC, while T2 runs | the whole suite, -j10 on the GPU, test cache on | < 5 min |
| T2 platforms | once per merge batch | CI on the branch staging |
every platform, VFX Reference Platform, Blender | the gate for main |
| T3 nightly | once a day | CI (scheduled) + the PC | label nightly: benchmarks, stress, heavy meshes, floor-tier perf |
T0: what can this change break?
tools/test/impacted.py turns the changed files (branch vs origin/main, plus uncommitted ones) into a
ctest -R regex:
- build graph:
build.ninjaplusninja -t deps, so headers count. A file maps to the objects that include it, then to the libraries and programs that link them, then to the tests that run those programs. A core header reaches everysculptforge_testsshard,sculptforge-cliand the Python API. - a test's own files: the script it runs, any file its command names, and that file's folder.
- UI data: scripts that name a changed
tests/ui/datafile. - hand map (top of impacted.py): docs, site, release scripts, note fragments, packaging.
- app changes (every UI test runs the app, so the graph alone would pick all of them): UI tests whose name
shares a word with the changed file (
PanelMicro.cpppicksui_micro_levels), UI tests changed in the same commits as it in the last 600 commits, and a smoke set.
marks (the IP check) always runs. CMake files mean everything runs (.). -v explains each pick.
Run it with ~/Work/.sf/t0.sh: pc.sh builds on the PC, runs impacted.py there against the PC's build graph, then
runs the tests through tools/test/run.py.
T0 on GitHub Actions
SF_T0_CI=1 ~/Work/.sf/t0.sh (automatic while more than 3 jobs wait for the PC, never with SF_PC_PRIO=1;
SF_T0_CI=0 turns it off) runs the same T0 on GitHub's free runners, so branches test in parallel instead of
queueing for the PC. ~/Work/.sf/t0-ci.sh <worktree> pushes the worktree's state to t0/<branch> (uncommitted work
as a temporary commit; the worktree is not touched), waits for .github/workflows/t0.yml, prints the same lines as
t0.sh and exits 0 green, 1 red, 3 unavailable (t0.sh then uses the PC); it deletes the t0/ ref after. Results
stay in ~/Work/.sf/t0-ci/<worktree>/.
- The run (
tools/ci/t0.sh):check_marks, a Ninja build, impacted.py against the merge base withorigin/main, the picked tests throughrun.pyon the CPU (Mesa llvmpipe,SDL_VIDEO_DRIVER=offscreen,SF_TEST_BUDGET=150;GITHUB_ACTIONSgives the in-test time limits their CI slack). - Always green: the job is
continue-on-errorand the script never fails, so a red T0 emails nobody. The verdict is the run summary and the artifactt0-result(result.json: green/red, stage, failed tests;failed.txt: LastTestsFailed + FAILED lines;output.txt:ctest --output-on-failureon the failed tests). - Cache: every push to main builds once and saves
ccache-t0-linux.<commit>(main's own CI mostly reuses staging's run and builds nothing, and at0/ref can only read its own and main's caches).t0/runs only restore. - Locked down (
tools/release/check_workflows.py): pushes to main andt0/**only,contents: read, no secrets, pinned actions; other branch-push workflows ignoret0/**. - A branch older than t0.yml gets it (and
tools/ci/t0.sh) from origin/main in one extra commit, "t0-ci overlay"; the tests are still picked from the branch's own files. - The PC stays for GPU-specific checks (it runs the D3D12 GPU path), T1 and release staging.
T1 and T2: one batch, two machines at once
The merger (~/Work/.sf/merge-batch.sh) merges the green branches, then calls
scripts/merge-queue.sh staging --with ~/Work/.sf/t1.sh HEAD:
- the batch commit is pushed to
stagingand the full CI runs there (workflow_dispatch,full=true); - at the same time,
t1.shruns the whole suite on the PC; - when both are green, main fast-forwards to exactly that commit. When either is red, main doesn't move.
The PC runs one job at a time, build and tests together. A build overlapping another job's tests drove the
load to 65 on 12 threads and the compiler crashed. Each worktree fetches and builds its own dependencies
(/root/wt/<name>/fc); ccache (CCACHE_BASEDIR=/root/wt) is shared, so another worktree's objects are cache hits.
CI on main then doesn't build it all again. The Changes job sees that this exact commit passed a full run on
staging, with both VFX jobs green, and passes as reused. The merger no longer runs a local full ctest. The release
packages (packages.yml) are built on staging beside CI, and main's Packages run reuses them (job reuse). A release
batch carries the bump commit (RELEASE=X.Y.Z ~/Work/.sf/merge-batch.sh ..., docs/releasing.md), so its packages
are the ones the release publishes.
Release: the bump reuses CI (or rides in the batch)
scripts/release.sh step ci used to wait about 20 minutes for CI on the bump commit. That commit only adds
paperwork to a commit CI had already passed. tools/release/paperwork_only.py BASE HEAD checks this file by file
and line by line: the version lines, CHANGELOG, Markdown docs, the generated site, deleted fragments. Anything else
fails safe. When the check passes, step ci and release.yml (job locate) take the earlier commit's CI result.
On main, a newer push never cancels a run that is still going (ci.yml concurrency), so that earlier result exists. The
bump's own Packages run still has to pass, because the version is built into the app. Step check runs the bump's
own tests (What's New, the guide, notes) on the bumped tree.
tools/test/run.py: cache, flakes, budget
Every tier runs ctest through tools/test/run.py --build build [-- ctest args]:
- Nightly tests (label
nightly) run only when the ctest arguments select them (-L nightly). - Test cache (
--cache DIR,$SF_TEST_CACHE; the PC keeps/root/.cache/sf-test): every test gets a fingerprint, a hash of everything it can read. That covers its program, every file or folder its command names, the app's assets and test data for app tests,tests/datafor the unit tests, its ENVIRONMENT, and theSF_,SCULPTFORGE_,SDL_and GL driver variables. Script tests that nobody listed inREADShash every tracked file. A fingerprint that passed on the first try is skipped next time, so a rerun of the full suite runs only what changed. The cache is never used on CI, and the nightly run on the PC runs everything without it once. - Flakes: failed tests run once more. A test that passes on the rerun is logged as
flaky: name, added totests/flaky.md(by pc.sh, t0.sh and t1.sh), and doesn't fail the run. A test listed there twice gets the labelflaky, and a failure then never blocks a release. Its owner fixes it or deletes it. - Budget: a test slower than
--budgetseconds fails the run. For thesculptforge_testsshards, each TEST case counts on its own. The default is 20 s on the PC's GPU; CI setsSF_TEST_BUDGET=150for software OpenGL on 4-core runners.tests/slow_tests.txtlists the tests over budget today, with their PC seconds and a plan for each. A listed test may grow to 1.5× its seconds (scaled by budget ÷ 20 on CI), so the list can only shrink. A slow test should wait on events rather than fixed frames, use a smaller mesh or texture, be split, or be labelled nightly. - Shards by time:
tests/shard_times.inclists the TEST cases of 0.5 s or more with their PC seconds; the shards take them longest first, each to the shard with the least work so far (tests/test_main.cpp). When a test gets much faster or slower, re-runtools/test/shard_times.py /tmp/sf-pc-*-times.tsv(--overrideyour branch's times file with--matchfor the tests you changed). - Where the time goes:
SF_UI_PROFILE=1makes a UI test print its slowest script lines with real seconds (SF_PC_ENV=SF_UI_PROFILE=1 ~/Work/.sf/pc.sh . -R ui_bake). - Parallel runs (
ctest -jN, N > 1): timing checks get CI's slack (SF_TIME_LIMIT_SCALE=4,timeLimit()in tests/TestHarness.h). With--threads-per-test auto(pc.sh), each test process gets its share of the cores (SF_THREADS, core/ThreadPool.cpp), instead of ten pools each the size of the machine. - GPU lost: a failure whose output says the graphics device went away is the machine, not the code (exit 4).
pc.sh says so, and such a failure is never logged as a flake. Tests that the PC's WSL GPU path (Mesa on D3D12) can't
run carry the label
no_d3d12, which pc.sh leaves out. ui_micro_levels makes D3D12 remove the device, yet passes on CI and on the same PC's native driver.
T3 nightly
TEST(nightly_*)cases (tests/test_main.cpp) are left out of every shard. They run insculptforge_tests_nightly. Thetests/perf/*.sfuiscripts areperf_*(labels ui and nightly)..github/workflows/nightly.ymlruns the nightly tests that don't need a GPU (-L nightly -LE ui) once a day.~/Work/.sf/nightly.shis run by hand at the end of the day, not from cron. It runsctest -L nightlyon the PC's GPU, the whole suite with no cache, floor-tier stroke benchmarks on the laptop (SF_HW_SIMULATE=floor, only at load < 2), and, only on request (SF_NIGHTLY_WIN=1: it opens a window on that desktop), an idle and calibration check on the Windows desktop (win.sh). Every other app run is offscreen (SDL_VIDEO_DRIVER=offscreen). The summary goes to~/Work/.sf/nightly/<date>.md.
Speculative CI
~/Work/.sf/sfwatch.sh runs in the background and tests every new commit as it lands. A worktree's HEAD gets T0
and a new origin/main gets T1. Results go to ~/Work/.sf/watch/<worktree>.md. A red result is sent to the
worktree's herdr agent. t0.sh prints that result instead of running again when the commit matches and the tree is
clean. By the time an agent asks, the answer is usually there.