Free, open-source 3D sculpting for Linux, macOS and Windows: dense meshes, remeshing, painting, built-in AI (image/text to 3D), Send to Unreal, game-ready LODs, Gaussian splats and OpenUSD.
C++ 3d3d-sculptingcppdear-imguidigital-sculptingfbxgame-developmentgaussian-splattinggltfimage-to-3dmcpopenglopenusdsculptingtext-to-3dunreal-engine
REPORT.md: CI and release pipeline results, runner options, follow-ups
islandmed97e5bc9e6449
diff --git a/REPORT.md b/REPORT.mdnew file mode 100644index 00000000..13959726--- /dev/null+++ b/REPORT.md@@ -0,0 +1,228 @@+# cifast report: fast CI and a fast, gated release++Branch `feat/cifast` (pushed at d3782bb, rebased on origin/main b78969e: 0.10.4 in progress, not shipped). It changes+workflows, the test runner, test registration and smoke tests, plus `scripts/release.sh` and docs. **No app code.**++## Result++| | before | now |+|---|---|---|+| **CI wall clock** (warm) | 15-21 min | **6.0 min** (run 41336529771, all 6 required checks green) |+| **Release** (tag → published) | 20.9 min (v0.10.3, run 41331627109) | **about 1.5 min** when it promotes: locate + 5 smoke tests ran in 55 s (dry run 41338068684), plus publish (about 30 s) |+| Release that has to build (no Packages run for the commit) | 20.9 min | 16.6 min (dry run 41336601308; Flatpak x86_64 15.5 min is the long pole) |+| Packaged files tested before publishing | none | Windows installer + .zip, .dmg, .deb + AppImage (x86_64, arm64), .rpm |++The cold CI case (no cache, or after a change to a header everything includes): about 12-14 min (run 41335208645).+Today's old workflow took 15-21 min for the same.++## What changed, by request++### 1. ccache + parallel ctest + sharding, now in releases too++- **Where the build lives:** the release build matrix moved into `packages.yml`. That workflow runs on every push to+ main that changes the app, and release.yml can also call it.+- **Compiler cache:** every compiling job uses ccache. Linux uses Ubuntu's package; macOS, Windows and the Rocky+ containers download the ccache 4.14.2 release binary, checked against SHA-256.+ - Windows keeps the Visual Studio generator the release uses: ccache is put in as `cl.exe`, and MSBuild compiles+ one file per core.+ - Depend mode is on (`-MD` / `/showIncludes`), so a cache miss doesn't run the preprocessor twice.+- **Who can save the cache:** pull requests only read it. Pushes to main and maintainers' own dispatches save it.+ **Tags never save** (`SAVE_CACHE` is false for any `refs/tags/` run). A release that builds restores main's cache;+ GitHub lets a tag read only its own and the default branch's caches.+- **Parallel tests:** `sculptforge_tests --shard K/N` is registered as 8 ctest entries, each with its own temp folder.+ ctest runs at `-j 4`. Timed UI measurements and the scripts that use the clipboard run alone (`RUN_SERIAL`). The+ clipboard rule is detected from each .sfui, after an X11 BadWindow crash seen in CI.+- **Why cache restores into release builds are safe:** the cache can only be written by trusted runs (main pushes),+ ccache keys on the compiler binary, every flag and every include, and a hit is the object that compiler would+ produce. Release packages are still built from the tag's own source.++### 2. Runner variables++Every build and test job's `runs-on` reads a repository variable, falling back to the standard runner.++| variable | default |+|---|---|+| `SF_RUNNER_LINUX` | `ubuntu-24.04` |+| `SF_RUNNER_LINUX_ARM` | `ubuntu-24.04-arm` |+| `SF_RUNNER_LINUX_OLD` | `ubuntu-22.04` (.deb and AppImage, for the older glibc) |+| `SF_RUNNER_LINUX_OLD_ARM` | `ubuntu-22.04-arm` |+| `SF_RUNNER_MAC` | `macos-15` |+| `SF_RUNNER_WIN` | `windows-2022` |++- **Jobs pinned to GitHub's own runners:** CI `changes`, Release `locate`, the required-check result jobs, and+ `publish`.+- **Tokens:** `release.yml` is now `contents: read` at the top. **Only `publish` gets `contents: write`.** Before+ this, every release build job held a write token. So a third-party runner would only ever see a read-only token,+ and no build or test job has any secret.+- docs/releasing.md has the table.++### 3. Smoke tests of the packaged files before Publish++- **What runs:** `release.yml` job `smoke`, matrix of 5, each on a fresh runner:+ - Windows: NSIS installer run silently, and the .zip unpacked;+ - macOS: .dmg mounted, the app copied out, `codesign --verify`;+ - Linux x86_64 and arm64: `apt install ./*.deb`, and the AppImage extracted;+ - .rpm: `dnf install` in rockylinux:9.+- **Each install or unpack runs two checks:**+ - the GUI (`tests/ui/release_smoke.sfui`: drop the bust model, Clay stroke, can_undo / dirty);+ - `tests/release_smoke.py` on that package's `sculptforge-cli`: open, Clay stroke, save_project, reopen, then check+ the triangle count, that vertices moved, and that the reopened mesh is identical.+- **Verified:** every check ran and passed in the dry runs, macOS GUI included (Apple's software renderer). The+ Flatpak is still installed and UI-tested inside its own build job.+- **Gate:** publish needs smoke to succeed. Both test files also run under ctest (`release_smoke`,+ `ui_release_smoke`), so CI keeps them working.++### 4. scripts/release.sh and docs/releasing.md++- **Steps:** check → ci → tag → release → site → aur → brew → omarchy → replies → status. Run them all, or use+ `--from STEP` / `--only STEP`.+- **Idempotent:** each step checks live state and skips what's done (e.g. AUR already at that pkgver and checksum).+ On a failure it prints `STOPPED`, the exact `fix:`, and the resume command.+- **Checksums are computed here:**+ - AUR and Omarchy: the tag tarball is downloaded and hashed;+ - brew: the .dmg is downloaded, hashed and compared with SHA256SUMS before the tap update.+- **Writing it never does:** the feed highlights come from a file you hand it (`HIGHLIGHTS=`). For the "new in X.Y.Z"+ card it checks and stops with the fix. Issue replies are **drafts only** (`release-replies-X.Y.Z.md`); it never+ posts.+- **Outside commits:** made as islandmed noreply.+- **Tested** against the real repo, read-only, in a throwaway clone:+ - `check` passes at the v0.10.3 tag, and stops with the right fix when CMakeLists isn't bumped or a tag is+ elsewhere;+ - `release` verified v0.10.3's 35 assets;+ - `replies` drafted 7 replies.+- **Not run yet:** tag, site, aur, brew and omarchy. They push or publish, so they first run for real on the next+ release.++### Addendum items++- **(a) More free runners in parallel:** the CI tests are split with `ctest -I K,,N`: Windows ×3, Linux x86_64 ×2,+ ARM64 ×2. The shards together run every test once (checked: 3 × 29 = 87). One shard saves the cache.+ - **Required checks:** `Linux x86_64`, `Windows`, … are now tiny result jobs, so branch protection needs no change.+ - **Load:** CI is 13 jobs and Packages 11, both on a main push. That is at the free plan's 20 concurrent jobs, so+ a few may queue for a minute.+- **(b) Build once, promote:**+ - **Locate:** `locate` checks that the tag's CMakeLists version equals the tag. It then looks for a successful CI+ run and a successful Packages run for exactly that commit: event push, branch main, this repository, the right+ workflow file.+ - **Promote:** if both are there, it downloads that Packages run's artifacts, smoke-tests them and publishes. If+ not, it builds via packages.yml.+ - **Signing:** there is none today; macOS is ad-hoc signed inside CPack at build time, which survives promotion.+ Future SignPath / notarization would sign the promoted files between smoke and publish.+ - **Reproducibility:** the promoted packages are byte-for-byte the tested ones, which is stronger than+ rebuilding.+ - **Naming:** file names use the CMakeLists version, and locate fails a mismatched tag early (it used to fail late,+ at publish).+- **(c) Path filters:**+ - **CI:** a `changes` job reads the compare API. Pushes and PRs touching only `website/` (not `website/link/`,+ which is built into the app), pages/download-stats workflows, `packaging/arch/`, `tools/publish/`, the release+ script or this doc skip the build and test jobs. Their required checks pass as skipped.+ - **Tested** on real history: 5868d6f..27848d1 skips, the 0.10.3 range builds.+ - **Packages** uses the same list as a workflow path filter.+ - **The Flatpak PR check** (flatpak.yml) stays path-filtered. On main, Packages builds both Flatpaks every push,+ so its nightly run was removed.+- **(d) Blacksmith** and **(e) self-hosted:** see the runner section below.++## Paid runners (nothing signed up for or installed)++Facts were checked by a research agent on 2026-11-01; sources are in its notes. Prices are USD per minute.++- **Today's setup costs $0.** Standard runners are free and unlimited for this public repo: 20 concurrent jobs,+ 5 of them macOS. They have 4 vCPU for Linux and Windows, and 3-core M1 for macOS.+- **Every paid option requires an organization.**+ - GitHub larger runners: Team or Enterprise Cloud only ($4 / user / month for Team), billed per minute even on+ public repos. **A personal account like islandmed can't use them.**+ - Blacksmith: "limited to GitHub organizations and not available for personal repositories."+ - WarpBuild: describes only organization setup (personal accounts unverified).+ - So any paid runner means **transferring the repo to an organization**. GitHub redirects the URLs, but releases,+ the update feed, the brew tap, AUR sources and the Pages mirror would all need checking. That decision is+ Tony's.+- **Prices for runners bigger than the free ones** (only bigger ones are faster than free; 2-vCPU paid runners are+ slower):++| runner | GitHub larger | Blacksmith | WarpBuild |+|---|---|---|---|+| Linux x64 8 vCPU | $0.022 | $0.016* | $0.016 |+| Linux arm64 8 vCPU | $0.014 | $0.010* | $0.012 |+| Windows 8 vCPU | $0.042 | $0.032* (beta) | $0.032 |+| macOS | M2 Pro XL $0.102 | M4 6 vCPU $0.08 | M4 Pro $0.08 |++ \* scaled from Blacksmith's 2-vCPU prices; its page shows no per-size prices.+- **Free tiers:**+ - Blacksmith: 3,000 x64 2-vCPU minutes / month per org. A Windows minute counts as 2, a macOS minute as 20, so it+ covers about 150 macOS minutes. Its OSS program is by application.+ - WarpBuild: a one-time $10 credit.+- **Usage now** (from the last 8 days): about 440 CI runs and 300 main pushes a month (each push now also runs+ Packages). Warm job-minutes per run:+ - CI: Linux x64 17, arm64 7, Windows 14, macOS 3.5;+ - Packages: Linux x64 33 (Flatpak 15.5 of it), arm64 26, Windows 7.4, macOS 6.3.+- **Assumption for paid costs:** 8-vCPU runners take about 0.7× the job time (compile scales, llvmpipe UI tests+ only partly).++| scenario | per release commit (CI + Packages) | per month |+|---|---|---|+| Today, standard runners | $0, 6 min CI, 1.5 min release | $0 |+| Blacksmith 8 vCPU everywhere (+ M4 mac) | about $1.80 | about $650 (+ org) |+| GitHub larger 8-core everywhere | about $2.40 | about $860 + $4/user Team |+| Only Windows on Blacksmith 8 vCPU | about $0.48 | about $185 |++**What paying would buy now:** CI 6 → about 4 min; the release is already about 1.5 min when it promotes. The+remaining wait was Packages on main (Flatpak x86_64, 15.5 min), which a tag waits for before it can promote. Tony+approved the free fix and it is **done**: `packaging/flatpak/build.sh` takes `SF_FLATPAK_CCACHE=1` (flatpak-builder+`--ccache`, the SDK's own ccache), and packages.yml restores and saves `build-flatpak/state/ccache` (never from a+tag). Cold run 41339709154: both Flatpaks green (x86_64 12m14s), 216 MB caches saved. **The warm speed-up is not+measured yet** (it was time-boxed). Expected 4-5 min; the first main pushes after merge will show it.++**Decision (Tony, via the coordinator):** no paid runners. On a personal account every paid option needs an org transfer, and free runners are enough.++- **Blacksmith security (d):**+ - jobs run in Firecracker microVMs, one per job, destroyed afterwards; SOC 2 Type 2;+ - its GitHub App asks for write access to Contents, Workflows, Actions, Pull requests and Checks, which is broad.+ - Our build and test jobs carry only a read-only GITHUB_TOKEN and no secrets, and publish stays on GitHub. That is+ acceptable for build and test jobs only.+ - Its Windows runners are beta and give no edge over our free 3-way split. Its macOS M4 ($0.08/min) is the only+ clear speed-up, but macOS isn't our long pole.+- **Self-hosted on Tony's machine (e):**+ - **Cost:** free, and no platform fee for public repos (GitHub postponed the self-hosted fee).+ - **The risk:** GitHub says self-hosted runners "should almost never be used for public repositories". Any fork PR+ that gets approved, or slips past approval, runs its code on that machine.+ - **If ever used:**+ - only for jobs that can't come from a PR (push to main and tags);+ - an ephemeral, just-in-time runner inside a throwaway container or VM with no access to his home directory,+ keys or LAN;+ - fork PR approval set to "all external contributors";+ - a personal account has no runner groups, so only workflow-level `if:` guards keep PR jobs off it. One mistake+ in a workflow exposes the machine.+ - **Not recommended**, and not set up.++## Tests and verification++- **New tests:**+ - `tests/test_harness.cpp`: every test lands in exactly one shard;+ - `tests/release_smoke.py`, registered as ctest `release_smoke`;+ - `tests/ui/release_smoke.sfui`.+- **Local, rebased tree (aarch64 Release):** `sculptforge_tests` 663 tests, 0 failures; `ctest -j 4` 101/101 green,+ including every UI test.+- **`ui_idle_redraw` locally:** it fails some runs on this live desktop (25 frames in 4 s when the session is busy;+ its comment says pointer activity draws frames). It is unchanged, and it passes in CI every time.+- **CI and release runs:**+ - CI 41336529771: green, 6.0 min.+ - Release dry run 41336601308: builds, 5 smoke tests and publish skipped, all as designed.+ - Promote dry run 41338068684: 55 s.+ - On the final rebased head: CI 41338301432 green (15.4 min, mostly cold because main's 0.10.4 merge touched most+ files: the "header change" case), and release dry run 41338303633 green (builds 16.1 min, all 5 smoke tests,+ publish skipped).+- **Also found:** dry-running the old v0.10.3 tag through the new pipeline failed `ui_stroke_path`. Old tags lack the+ `RUN_SERIAL` markers, so the timed test ran under parallel load. Tags made after this merge carry them.+ Rebuilding a pre-merge tag with `gh workflow run release.yml -f tag=…` can hit that. Deliberately not "fixed"+ with retries.++## For the coordinator / Tony++1. **Merge** feat/cifast **right after v0.10.4 is tagged**, so 0.10.5 is the first release on the new pipeline. Then run `gh workflow run Flatpak` once; packages.yml and flatpak.yml only dispatch after+ they exist on main.+2. **The first main pushes after merge** run Packages cold (about 15-20 min, off the CI path). The release after that+ promotes.+3. Check that the second Packages run on main shows the warm Flatpak (about 4-5 min expected). If ccache in the SDK+ misbehaves, unset `SF_FLATPAK_CCACHE` in packages.yml (one line) to go back to a plain build.+4. **Optional:** change the required checks to the single set of result jobs. They already work as they are.+5. **Payment:** only if Tony wants it, and only after moving the repo to an organization; numbers above.