Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

fix(release): every bundle is launched before it ships, Windows links an ONNX Runtime any x86-64 can run, and the Windows tests can finish

Jareerd11848401b3b
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
new file mode 100644
index 00000000..d070ec43
--- /dev/null
+++ b/.github/dependabot.yml
@@ -0,0 +1,36 @@
+# Dependabot keeps two kinds of pin current, weekly, as pull requests that
+# CI tests like any other:
+#
+# - The actions every workflow uses, each pinned to a commit with its tag
+# in a comment. Dependabot moves the commit and the comment together.
+# - The Rust dependencies in src/Cargo.lock, grouped into one pull request
+# a week (security updates still come on their own, as they are found),
+# so a week's bumps are reviewed and tested once rather than twenty
+# times.
+#
+# Not covered: the prebuilt libraries in scripts/ci/deps.lock (FFmpeg,
+# ONNX Runtime, sherpa-onnx, the packagers, LLVM, Intel's emulator), which
+# are bumped by hand as that file says, and the toolchain in
+# src/rust-toolchain.toml.
+version: 2
+updates:
+ - package-ecosystem: github-actions
+ directory: /
+ schedule:
+ interval: weekly
+ groups:
+ actions:
+ patterns: ["*"]
+ commit-message:
+ prefix: ci
+
+ - package-ecosystem: cargo
+ directory: /src
+ schedule:
+ interval: weekly
+ open-pull-requests-limit: 5
+ groups:
+ cargo:
+ patterns: ["*"]
+ commit-message:
+ prefix: build
diff --git a/.github/workflows/build-app.yml b/.github/workflows/build-app.yml
index 996914bd..5e5299f5 100644
--- a/.github/workflows/build-app.yml
+++ b/.github/workflows/build-app.yml
@@ -30,6 +30,17 @@ name: Build app
# set on the repository, and ad-hoc signed when they are not, so a build
# from a fork still opens (with Gatekeeper's warning).
#
+# - Every bundle is launched before it is uploaded: `concat --version` and
+# `concat --smoke` from the staged folder, on a machine with nothing else
+# on the path (scripts/ci/smoke-*). On x86_64 the bundle is also started
+# under Intel's emulator posing as an Ivy Bridge, and its binaries are
+# scanned for instructions newer than AVX (scripts/ci/isa-scan.sh): the
+# x86_64 baseline is AVX, and twice now a release died at launch on
+# older CPUs (#193, #313). A target whose bundle does not start fails,
+# and release.yml publishes nothing.
+# - Every prebuilt dependency downloaded here is pinned to one file and its
+# digest in scripts/ci/deps.lock, and every action to a commit.
+#
# The six targets are a matrix so one fails without taking the rest down;
# `fail-fast` is off for the same reason.
on:
@@ -55,22 +66,37 @@ on:
APPLE_APP_PASSWORD:
description: An app-specific password for notarytool
required: false
+ WINDOWS_CERTIFICATE_PFX:
+ description: Base64 of an Authenticode code-signing .pfx
+ required: false
+ WINDOWS_CERTIFICATE_PASSWORD:
+ description: The .pfx's password
+ required: false
env:
CARGO_TERM_COLOR: always
- # BtbN's autobuilds: GPL (x264 in), shared, FFmpeg 8.1, one per arch.
- FFMPEG_RELEASE: https://github.com/BtbN/FFmpeg-Builds/releases/download/latest
- FFMPEG_VERSION: 8.1
- # ONNX Runtime, for the Linux bundles: the version the ort crate is built
- # against (see ort-sys's dist table), as Microsoft's own shared build.
+ # The prebuilt dependencies - BtbN's FFmpeg 8.1 (GPL, shared, one per
+ # arch), ONNX Runtime, sherpa-onnx, nfpm, appimagetool, LLVM for Windows
+ # and Intel's emulator - are each pinned to one file and its digest in
+ # scripts/ci/deps.lock, and fetched through scripts/ci/fetch.sh, which
+ # refuses a file whose digest has changed. The versions below are the
+ # ones those files carry, for the names inside them; a bump changes both.
+ #
+ # ONNX Runtime, for the Linux and Windows bundles: the version the ort
+ # crate is built against (see ort-sys's dist table), as Microsoft's own
+ # shared build.
ORT_VERSION: 1.28.0
# And for Intel Macs, which neither ort nor Microsoft build for any more:
- # the last release Microsoft made for them. The ort crate asks for the
- # runtime's API 17 and up, which this carries.
+ # the last release Microsoft made for them.
+ #
+ # NOTE: ort 2.0.0-rc.13 with its default features asks the runtime for
+ # API 27 (its `api-27` feature), and 1.23.2 serves API 23 and below, so
+ # on an Intel Mac the first ONNX session (a cutout, Chatterbox) should
+ # fail to get the API. That predates the pinning here, and wants a look
+ # on an Intel Mac: either an older `api-*` feature for ort, or no
+ # cutouts there.
ORT_INTEL_MAC_VERSION: 1.23.2
- # The Linux packagers: nfpm for the .deb and .rpm, appimagetool for the
- # AppImage; and WiX for the Windows .msi.
- NFPM_VERSION: 2.47.0
+ # WiX for the Windows .msi, as a dotnet tool, pinned by version.
WIX_VERSION: 6.0.2
# Oldest macOS the bundles run on. Apple silicon Macs all run newer than
# this; it is the Intel floor.
@@ -139,6 +165,9 @@ jobs:
target: x86_64-pc-windows-msvc
ffmpeg: win64
sherpa: win-x64
+ ort: win-x64
+ llvm: win64
+ vc: x64
inno: x64compatible
wix: x64
- platform: windows
@@ -147,16 +176,22 @@ jobs:
target: aarch64-pc-windows-msvc
ffmpeg: winarm64
sherpa: win-arm64
+ ort: win-arm64
+ llvm: woa64
+ vc: arm64
inno: arm64
wix: arm64
runs-on: ${{ matrix.os }}
- # The slowest target (Intel macOS, fat LTO) finishes in under half an
- # hour. A job still running past this is stuck, not slow.
- timeout-minutes: 90
+ # The slowest target (Intel macOS, fat LTO) builds in under half an
+ # hour, and the checks after the build add a few minutes. The steps
+ # that can hang have timeouts of their own, at most 115 minutes
+ # between them on any one target, so a stuck step is named by its own
+ # timeout; this is for a job stuck anywhere else.
+ timeout-minutes: 150
steps:
# The ref release.yml names, which is its tag: without one a run
# started by hand builds main and publishes it under the tag's name.
- - uses: actions/checkout@v7
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.ref }}
# The toolchain src/rust-toolchain.toml pins, installed by name, so the
@@ -165,7 +200,7 @@ jobs:
id: toolchain
shell: bash
run: echo "channel=$(sed -n 's/^channel = "\(.*\)"/\1/p' src/rust-toolchain.toml)" >> "$GITHUB_OUTPUT"
- - uses: dtolnay/rust-toolchain@master
+ - uses: dtolnay/rust-toolchain@e2a55d2ffb04f378e9626c28d38b36d230d1e12f # master (v1), 2026-10-09
with:
toolchain: ${{ steps.toolchain.outputs.channel }}
@@ -194,8 +229,7 @@ jobs:
if: matrix.platform == 'linux'
run: |
set -euo pipefail
- url="$FFMPEG_RELEASE/ffmpeg-n$FFMPEG_VERSION-latest-${{ matrix.ffmpeg }}-gpl-shared-$FFMPEG_VERSION.tar.xz"
- curl -fsSL --retry 5 --retry-all-errors -o ffmpeg.tar.xz "$url"
+ scripts/ci/fetch.sh "ffmpeg-${{ matrix.ffmpeg }}" ffmpeg.tar.xz
mkdir -p ffmpeg-dev
tar -xf ffmpeg.tar.xz -C ffmpeg-dev --strip-components=1
{
@@ -223,8 +257,7 @@ jobs:
run: |
set -euo pipefail
stem="onnxruntime-linux-${{ matrix.ort }}-$ORT_VERSION"
- curl -fsSL --retry 5 --retry-all-errors -o onnxruntime.tgz \
- "https://github.com/microsoft/onnxruntime/releases/download/v$ORT_VERSION/$stem.tgz"
+ scripts/ci/fetch.sh "onnxruntime-linux-${{ matrix.ort }}" onnxruntime.tgz
tar -xzf onnxruntime.tgz
test -f "$stem/lib/libonnxruntime.so.$ORT_VERSION"
# The tarball carries the soname and the full version; the linker
@@ -267,8 +300,7 @@ jobs:
run: |
set -euo pipefail
stem="onnxruntime-osx-x86_64-$ORT_INTEL_MAC_VERSION"
- curl -fsSL --retry 5 --retry-all-errors -o onnxruntime.tgz \
- "https://github.com/microsoft/onnxruntime/releases/download/v$ORT_INTEL_MAC_VERSION/$stem.tgz"
+ scripts/ci/fetch.sh onnxruntime-osx-x86_64 onnxruntime.tgz
tar -xzf onnxruntime.tgz
test -f "$stem/lib/libonnxruntime.$ORT_INTEL_MAC_VERSION.dylib"
ln -sf "libonnxruntime.$ORT_INTEL_MAC_VERSION.dylib" "$stem/lib/libonnxruntime.dylib"
@@ -279,10 +311,12 @@ jobs:
} >> "$GITHUB_ENV"
# ── Windows ────────────────────────────────────────────────────────
- # LLVM: libclang for bindgen on both architectures, and the C/C++
- # compiler on arm64. Chocolatey's LLVM is x64; the arm64 runner takes
- # LLVM's own Windows-on-Arm installer. The BtbN zip unpacks to one
- # top-level folder.
+ # LLVM: libclang for bindgen on both architectures, the C/C++
+ # compiler on arm64, and llvm-objdump for the instruction-set scan.
+ # LLVM's own installer for the architecture, at the version deps.lock
+ # pins, into a folder of its own rather than over the LLVM the runner
+ # image may already carry. The BtbN zip unpacks to one top-level
+ # folder.
#
# On arm64 the whole C side is compiled by clang-cl rather than cl:
# ggml refuses MSVC for ARM outright (its NEON kernels need clang),
@@ -299,31 +333,83 @@ jobs:
# crate into those flags - spelt with a dash, because the build
# step runs under Git's bash, whose MSYS layer rewrites a value
# that starts with a slash into a Windows path.
- - name: FFmpeg development libraries (Windows)
+ #
+ # fetch.sh runs under Git's bash, named by its full path: a bare
+ # `bash` from PowerShell can be WSL's.
+ - name: FFmpeg development libraries and LLVM (Windows)
if: matrix.platform == 'windows'
+ timeout-minutes: 15
shell: pwsh
run: |
$ErrorActionPreference = "Stop"
+ $bash = "$env:ProgramFiles\Git\bin\bash.exe"
+ & $bash scripts/ci/fetch.sh "llvm-${{ matrix.llvm }}" llvm.exe
+ if ($LASTEXITCODE -ne 0) { throw "the LLVM download failed" }
+ Start-Process -Wait -FilePath .\llvm.exe -ArgumentList "/S", "/D=C:\llvm"
+ if (-not (Test-Path C:\llvm\bin\libclang.dll)) { throw "LLVM did not install to C:\llvm" }
+ Remove-Item llvm.exe
if ("${{ matrix.arch }}" -eq "aarch64") {
- $llvm = "21.1.0"
- curl.exe -fsSL --retry 5 --retry-all-errors -o llvm.exe "https://github.com/llvm/llvm-project/releases/download/llvmorg-$llvm/LLVM-$llvm-woa64.exe"
- Start-Process -Wait -FilePath .\llvm.exe -ArgumentList "/S"
"CC=clang-cl" | Out-File -FilePath $env:GITHUB_ENV -Append
"CXX=clang-cl" | Out-File -FilePath $env:GITHUB_ENV -Append
"CXXFLAGS=-EHsc" | Out-File -FilePath $env:GITHUB_ENV -Append
"CMAKE_GENERATOR=Ninja" | Out-File -FilePath $env:GITHUB_ENV -Append
- } else {
- choco install llvm -y --no-progress
}
- "C:\Program Files\LLVM\bin" | Out-File -FilePath $env:GITHUB_PATH -Append
- "LIBCLANG_PATH=C:\Program Files\LLVM\bin" | Out-File -FilePath $env:GITHUB_ENV -Append
- $url = "$env:FFMPEG_RELEASE/ffmpeg-n$env:FFMPEG_VERSION-latest-${{ matrix.ffmpeg }}-gpl-shared-$env:FFMPEG_VERSION.zip"
- curl.exe -fsSL --retry 5 --retry-all-errors -o ffmpeg-win.zip "$url"
+ "C:\llvm\bin" | Out-File -FilePath $env:GITHUB_PATH -Append
+ "LIBCLANG_PATH=C:\llvm\bin" | Out-File -FilePath $env:GITHUB_ENV -Append
+ & $bash scripts/ci/fetch.sh "ffmpeg-${{ matrix.ffmpeg }}" ffmpeg-win.zip
+ if ($LASTEXITCODE -ne 0) { throw "the FFmpeg download failed" }
Expand-Archive ffmpeg-win.zip -DestinationPath ffdl
$dir = (Get-ChildItem ffdl -Directory | Select-Object -First 1).FullName
"FFMPEG_DIR=$dir" | Out-File -FilePath $env:GITHUB_ENV -Append
"$dir\bin" | Out-File -FilePath $env:GITHUB_PATH -Append
+ # ONNX Runtime on Windows: Microsoft's shared build, linked
+ # dynamically, and onnxruntime.dll shipped beside concat.exe.
+ #
+ # Left to itself ort-sys links pyke's static build into concat.exe,
+ # and that build is compiled for Haswell from end to end: BMI2
+ # shifts in ordinary code, run at start-up, so 0.2.5 and 0.2.6 died
+ # with STATUS_ILLEGAL_INSTRUCTION on every CPU before 2013 (#313).
+ # Microsoft's build keeps its general code to the baseline and picks
+ # AVX2 kernels at run time. pyke's is also the DirectML build, which
+ # imports DirectML.dll when the program loads - and the one Windows
+ # keeps in System32 is too old on many machines, so the program would
+ # not start at all ("DMLCreateDevice1 not found", #303).
+ #
+ # Why the CPU build and not Microsoft's DirectML one: two runtimes
+ # cannot share a process under one name, and sherpa-onnx's DLL loads
+ # onnxruntime.dll too. sherpa-onnx 1.13.7 is built against ONNX
+ # Runtime 1.27.1 and asks for its API 27 as it loads; ort 2.0.0-rc.13
+ # asks for API 27 as well (its default `api-27` feature). Microsoft
+ # stopped publishing the DirectML build at 1.24.4, which serves API
+ # 24 and below, so it can serve neither. 1.28.0 serves both. The cost
+ # is that the cutouts and Chatterbox run on the CPU on Windows: the
+ # DirectML provider ort asks for is not in this runtime, and ort
+ # carries on without it (it logs the failed registration and falls
+ # back; error_on_failure is not set). Getting the GPU back means a
+ # DirectML (or WebGPU) runtime at API 27 or later that is built for
+ # the baseline - pyke's is neither the one nor shareable with
+ # sherpa-onnx - and no DirectML.dll is shipped until then, since
+ # nothing in the bundle imports it.
+ #
+ # The DLL's folder goes on the path too, so anything the build runs
+ # that links it (a build script's helper, say) finds it.
+ - name: ONNX Runtime (Windows)
+ if: matrix.platform == 'windows'
+ shell: pwsh
+ run: |
+ $ErrorActionPreference = "Stop"
+ & "$env:ProgramFiles\Git\bin\bash.exe" scripts/ci/fetch.sh "onnxruntime-${{ matrix.ort }}" onnxruntime.zip
+ if ($LASTEXITCODE -ne 0) { throw "the ONNX Runtime download failed" }
+ Expand-Archive onnxruntime.zip -DestinationPath .
+ $dir = "$PWD\onnxruntime-${{ matrix.ort }}-$env:ORT_VERSION"
+ if (-not (Test-Path "$dir\lib\onnxruntime.lib")) { throw "$dir\lib\onnxruntime.lib is not there" }
+ if (-not (Test-Path "$dir\lib\onnxruntime.dll")) { throw "$dir\lib\onnxruntime.dll is not there" }
+ "ORT_LIB_LOCATION=$dir\lib" | Out-File -FilePath $env:GITHUB_ENV -Append
+ "ORT_PREFER_DYNAMIC_LINK=1" | Out-File -FilePath $env:GITHUB_ENV -Append
+ "ORT_DIR=$dir" | Out-File -FilePath $env:GITHUB_ENV -Append
+ "$dir\lib" | Out-File -FilePath $env:GITHUB_PATH -Append
+
# ── sherpa-onnx ────────────────────────────────────────────────────
# The prebuilt library the sys crate would otherwise fetch for
# itself, fetched here instead and handed over through
@@ -332,17 +418,22 @@ jobs:
# cache then replays the build script's answer without the files
# it points at, and the crate fails to link the moment anything
# makes it recompile. Outside target/, the files are there on every
- # run. The version is the one the lockfile resolved, so a bump
- # cannot desynchronise.
+ # run. The version is the one the lockfile resolved, and deps.lock
+ # must pin that version's file: a bump to one without the other
+ # stops here, saying so.
- name: sherpa-onnx static libraries (macOS and Linux)
if: matrix.platform != 'windows'
shell: bash
run: |
set -euo pipefail
version=$(grep -A1 'name = "sherpa-onnx-sys"' src/Cargo.lock | sed -n 's/^version = "\(.*\)"/\1/p')
+ name="sherpa-onnx-${{ matrix.sherpa }}-static-lib"
+ case "$(scripts/ci/fetch.sh --url "$name")" in
+ */v"$version"/*) ;;
+ *) echo "scripts/ci/deps.lock pins $name for another version than Cargo.lock's $version" >&2; exit 1 ;;
+ esac
stem="sherpa-onnx-v$version-${{ matrix.sherpa }}-static-lib"
- echo "Downloading $stem"
- curl -fsSL --retry 5 --retry-all-errors -o sherpa.tar.bz2 "https://github.com/k2-fsa/sherpa-onnx/releases/download/v$version/$stem.tar.bz2"
+ scripts/ci/fetch.sh "$name" sherpa.tar.bz2
mkdir -p sherpa
tar -xjf sherpa.tar.bz2 -C sherpa
test -d "sherpa/$stem/lib"
@@ -358,6 +449,10 @@ jobs:
# DLLs are kept apart and the packaging step ships them beside the
# binary.
#
+ # sherpa-onnx's own onnxruntime.dll and its import library are left
+ # out: the bundle carries one ONNX Runtime, the one above, which
+ # serves sherpa-onnx and ort alike (see that step).
+ #
# The archive is .tar.bz2, and Windows' own tar has no bzip2 of its
# own: it pipes through an external decoder, and on the x64 image
# that pipe can hang without a word. Python's tarfile unpacks it
@@ -369,29 +464,34 @@ jobs:
run: |
$ErrorActionPreference = "Stop"
$version = (Select-String -Path src\Cargo.lock -Pattern 'name = "sherpa-onnx-sys"' -Context 0,1).Context.PostContext[0] -replace '.*"(.*)".*', '$1'
+ $name = "sherpa-onnx-${{ matrix.sherpa }}-shared-MT-Release-lib"
$stem = "sherpa-onnx-v$version-${{ matrix.sherpa }}-shared-MT-Release-lib"
+ $bash = "$env:ProgramFiles\Git\bin\bash.exe"
+ $url = & $bash scripts/ci/fetch.sh --url $name
+ if ($url -notmatch "/v$([regex]::Escape($version))/") { throw "scripts/ci/deps.lock pins $name for another version than Cargo.lock's $version" }
Write-Host "Downloading $stem"
- curl.exe -fsSL --retry 5 --retry-all-errors -o sherpa.tar.bz2 "https://github.com/k2-fsa/sherpa-onnx/releases/download/v$version/$stem.tar.bz2"
+ & $bash scripts/ci/fetch.sh $name sherpa.tar.bz2
+ if ($LASTEXITCODE -ne 0) { throw "the sherpa-onnx download failed" }
Write-Host "Unpacking"
python -c "import tarfile; t = tarfile.open('sherpa.tar.bz2'); t.extractall(**({'filter': 'data'} if hasattr(tarfile, 'data_filter') else {}))"
if (-not (Test-Path "$stem\lib")) { throw "$stem\lib is not there after unpacking" }
New-Item -ItemType Directory -Force sherpa-libs, sherpa-dlls | Out-Null
- Copy-Item "$stem\lib\*.lib" sherpa-libs\
- Copy-Item "$stem\lib\*.dll" sherpa-dlls\
+ Copy-Item "$stem\lib\sherpa-onnx-*.lib" sherpa-libs\
+ Copy-Item "$stem\lib\sherpa-onnx-*.dll" sherpa-dlls\
"SHERPA_ONNX_LIB_DIR=$PWD\sherpa-libs" | Out-File -FilePath $env:GITHUB_ENV -Append
"SHERPA_DLL_DIR=$PWD\sherpa-dlls" | Out-File -FilePath $env:GITHUB_ENV -Append
# ── Build ──────────────────────────────────────────────────────────
# The cache comes after the environment is set, not before: its key
# is made from the compiler and linker variables in force (RUSTFLAGS,
- # CC, CXX, CMAKE_*, and GGML_* added to the default set), so a change
- # to any of them is a new key rather than a stale target/ served
- # under the old one.
- - uses: Swatinem/rust-cache@v2
+ # CC, CXX, CMAKE_*, and GGML_* and ORT_* added to the default set),
+ # so a change to any of them is a new key rather than a stale
+ # target/ served under the old one.
+ - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: src
key: ${{ matrix.target }}
- env-vars: CARGO CC CFLAGS CXX CMAKE RUST GGML
+ env-vars: CARGO CC CFLAGS CXX CMAKE RUST GGML ORT
# The shipping profile: fat LTO, panic=abort, stripped. See the
# workspace Cargo.toml for what each knob costs.
@@ -400,6 +500,7 @@ jobs:
# read, but the step summary is public - so on failure the tail of the
# build goes there, and a red run explains itself to anyone.
- name: Build
+ timeout-minutes: 60
shell: bash
working-directory: src
run: |
@@ -544,6 +645,14 @@ jobs:
fi
mkdir -p stage && mv "$dmg" stage/
+ # The bundle that went into the .dmg, launched: its signature holds,
+ # nothing in it loads from Homebrew, and `--version` and `--smoke`
+ # exit 0. See scripts/ci/smoke-macos.sh.
+ - name: Smoke test (macOS)
+ if: matrix.platform == 'macos'
+ timeout-minutes: 10
+ run: scripts/ci/smoke-macos.sh Concat.app
+
# ── Package: Linux ─────────────────────────────────────────────────
# The binary, the FFmpeg runtime libraries it loads from lib/ beside
# it (the rpath above), a desktop entry and an icon, in one folder.
@@ -587,6 +696,42 @@ jobs:
exit 1
fi
+ # The staged folder, launched on this machine with nothing but the
+ # system and the folder to load from: `--version`, then `--smoke`
+ # under Xvfb with every file it opens traced, so a start-up that so
+ # much as reaches for CUDA or NVENC fails. On x86_64, `--version`
+ # again under Intel's emulator posing as an Ivy Bridge. See
+ # scripts/ci/smoke-linux.sh.
+ - name: Smoke test (Linux)
+ if: matrix.platform == 'linux'
+ timeout-minutes: 15
+ env:
+ VERSION: ${{ steps.version.outputs.version }}
+ run: |
+ set -euo pipefail
+ sudo apt-get install -y xvfb xauth strace libgl1-mesa-dri mesa-vulkan-drivers
+ args=()
+ if [ "${{ matrix.arch }}" = x86_64 ]; then
+ scripts/ci/fetch.sh sde-lin sde.tar.xz
+ mkdir -p sde
+ tar -xJf sde.tar.xz -C sde --strip-components=1
+ args+=(--sde "$PWD/sde/sde64")
+ fi
+ scripts/ci/smoke-linux.sh "${args[@]}" "stage/Concat-${VERSION}-linux-${{ matrix.arch }}"
+
+ # Every x86_64 binary in the folder, disassembled and counted for
+ # instructions an Ivy Bridge does not have, against what
+ # scripts/ci/isa-allow.txt allows each one. See scripts/ci/isa-scan.sh.
+ - name: Instruction-set scan (Linux x86_64)
+ if: matrix.platform == 'linux' && matrix.arch == 'x86_64'
+ timeout-minutes: 15
+ env:
+ VERSION: ${{ steps.version.outputs.version }}
+ run: |
+ set -euo pipefail
+ sudo apt-get install -y llvm
+ scripts/ci/isa-scan.sh "stage/Concat-${VERSION}-linux-${{ matrix.arch }}"
+
# Arch Linux package, from the same folder as the .deb and .rpm.
# Arch ships x86_64 only, so aarch64 comes from Arch Linux ARM.
- name: Package (Arch Linux)
@@ -653,8 +798,7 @@ jobs:
chmod +x "$STAGE/pkg/concat"
sed 's|^Exec=concat|Exec=/opt/concat/concat %U|' "$STAGE/concat.desktop" > "$STAGE/pkg/concat.desktop"
- curl -fsSL --retry 5 --retry-all-errors -o nfpm.tar.gz \
- "https://github.com/goreleaser/nfpm/releases/download/v$NFPM_VERSION/nfpm_${NFPM_VERSION}_Linux_$( [ "${{ matrix.arch }}" = x86_64 ] && echo x86_64 || echo arm64 ).tar.gz"
+ scripts/ci/fetch.sh "nfpm-$( [ "${{ matrix.arch }}" = x86_64 ] && echo x86_64 || echo arm64 )" nfpm.tar.gz
mkdir -p nfpm && tar -xzf nfpm.tar.gz -C nfpm
# From inside the folder: nfpm names files relative to where it
# runs, and expands no variables in their paths.
@@ -684,31 +828,179 @@ jobs:
cp "$STAGE/concat.png" "$appdir/usr/share/icons/hicolor/256x256/apps/concat.png"
ln -s concat.png "$appdir/.DirIcon"
ln -s usr/bin/concat "$appdir/AppRun"
- curl -fsSL --retry 5 --retry-all-errors -o appimagetool \
- "https://github.com/AppImage/appimagetool/releases/download/continuous/appimagetool-$ARCH.AppImage"
+ # The tool and the runtime it puts at the head of the AppImage,
+ # both a named release: left to itself the tool downloads the
+ # runtime's `continuous` build as it runs.
+ scripts/ci/fetch.sh "appimagetool-$ARCH" appimagetool
+ scripts/ci/fetch.sh "appimage-runtime-$ARCH" appimage-runtime
chmod +x appimagetool
# Extract-and-run: the runner has no FUSE to mount the tool with.
# No "linux" in its name: every AppImage is for Linux, and the
# AppImage catalog's check warns about names that say so.
appimage="stage/Concat-${VERSION}-${{ matrix.arch }}.AppImage"
- ./appimagetool --appimage-extract-and-run "$appdir" "$appimage"
+ ./appimagetool --appimage-extract-and-run --runtime-file appimage-runtime "$appdir" "$appimage"
chmod +x "$appimage"
rm -r "$STAGE" "$appdir"
ls -l stage
+ # The .deb and the AppImage in two clean distributions, the oldest
+ # and a Debian: apt installs the .deb with exactly the dependencies it
+ # declares, and both start (`--version`, and `--smoke` under Xvfb).
+ # A library the package forgot to depend on fails here rather than
+ # on somebody's machine. See scripts/ci/smoke-linux.sh.
+ - name: Smoke test in clean distributions (Linux)
+ if: matrix.platform == 'linux'
+ timeout-minutes: 25
+ env:
+ VERSION: ${{ steps.version.outputs.version }}
+ run: |
+ set -euo pipefail
+ for image in ubuntu:22.04 debian:12; do
+ scripts/ci/smoke-linux.sh --container "$image" \
+ "stage/Concat-${VERSION}-linux-${{ matrix.arch }}.deb" \
+ "stage/Concat-${VERSION}-${{ matrix.arch }}.AppImage"
+ done
+
# ── Package: Windows ───────────────────────────────────────────────
- # Authenticode-signed with the repository's certificate when it has
- # one - the program, then each installer - so SmartScreen and the
- # UAC prompt name the publisher instead of warning about an unknown
- # one; unsigned otherwise, which is what a fork's build gets.
+ # The folder the installers are made from: the program, FFmpeg's
+ # DLLs, sherpa-onnx's, ONNX Runtime's (one runtime for both; see its
+ # step above) and the Visual C++ runtime the program and its DLLs
+ # import. The C++ runtime is copied in from Visual Studio's
+ # redistributable folder, as Microsoft allows: a clean Windows does
+ # not have it, and a machine that does may have an older one.
- name: Stage (Windows)
+ if: matrix.platform == 'windows'
+ shell: pwsh
+ env:
+ VERSION: ${{ steps.version.outputs.version }}
+ run: |
+ $ErrorActionPreference = "Stop"
+ $name = "Concat-$env:VERSION-windows-${{ matrix.arch }}"
+ $dir = "stage\$name"
+ New-Item -ItemType Directory -Force $dir | Out-Null
+ Copy-Item "src\target\${{ matrix.target }}\app\concat.exe" $dir
+ Copy-Item "$env:FFMPEG_DIR\bin\*.dll" $dir
+ # The sherpa-onnx runtime, set aside by the download step above.
+ Copy-Item "$env:SHERPA_DLL_DIR\*.dll" $dir
+ Copy-Item "$env:ORT_DIR\lib\*.dll" $dir
+ Copy-Item LICENSE, THIRD_PARTY_NOTICES.md $dir
+ # The Visual C++ runtime DLLs anything in the folder imports, from
+ # the newest redistributable folder Visual Studio carries.
+ $vs = & "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe" -latest -products * -property installationPath
+ $crt = Get-ChildItem "$vs\VC\Redist\MSVC\*\${{ matrix.vc }}\Microsoft.VC14*.CRT" -Directory |
+ Sort-Object FullName -Descending | Select-Object -First 1
+ if (-not $crt) { throw "no Visual C++ redistributable folder for ${{ matrix.vc }} under $vs" }
+ $dumpbin = & "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe" -latest -products * -find "VC\Tools\MSVC\**\bin\Host*\*\dumpbin.exe" | Select-Object -First 1
+ # Twice over: msvcp140.dll, once copied, imports vcruntime140.dll.
+ foreach ($pass in 1, 2) {
+ foreach ($pe in Get-ChildItem $dir -File -Include *.exe, *.dll -Recurse) {
+ foreach ($line in & $dumpbin /nologo /dependents $pe.FullName) {
+ $dll = $line.Trim()
+ if ($dll -match "\.dll$" -and (Test-Path "$($crt.FullName)\$dll") -and -not (Test-Path "$dir\$dll")) {
+ Write-Host "$($pe.Name) imports $dll; copying it from $($crt.FullName)"
+ Copy-Item "$($crt.FullName)\$dll" $dir
+ }
+ }
+ }
+ }
+ Get-ChildItem $dir | Format-Table Name, Length
+
+ # Authenticode, with the repository's certificate when it has one:
+ # every program and DLL in the folder that does not already carry a
+ # valid signature (Microsoft's ONNX Runtime does, and keeps it), here,
+ # and the installers in the step after the checks. Unsigned files
+ # are what Smart App Control refuses to load ("Bad Image", 0xc0e90002,
+ # #306, #311), and an unsigned DLL is refused even beside a signed
+ # program. SmartScreen and the UAC prompt name the publisher instead
+ # of warning about an unknown one. A fork's build is unsigned.
+ #
+ # The certificate is imported into the user's store and signtool is
+ # told its thumbprint, so the password never reaches a command line:
+ # it goes from the environment to the import as a SecureString. The
+ # store is emptied of it again in the last step, whatever happens.
+ - name: Sign (Windows)
if: matrix.platform == 'windows'
shell: pwsh
env:
VERSION: ${{ steps.version.outputs.version }}
CERT_PFX: ${{ secrets.WINDOWS_CERTIFICATE_PFX }}
CERT_PASSWORD: ${{ secrets.WINDOWS_CERTIFICATE_PASSWORD }}
+ run: |
+ $ErrorActionPreference = "Stop"
+ if (-not $env:CERT_PFX) {
+ Write-Host "No signing certificate: the build is unsigned."
+ exit 0
+ }
+ $pfx = "$env:RUNNER_TEMP\cert.pfx"
+ [IO.File]::WriteAllBytes($pfx, [Convert]::FromBase64String($env:CERT_PFX))
+ $password = ConvertTo-SecureString $env:CERT_PASSWORD -AsPlainText -Force
+ $flags = [Security.Cryptography.X509Certificates.X509KeyStorageFlags]"UserKeySet, PersistKeySet"
+ $cert = [Security.Cryptography.X509Certificates.X509Certificate2]::new($pfx, $password, $flags)
+ Remove-Item -Force $pfx
+ if (-not $cert.HasPrivateKey) { throw "the certificate came in without its private key" }
+ $store = [Security.Cryptography.X509Certificates.X509Store]::new("My", "CurrentUser")
+ $store.Open("ReadWrite")
+ $store.Add($cert)
+ $store.Close()
+ $signtool = Get-ChildItem "${env:ProgramFiles(x86)}\Windows Kits\10\bin\*\x64\signtool.exe" |
+ Sort-Object FullName -Descending | Select-Object -First 1
+ if (-not $signtool) { throw "signtool.exe is not in the Windows SDK" }
+ "SIGNTOOL=$($signtool.FullName)" | Out-File -FilePath $env:GITHUB_ENV -Append
+ "SIGNING_THUMBPRINT=$($cert.Thumbprint)" | Out-File -FilePath $env:GITHUB_ENV -Append
+ $dir = "stage\Concat-$env:VERSION-windows-${{ matrix.arch }}"
+ $unsigned = Get-ChildItem $dir -Recurse -File -Include *.exe, *.dll |
+ Where-Object { (Get-AuthenticodeSignature $_.FullName).Status -ne "Valid" }
+ foreach ($file in $unsigned) {
+ & $signtool.FullName sign /sha1 $cert.Thumbprint /s My /fd SHA256 `
+ /tr http://timestamp.digicert.com /td SHA256 /d Concat $file.FullName
+ if ($LASTEXITCODE -ne 0) { throw "signtool failed on $($file.Name)" }
+ }
+ Write-Host "Signed $(@($unsigned).Count) files."
+
+ # The staged folder, launched before any installer is made of it:
+ # every DLL it imports is in it or is Windows' own; `--version` and
+ # `--smoke` exit 0 with only Windows on the path; and on x86_64,
+ # `--version` again under Intel's emulator posing as an Ivy Bridge
+ # (#313). See scripts/ci/smoke-windows.ps1.
+ - name: Smoke test (Windows)
+ if: matrix.platform == 'windows'
+ timeout-minutes: 15
+ shell: pwsh
+ env:
+ VERSION: ${{ steps.version.outputs.version }}
+ run: |
+ $ErrorActionPreference = "Stop"
+ $sde = ""
+ if ("${{ matrix.arch }}" -eq "x86_64") {
+ & "$env:ProgramFiles\Git\bin\bash.exe" scripts/ci/fetch.sh sde-win sde.tar.xz
+ if ($LASTEXITCODE -ne 0) { throw "the SDE download failed" }
+ New-Item -ItemType Directory -Force sde | Out-Null
+ tar -xf sde.tar.xz -C sde --strip-components=1
+ if ($LASTEXITCODE -ne 0) { throw "unpacking SDE failed" }
+ $sde = "sde\sde.exe"
+ }
+ scripts/ci/smoke-windows.ps1 -Bundle "stage\Concat-$env:VERSION-windows-${{ matrix.arch }}" -Sde $sde
+
+ # Every x86_64 binary in the folder, disassembled and counted for
+ # instructions an Ivy Bridge does not have, against what
+ # scripts/ci/isa-allow.txt allows each one. See scripts/ci/isa-scan.sh.
+ - name: Instruction-set scan (Windows x86_64)
+ if: matrix.platform == 'windows' && matrix.arch == 'x86_64'
+ timeout-minutes: 15
+ shell: bash
+ env:
+ VERSION: ${{ steps.version.outputs.version }}
+ # llvm-objdump from the LLVM installed above, on the path.
+ run: scripts/ci/isa-scan.sh "stage/Concat-${VERSION}-windows-${{ matrix.arch }}"
+
+ # The installers, from the checked folder: Inno Setup's .exe and
+ # WiX's .msi, each signed when the folder was.
+ - name: Installers (Windows)
+ if: matrix.platform == 'windows'
+ shell: pwsh
+ env:
+ VERSION: ${{ steps.version.outputs.version }}
# The identity Partner Center reserved for the app. Unset, no
# .msix is made: a fork has no Store listing to make one for.
MSSTORE_IDENTITY_NAME: ${{ vars.MSSTORE_IDENTITY_NAME }}
@@ -718,29 +1010,12 @@ jobs:
run: |
$ErrorActionPreference = "Stop"
$name = "Concat-$env:VERSION-windows-${{ matrix.arch }}"
- $pfx = $null
- if ($env:CERT_PFX) {
- $pfx = "$env:RUNNER_TEMP\cert.pfx"
- [IO.File]::WriteAllBytes($pfx, [Convert]::FromBase64String($env:CERT_PFX))
- $signtool = Get-ChildItem "${env:ProgramFiles(x86)}\Windows Kits\10\bin\*\x64\signtool.exe" |
- Sort-Object FullName -Descending | Select-Object -First 1
- if (-not $signtool) { throw "signtool.exe is not in the Windows SDK" }
- } else {
- Write-Host "No signing certificate: the build is unsigned."
- }
function Sign($file) {
- if (-not $pfx) { return }
- & $signtool.FullName sign /f $pfx /p $env:CERT_PASSWORD /fd SHA256 `
+ if (-not $env:SIGNING_THUMBPRINT) { return }
+ & $env:SIGNTOOL sign /sha1 $env:SIGNING_THUMBPRINT /s My /fd SHA256 `
/tr http://timestamp.digicert.com /td SHA256 /d Concat $file
if ($LASTEXITCODE -ne 0) { throw "signtool failed on $file" }
}
- New-Item -ItemType Directory -Force "stage\$name" | Out-Null
- Copy-Item "src\target\${{ matrix.target }}\app\concat.exe" "stage\$name\"
- Sign "stage\$name\concat.exe"
- Copy-Item "$env:FFMPEG_DIR\bin\*.dll" "stage\$name\"
- # The sherpa-onnx runtime, set aside by the download step above.
- Copy-Item "$env:SHERPA_DLL_DIR\*.dll" "stage\$name\"
- Copy-Item LICENSE, THIRD_PARTY_NOTICES.md "stage\$name\"
# The installer, from that folder: Inno Setup, which the x64
# image carries and the ARM image is given. See
# assets/windows/concat.iss.
@@ -765,7 +1040,6 @@ jobs:
if ($LASTEXITCODE -ne 0) { throw "WiX failed with $LASTEXITCODE" }
if (-not (Test-Path "stage\$name.msi")) { throw "stage\$name.msi is not there" }
Sign "stage\$name.msi"
- if ($pfx) { Remove-Item -Force $pfx }
# And the .msix, for the Microsoft Store, from the same folder with
# the manifest and logos beside it. See assets/windows/AppxManifest.xml.
# Into store/, not stage/: the Store signs it and takes it, and an
@@ -803,7 +1077,14 @@ jobs:
# The folder itself is not published: the two installers are.
Remove-Item -Recurse -Force "stage\$name"
- - uses: actions/upload-artifact@v7
+ # The signing certificate out of the user's store again, whether the
+ # steps above passed or not.
+ - name: Forget the certificate (Windows)
+ if: always() && matrix.platform == 'windows' && env.SIGNING_THUMBPRINT != ''
+ shell: pwsh
+ run: Remove-Item -Force -DeleteKey "Cert:\CurrentUser\My\$env:SIGNING_THUMBPRINT"
+
+ - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: Concat-${{ matrix.platform }}-${{ matrix.arch }}
path: stage/*
@@ -811,7 +1092,7 @@ jobs:
# The Store's package, apart from the release's bundles: release.yml
# bundles the two architectures and submits them.
- - uses: actions/upload-artifact@v7
+ - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
if: matrix.platform == 'windows' && hashFiles('store/*.msix') != ''
with:
name: Store-windows-${{ matrix.arch }}
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 7284a1af..7ac73be7 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -2,6 +2,12 @@ name: CI
# The engine's tests and lints on every push and pull request. The bundles
# are built by release.yml, on a tag; see that file.
+#
+# Every action is pinned to a commit, with the tag it was in a comment
+# beside it: a tag can be moved to other code after a run has trusted it,
+# a commit cannot. Dependabot proposes the bumps (.github/dependabot.yml).
+# The prebuilt libraries the jobs download are pinned the same way, by
+# digest, in scripts/ci/deps.lock.
on:
push:
branches: [main]
@@ -18,21 +24,27 @@ on:
default: ""
# Keyed on what is tested, so a release testing its tag and a push to
-# main never cancel each other.
+# main never cancel each other. Only a pull request's run is cancelled by
+# the next push to it: a run on main is the record of whether main is
+# green, and cancelling it leaves a gap where the answer should be.
concurrency:
group: ci-${{ inputs.ref || github.ref }}
- cancel-in-progress: true
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
engine:
name: Engine tests and lints
runs-on: ubuntu-latest
+ # The steps that can run long have timeouts of their own, 110 minutes
+ # between them, so a stuck step is named by its own timeout; this is
+ # for a job stuck anywhere else. A warm run takes about twenty minutes.
+ timeout-minutes: 130
env:
- FFMPEG_LINUX: https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-linux64-gpl-shared-8.1.tar.xz
# See build-app.yml: the version the ort crate is built against.
+ # scripts/ci/deps.lock pins the file.
ORT_VERSION: 1.28.0
steps:
- - uses: actions/checkout@v7
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.ref }}
# The toolchain src/rust-toolchain.toml pins, installed by name: every
@@ -41,13 +53,22 @@ jobs:
- name: The pinned toolchain
id: toolchain
run: echo "channel=$(sed -n 's/^channel = "\(.*\)"/\1/p' src/rust-toolchain.toml)" >> "$GITHUB_OUTPUT"
- - uses: dtolnay/rust-toolchain@master
+ - uses: dtolnay/rust-toolchain@e2a55d2ffb04f378e9626c28d38b36d230d1e12f # master (v1), 2026-10-09
with:
toolchain: ${{ steps.toolchain.outputs.channel }}
components: clippy, rustfmt
- - uses: Swatinem/rust-cache@v2
+ # cargo-nextest runs the tests: each in a process of its own, with a
+ # timeout per test and a JUnit report. See src/.config/nextest.toml.
+ - uses: taiki-e/install-action@ab6895351a88b908e96a4697860d4ffea5e57dbe # v2.87.27
+ with:
+ # Saved even when the job fails: a cache saved only by a green run
+ # is never saved while the job is red, and every red run then
+ # compiles from nothing.
+ - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: src
+ cache-on-failure: true
# Slint's Skia renderer links fontconfig and freetype; cpal links
# ALSA; bindgen needs libclang; whisper.cpp is built by cmake; the
@@ -67,7 +88,7 @@ jobs:
- name: FFmpeg development libraries
run: |
set -euo pipefail
- curl -fsSL --retry 5 --retry-all-errors -o ffmpeg.tar.xz "$FFMPEG_LINUX"
+ scripts/ci/fetch.sh ffmpeg-linux64 ffmpeg.tar.xz
mkdir -p ffmpeg-dev
tar -xf ffmpeg.tar.xz -C ffmpeg-dev --strip-components=1
echo "FFMPEG_DIR=$PWD/ffmpeg-dev" >> "$GITHUB_ENV"
@@ -80,9 +101,9 @@ jobs:
run: |
set -euo pipefail
stem="onnxruntime-linux-x64-$ORT_VERSION"
- curl -fsSL --retry 5 --retry-all-errors -o onnxruntime.tgz \
- "https://github.com/microsoft/onnxruntime/releases/download/v$ORT_VERSION/$stem.tgz"
+ scripts/ci/fetch.sh onnxruntime-linux-x64 onnxruntime.tgz
tar -xzf onnxruntime.tgz
+ test -f "$stem/lib/libonnxruntime.so.$ORT_VERSION"
ln -sf "libonnxruntime.so.$ORT_VERSION" "$stem/lib/libonnxruntime.so"
{
echo "ORT_LIB_LOCATION=$PWD/$stem/lib"
@@ -94,14 +115,19 @@ jobs:
# below fetch theirs. Left to its own build script, sherpa-onnx-sys
# unpacks it into target/sherpa-onnx-prebuilt, which rust-cache does
# not keep while the build script's run is kept: a cache hit then
- # links against a folder that is not there.
+ # links against a folder that is not there. The version is the
+ # lockfile's, and deps.lock must pin that version's file.
- name: sherpa-onnx libraries
run: |
set -euo pipefail
version=$(grep -A1 'name = "sherpa-onnx-sys"' src/Cargo.lock | sed -n 's/^version = "\(.*\)"/\1/p')
+ name=sherpa-onnx-linux-x64-static-lib
+ case "$(scripts/ci/fetch.sh --url "$name")" in
+ */v"$version"/*) ;;
+ *) echo "scripts/ci/deps.lock pins $name for another version than Cargo.lock's $version" >&2; exit 1 ;;
+ esac
stem="sherpa-onnx-v$version-linux-x64-static-lib"
- curl -fsSL --retry 5 --retry-all-errors -o sherpa.tar.bz2 \
- "https://github.com/k2-fsa/sherpa-onnx/releases/download/v$version/$stem.tar.bz2"
+ scripts/ci/fetch.sh "$name" sherpa.tar.bz2
mkdir -p sherpa
tar -xjf sherpa.tar.bz2 -C sherpa
test -d "sherpa/$stem/lib"
@@ -123,25 +149,69 @@ jobs:
# before tests: they finish in a fraction of the time and catch the
# most.
- run: cargo clippy --workspace --all-targets -- -D warnings
+ timeout-minutes: 25
+ working-directory: src
+
+ # The tests are compiled in a step of their own, so a slow compile
+ # and a slow test are told apart by which step ran out of time. One
+ # job at a time and no debug info, to stay inside the runner's
+ # memory; the run below finds everything already built.
+ - name: Build the tests
+ timeout-minutes: 25
working-directory: src
+ env:
+ CARGO_PROFILE_DEV_DEBUG: "0"
+ run: cargo nextest run --workspace --no-run --build-jobs 1
# CONCAT_REQUIRE_GPU: the GPU tests fail rather than skip when no
- # adapter is usable, so the parity suite is known to have run.
- - run: cargo test --workspace -j 1
+ # adapter is usable, so the parity suite is known to have run. The
+ # report and the log are kept when it fails (the last step).
+ - name: Run the tests
+ timeout-minutes: 20
working-directory: src
env:
CARGO_PROFILE_DEV_DEBUG: "0"
CONCAT_REQUIRE_GPU: "1"
+ run: |
+ status=0
+ cargo nextest run --workspace --build-jobs 1 --profile ci 2>&1 | tee nextest-workspace.log || status=$?
+ mv target/nextest/ci/junit.xml target/nextest/ci/junit-workspace.xml 2>/dev/null || true
+ exit "$status"
+ # nextest does not run the examples in the documentation; cargo does.
+ - name: Run the documentation's examples
+ timeout-minutes: 10
+ working-directory: src
+ env:
+ CARGO_PROFILE_DEV_DEBUG: "0"
+ run: cargo test --workspace --doc -j 1
# The performance table's quick scenarios against their budgets, in
# the `quick` profile: planning, undo, the document, publishing.
# The media scenarios need a machine that is not a shared runner.
- run: cargo run -p concat-perf --profile quick -- --check --quick
+ timeout-minutes: 10
working-directory: src
# The gRPC transport is a feature nothing else in the workspace turns
# on, so it is linted and tested here on its own. See concat-server.
- run: cargo clippy -p concat-server -p concat-cli --features grpc --all-targets -- -D warnings
+ timeout-minutes: 10
working-directory: src
- - run: cargo test -p concat-server --features grpc
+ - name: Run the gRPC tests
+ timeout-minutes: 10
working-directory: src
+ run: |
+ status=0
+ cargo nextest run -p concat-server --features grpc --profile ci 2>&1 | tee nextest-grpc.log || status=$?
+ mv target/nextest/ci/junit.xml target/nextest/ci/junit-grpc.xml 2>/dev/null || true
+ exit "$status"
+
+ - name: Keep the test report (failure only)
+ if: failure()
+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+ with:
+ name: test-report-linux
+ path: |
+ src/target/nextest/ci/*.xml
+ src/nextest-*.log
+ if-no-files-found: ignore
# The engine crates that carry no native library build for the web. This
# job is what keeps that true: a std::thread or a blocking wait that
@@ -150,22 +220,24 @@ jobs:
web:
name: Engine on wasm32
runs-on: ubuntu-latest
+ timeout-minutes: 30
steps:
- - uses: actions/checkout@v7
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.ref }}
- name: The pinned toolchain
id: toolchain
run: echo "channel=$(sed -n 's/^channel = "\(.*\)"/\1/p' src/rust-toolchain.toml)" >> "$GITHUB_OUTPUT"
- - uses: dtolnay/rust-toolchain@master
+ - uses: dtolnay/rust-toolchain@e2a55d2ffb04f378e9626c28d38b36d230d1e12f # master (v1), 2026-10-09
with:
toolchain: ${{ steps.toolchain.outputs.channel }}
components: clippy
targets: wasm32-unknown-unknown
- - uses: Swatinem/rust-cache@v2
+ - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: src
key: wasm32
+ cache-on-failure: true
- run: >
cargo clippy
-p concat-core -p concat-project -p concat-effects -p concat-render -p concat-text
@@ -196,80 +268,172 @@ jobs:
sherpa: win-x64
verb: Tests
runs-on: ${{ matrix.os }}
- # A compile is minutes and the Windows suite under an hour; a job past
- # this is stuck, and a stuck job is what kept main red for five days.
- timeout-minutes: 120
+ # Windows' steps that can run long have timeouts of their own, 115
+ # minutes between them, so whichever is stuck is named by its own
+ # timeout. A stuck job kept main red for a week once: the tests' one
+ # step held the compile and every test, ran into its ninety minutes
+ # every time, and said nothing about which test it was waiting on.
+ timeout-minutes: 135
env:
- FFMPEG_WINDOWS: https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-win64-gpl-shared-8.1.zip
+ # No debug info in the test binaries, as on Linux: smaller, faster
+ # to link, and nothing reads it here.
+ CARGO_PROFILE_DEV_DEBUG: "0"
+ ORT_VERSION: 1.28.0
steps:
- - uses: actions/checkout@v7
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.ref }}
- name: The pinned toolchain
id: toolchain
shell: bash
run: echo "channel=$(sed -n 's/^channel = "\(.*\)"/\1/p' src/rust-toolchain.toml)" >> "$GITHUB_OUTPUT"
- - uses: dtolnay/rust-toolchain@master
+ - uses: dtolnay/rust-toolchain@e2a55d2ffb04f378e9626c28d38b36d230d1e12f # master (v1), 2026-10-09
with:
toolchain: ${{ steps.toolchain.outputs.channel }}
+ - uses: taiki-e/install-action@ab6895351a88b908e96a4697860d4ffea5e57dbe # v2.87.27
+ if: runner.os == 'Windows'
+ with:
- name: FFmpeg (macOS)
if: runner.os == 'macOS'
run: |
set -euo pipefail
brew install ffmpeg cmake
echo "FFMPEG_DIR=$(brew --prefix ffmpeg)" >> "$GITHUB_ENV"
- - name: FFmpeg and LLVM (Windows)
+ # LLVM's own installer, at the version deps.lock pins, into a folder
+ # of its own: the runner image carries an LLVM of its own, at
+ # whatever version the image was made with, which this neither
+ # upgrades nor depends on.
+ - name: LLVM (Windows)
if: runner.os == 'Windows'
+ timeout-minutes: 10
shell: pwsh
run: |
$ErrorActionPreference = "Stop"
- choco install llvm -y --no-progress
- "C:\Program Files\LLVM\bin" | Out-File -FilePath $env:GITHUB_PATH -Append
- "LIBCLANG_PATH=C:\Program Files\LLVM\bin" | Out-File -FilePath $env:GITHUB_ENV -Append
- curl.exe -fsSL --retry 5 --retry-all-errors -o ffmpeg-win.zip "$env:FFMPEG_WINDOWS"
+ & "$env:ProgramFiles\Git\bin\bash.exe" scripts/ci/fetch.sh llvm-win64 llvm.exe
+ if ($LASTEXITCODE -ne 0) { throw "the LLVM download failed" }
+ Start-Process -Wait -FilePath .\llvm.exe -ArgumentList "/S", "/D=C:\llvm"
+ if (-not (Test-Path C:\llvm\bin\libclang.dll)) { throw "LLVM did not install to C:\llvm" }
+ Remove-Item llvm.exe
+ "C:\llvm\bin" | Out-File -FilePath $env:GITHUB_PATH -Append
+ "LIBCLANG_PATH=C:\llvm\bin" | Out-File -FilePath $env:GITHUB_ENV -Append
+ - name: FFmpeg (Windows)
+ if: runner.os == 'Windows'
+ shell: pwsh
+ run: |
+ $ErrorActionPreference = "Stop"
+ & "$env:ProgramFiles\Git\bin\bash.exe" scripts/ci/fetch.sh ffmpeg-win64 ffmpeg-win.zip
+ if ($LASTEXITCODE -ne 0) { throw "the FFmpeg download failed" }
Expand-Archive ffmpeg-win.zip -DestinationPath ffdl
$dir = (Get-ChildItem ffdl -Directory | Select-Object -First 1).FullName
"FFMPEG_DIR=$dir" | Out-File -FilePath $env:GITHUB_ENV -Append
"$dir\bin" | Out-File -FilePath $env:GITHUB_PATH -Append
+ # ONNX Runtime the way the Windows bundle links it (build-app.yml says
+ # why): Microsoft's shared build, through ORT_LIB_LOCATION, so the
+ # tests run against the runtime the app ships rather than the static
+ # one ort-sys would download. On the path for the test binaries.
+ - name: ONNX Runtime (Windows)
+ if: runner.os == 'Windows'
+ shell: pwsh
+ run: |
+ $ErrorActionPreference = "Stop"
+ & "$env:ProgramFiles\Git\bin\bash.exe" scripts/ci/fetch.sh onnxruntime-win-x64 onnxruntime.zip
+ if ($LASTEXITCODE -ne 0) { throw "the ONNX Runtime download failed" }
+ Expand-Archive onnxruntime.zip -DestinationPath .
+ $lib = "$PWD\onnxruntime-win-x64-$env:ORT_VERSION\lib"
+ if (-not (Test-Path "$lib\onnxruntime.lib")) { throw "$lib\onnxruntime.lib is not there" }
+ "ORT_LIB_LOCATION=$lib" | Out-File -FilePath $env:GITHUB_ENV -Append
+ "ORT_PREFER_DYNAMIC_LINK=1" | Out-File -FilePath $env:GITHUB_ENV -Append
+ $lib | Out-File -FilePath $env:GITHUB_PATH -Append
# The prebuilt sherpa-onnx library, as build-app.yml fetches it:
- # static on macOS, the import libraries alone on Windows.
+ # static on macOS, the import libraries alone on Windows, whose DLLs
+ # go on the path for the test binaries - all but sherpa's own
+ # onnxruntime.dll, so the one the tests load is the one above.
- name: sherpa-onnx libraries
shell: bash
run: |
set -euo pipefail
version=$(grep -A1 'name = "sherpa-onnx-sys"' src/Cargo.lock | sed -n 's/^version = "\(.*\)"/\1/p')
if [ "$RUNNER_OS" = "Windows" ]; then
- stem="sherpa-onnx-v$version-${{ matrix.sherpa }}-shared-MT-Release-lib"
+ kind=shared-MT-Release-lib
else
- stem="sherpa-onnx-v$version-${{ matrix.sherpa }}-static-lib"
+ kind=static-lib
fi
- echo "Downloading $stem"
- curl -fsSL --retry 5 --retry-all-errors -o sherpa.tar.bz2 "https://github.com/k2-fsa/sherpa-onnx/releases/download/v$version/$stem.tar.bz2"
+ name="sherpa-onnx-${{ matrix.sherpa }}-$kind"
+ stem="sherpa-onnx-v$version-${{ matrix.sherpa }}-$kind"
+ case "$(scripts/ci/fetch.sh --url "$name")" in
+ */v"$version"/*) ;;
+ *) echo "scripts/ci/deps.lock pins $name for another version than Cargo.lock's $version" >&2; exit 1 ;;
+ esac
+ scripts/ci/fetch.sh "$name" sherpa.tar.bz2
mkdir -p sherpa
python3 -c "import tarfile; t = tarfile.open('sherpa.tar.bz2'); t.extractall('sherpa', **({'filter': 'data'} if hasattr(tarfile, 'data_filter') else {}))"
test -d "sherpa/$stem/lib"
if [ "$RUNNER_OS" = "Windows" ]; then
mkdir -p sherpa-libs sherpa-dlls
- cp "sherpa/$stem/lib/"*.lib sherpa-libs/
- # The DLLs too: the test binaries load them as they start.
- cp "sherpa/$stem/lib/"*.dll sherpa-dlls/
+ cp "sherpa/$stem/lib/"sherpa-onnx-*.lib sherpa-libs/
+ cp "sherpa/$stem/lib/"sherpa-onnx-*.dll sherpa-dlls/
echo "SHERPA_ONNX_LIB_DIR=$(pwd -W)\\sherpa-libs" >> "$GITHUB_ENV"
echo "$(pwd -W)\\sherpa-dlls" >> "$GITHUB_PATH"
else
echo "SHERPA_ONNX_LIB_DIR=$PWD/sherpa/$stem/lib" >> "$GITHUB_ENV"
fi
- - uses: Swatinem/rust-cache@v2
+ - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: src
key: check-${{ matrix.os }}
+ cache-on-failure: true
- run: cargo check --workspace --all-targets
if: runner.os != 'Windows'
+ timeout-minutes: 45
working-directory: src
- # One job at a time and no debug info, as on Linux. The timeout is
- # for a hang: concat-media's tests once hung here for good.
- - run: cargo test --workspace -j 1
+
+ # Windows: compile, then run, then the one scenario that has hung
+ # here on its own. Each step has its own timeout, so a red run says
+ # whether the compile was slow, a test failed, or a test hung, and a
+ # test that hangs is killed by nextest after its slow-timeout and
+ # reported by name (src/.config/nextest.toml). Two compile jobs: the
+ # runner has the memory for two, and one took most of an hour.
+ - name: Build the tests (Windows)
+ id: build-tests
if: runner.os == 'Windows'
- timeout-minutes: 90
+ timeout-minutes: 60
working-directory: src
- env:
- CARGO_PROFILE_DEV_DEBUG: "0"
+ shell: bash
+ run: cargo nextest run --workspace --no-run --build-jobs 2
+ - name: Run the tests (Windows)
+ if: runner.os == 'Windows'
+ timeout-minutes: 30
+ working-directory: src
+ shell: bash
+ run: |
+ status=0
+ cargo nextest run --workspace --build-jobs 2 --profile ci \
+ -E 'not test(=every_edit_still_exports)' 2>&1 | tee nextest-workspace.log || status=$?
+ mv target/nextest/ci/junit.xml target/nextest/ci/junit-workspace.xml 2>/dev/null || true
+ exit "$status"
+ # every_edit_still_exports passes on Linux in under a minute and has
+ # hung on Windows. Alone, with its output shown as it comes: each
+ # export names itself before it starts and once it is written, so
+ # the last line before a hang is the edit, and the half of it, that
+ # never finished. Runs whether or not the rest passed.
+ - name: The editing session alone (Windows)
+ if: ${{ !cancelled() && runner.os == 'Windows' && steps.build-tests.outcome == 'success' }}
+ timeout-minutes: 15
+ working-directory: src
+ shell: bash
+ run: |
+ status=0
+ cargo nextest run -p concat-host --test export --build-jobs 2 --profile ci --no-capture \
+ -E 'test(=every_edit_still_exports)' 2>&1 | tee nextest-every-edit.log || status=$?
+ mv target/nextest/ci/junit.xml target/nextest/ci/junit-every-edit.xml 2>/dev/null || true
+ exit "$status"
+ - name: Keep the test report (failure only)
+ if: failure() && runner.os == 'Windows'
+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+ with:
+ name: test-report-windows
+ path: |
+ src/target/nextest/ci/*.xml
+ src/nextest-*.log
+ if-no-files-found: ignore
diff --git a/.github/workflows/mobile.yml b/.github/workflows/mobile.yml
index 758a6116..e36d2cac 100644
--- a/.github/workflows/mobile.yml
+++ b/.github/workflows/mobile.yml
@@ -22,8 +22,9 @@ name: Build for phones
#
# The APK is built under a key made for the run, and a release (the `sign`
# input, which release.yml alone passes) signs it over again with the
-# repository's key, ANDROID_KEYSTORE (the keystore, base64) and
-# ANDROID_KEYSTORE_PASSWORD, in a step of its own after the build: the key
+# repository's key, ANDROID_KEYSTORE (the keystore, base64),
+# ANDROID_KEYSTORE_PASSWORD, and optionally ANDROID_KEY_ALIAS and
+# ANDROID_KEY_PASSWORD, in a step of its own after the build: the key
# never sits in the environment of the build and the several hundred build
# scripts it runs, and a pull request's run never sees it at all. The .ipa
# is ad-hoc signed: a tester signs it with their own Apple ID on the way
@@ -45,15 +46,22 @@ on:
required: false
ANDROID_KEYSTORE_PASSWORD:
required: false
+ ANDROID_KEY_ALIAS:
+ description: The key's alias in the keystore; the keystore's first alias when unset
+ required: false
+ ANDROID_KEY_PASSWORD:
+ description: The key's own password; the keystore's when unset
+ required: false
pull_request:
workflow_dispatch:
# Keyed by the calling workflow too, so a call from a release never shares
-# a group with a pull request's run. A tag's release is never cancelled;
-# everything else is superseded by the next push.
+# a group with a pull request's run. Only a pull request's run is
+# superseded by the next push to it; a release's, or one started by hand,
+# runs to the end.
concurrency:
group: mobile-${{ github.workflow }}-${{ github.ref }}
- cancel-in-progress: ${{ !startsWith(github.ref, 'refs/tags/') }}
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
CARGO_TERM_COLOR: always
@@ -69,7 +77,7 @@ jobs:
steps:
# The ref release.yml names, which is its tag: without one a run
# started by hand builds main and publishes it under the tag's name.
- - uses: actions/checkout@v7
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.ref }}
- name: Read the version
@@ -83,11 +91,11 @@ jobs:
- name: The pinned toolchain
id: toolchain
run: echo "channel=$(sed -n 's/^channel = "\(.*\)"/\1/p' src/rust-toolchain.toml)" >> "$GITHUB_OUTPUT"
- - uses: dtolnay/rust-toolchain@master
+ - uses: dtolnay/rust-toolchain@e2a55d2ffb04f378e9626c28d38b36d230d1e12f # master (v1), 2026-10-09
with:
toolchain: ${{ steps.toolchain.outputs.channel }}
targets: aarch64-linux-android
- - uses: taiki-e/cache-cargo-install-action@v2
+ - uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2.3.1
with:
tool: cargo-apk
@@ -103,7 +111,7 @@ jobs:
# build after it.
- name: FFmpeg for Android (cached)
id: ffmpeg
- uses: actions/cache/restore@v4
+ uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: src/vendor/ffmpeg/aarch64-linux-android
key: ffmpeg-aarch64-linux-android-${{ hashFiles('src/scripts/ffmpeg-mobile.sh') }}
@@ -112,7 +120,7 @@ jobs:
run: src/scripts/ffmpeg-mobile.sh aarch64-linux-android
- name: FFmpeg for Android (save)
if: steps.ffmpeg.outputs.cache-hit != 'true'
- uses: actions/cache/save@v4
+ uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: src/vendor/ffmpeg/aarch64-linux-android
key: ${{ steps.ffmpeg.outputs.cache-primary-key }}
@@ -120,7 +128,7 @@ jobs:
- name: sherpa-onnx for Android
run: src/scripts/sherpa-mobile.sh aarch64-linux-android
- - uses: Swatinem/rust-cache@v2
+ - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: src
key: aarch64-linux-android
@@ -155,11 +163,20 @@ jobs:
# removed once the signature is on. apksigner replaces the run's
# signature rather than adding to it. A release with no key set keeps
# the run's key and says so in the run's annotations.
+ #
+ # apksigner is told everything it would otherwise stop and ask for:
+ # which key (ANDROID_KEY_ALIAS, or the keystore's first alias, which
+ # keytool lists), and the key's password (ANDROID_KEY_PASSWORD, or
+ # the keystore's, which is what a key made with the keystore has).
+ # A prompt on a runner is a step that waits for its timeout.
- name: Sign with the repository's key
if: inputs.sign
+ timeout-minutes: 10
env:
KEYSTORE_B64: ${{ secrets.ANDROID_KEYSTORE }}
KEYSTORE_PASSWORD: ${{ secrets.ANDROID_KEYSTORE_PASSWORD }}
+ KEY_ALIAS: ${{ secrets.ANDROID_KEY_ALIAS }}
+ KEY_PASSWORD: ${{ secrets.ANDROID_KEY_PASSWORD || secrets.ANDROID_KEYSTORE_PASSWORD }}
run: |
set -euo pipefail
if [ -z "$KEYSTORE_B64" ]; then
@@ -168,10 +185,22 @@ jobs:
fi
keystore=$RUNNER_TEMP/release.keystore
echo "$KEYSTORE_B64" | base64 --decode > "$keystore"
+ alias=$KEY_ALIAS
+ if [ -z "$alias" ]; then
+ alias=$(keytool -list -keystore "$keystore" -storepass:env KEYSTORE_PASSWORD 2>/dev/null \
+ | awk -F, '/PrivateKeyEntry/ { print $1; exit }')
+ if [ -z "$alias" ]; then
+ rm -f "$keystore"
+ echo "::error::the keystore holds no private key, or its password is wrong; set ANDROID_KEY_ALIAS"
+ exit 1
+ fi
+ echo "ANDROID_KEY_ALIAS is not set; signing with the keystore's first key, $alias"
+ fi
sdk=${ANDROID_HOME:-$ANDROID_SDK_ROOT}
- tools=$(ls -d "$sdk"/build-tools/*/ | sort -V | tail -1)
+ tools=$(find "$sdk/build-tools" -mindepth 1 -maxdepth 1 -type d | sort -V | tail -1)
apk=src/target/release/apk/Concat.apk
- "$tools/apksigner" sign --ks "$keystore" --ks-pass env:KEYSTORE_PASSWORD "$apk"
+ "$tools/apksigner" sign --ks "$keystore" --ks-pass env:KEYSTORE_PASSWORD \
+ --ks-key-alias "$alias" --key-pass env:KEY_PASSWORD "$apk" < /dev/null
rm -f "$keystore"
"$tools/apksigner" verify --print-certs "$apk"
@@ -197,7 +226,7 @@ jobs:
cp src/target/release/apk/Concat.apk "stage/Concat-$VERSION-android-arm64.apk"
ls -l stage
- - uses: actions/upload-artifact@v7
+ - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: Concat-android-arm64
path: stage/*
@@ -208,7 +237,7 @@ jobs:
runs-on: macos-15
timeout-minutes: 90
steps:
- - uses: actions/checkout@v7
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.ref }}
- name: Read the version
@@ -220,7 +249,7 @@ jobs:
- name: The pinned toolchain
id: toolchain
run: echo "channel=$(sed -n 's/^channel = "\(.*\)"/\1/p' src/rust-toolchain.toml)" >> "$GITHUB_OUTPUT"
- - uses: dtolnay/rust-toolchain@master
+ - uses: dtolnay/rust-toolchain@e2a55d2ffb04f378e9626c28d38b36d230d1e12f # master (v1), 2026-10-09
with:
toolchain: ${{ steps.toolchain.outputs.channel }}
targets: aarch64-apple-ios
@@ -232,7 +261,7 @@ jobs:
# build after it.
- name: FFmpeg for iOS (cached)
id: ffmpeg
- uses: actions/cache/restore@v4
+ uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: src/vendor/ffmpeg/aarch64-apple-ios
key: ffmpeg-aarch64-apple-ios-${{ hashFiles('src/scripts/ffmpeg-mobile.sh') }}
@@ -241,7 +270,7 @@ jobs:
run: src/scripts/ffmpeg-mobile.sh aarch64-apple-ios
- name: FFmpeg for iOS (save)
if: steps.ffmpeg.outputs.cache-hit != 'true'
- uses: actions/cache/save@v4
+ uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
with:
path: src/vendor/ffmpeg/aarch64-apple-ios
key: ${{ steps.ffmpeg.outputs.cache-primary-key }}
@@ -249,7 +278,7 @@ jobs:
- name: sherpa-onnx for iOS
run: src/scripts/sherpa-mobile.sh aarch64-apple-ios
- - uses: Swatinem/rust-cache@v2
+ - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
workspaces: src
key: aarch64-apple-ios
@@ -294,7 +323,7 @@ jobs:
(cd payload && ditto -c -k --keepParent Payload "../stage/Concat-$VERSION-ios-arm64.ipa")
ls -l stage
- - uses: actions/upload-artifact@v7
+ - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: Concat-ios-arm64
path: stage/*
diff --git a/.github/workflows/models.yml b/.github/workflows/models.yml
index c40f5529..54e42aa6 100644
--- a/.github/workflows/models.yml
+++ b/.github/workflows/models.yml
@@ -39,10 +39,13 @@ jobs:
mirror:
name: Fill and publish
runs-on: ubuntu-latest
+ # Two gigabytes down and two up, which takes a runner well under an
+ # hour; past three, something is stuck.
+ timeout-minutes: 180
permissions:
contents: write
steps:
- - uses: actions/checkout@v7
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: The table
id: table
@@ -91,7 +94,7 @@ jobs:
echo "patch=true" >> "$GITHUB_OUTPUT"
fi
- - uses: actions/upload-artifact@v7
+ - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
if: steps.recorded.outputs.patch == 'true'
with:
name: recorded-digests
diff --git a/.github/workflows/nix.yml b/.github/workflows/nix.yml
index 57ae1a0c..453f8114 100644
--- a/.github/workflows/nix.yml
+++ b/.github/workflows/nix.yml
@@ -23,19 +23,25 @@ on:
- .github/workflows/nix.yml
workflow_dispatch:
+# Only a pull request's run is superseded by the next push to it; a run on
+# main is the record of whether main builds.
concurrency:
group: nix-${{ github.ref }}
- cancel-in-progress: true
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
build:
name: nix build
runs-on: ubuntu-latest
+ # A cold build of the whole workspace and its native dependencies.
+ timeout-minutes: 120
+ # Read only: the job builds what is checked in and writes nothing back
+ # (the lock is refreshed by hand; see the top of this file).
permissions:
- contents: write
+ contents: read
steps:
- - uses: actions/checkout@v7
- - uses: DeterminateSystems/nix-installer-action@v22
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+ - uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 # v22
# Refreshes nothing: a lock that would change here is a lock the
# commit should have carried.
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 4f9f1ae8..86ee3707 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -14,10 +14,18 @@ name: Release
# (.deb, .rpm, .AppImage, Arch) and the Windows installers carry the
# FFmpeg runtime libraries beside the binary; the Android .apk is signed
# with the repository's key, and the iOS .ipa is signed by the tester on
-# the way onto the phone. Nothing is built until CI has passed on the tag.
+# the way onto the phone. Nothing is built until CI has passed on the tag,
+# and nothing is published until every bundle has been launched where it
+# was built: build-app.yml starts each one (`--version`, `--smoke`, and on
+# x86_64 under an emulated Ivy Bridge) before it uploads it, and a bundle
+# that does not start fails its job, which `publish` needs.
+#
+# The release is created as a draft. A person looks it over - the notes,
+# the files - and publishes it from the Releases page; nothing reaches
+# the updater or the downloads page before then.
#
# A release that is not a pre-release also goes to the Microsoft Store,
-# once it is published here, when the repository is set up for it; see
+# once its draft is made here, when the repository is set up for it; see
# the store job.
on:
push:
@@ -36,16 +44,33 @@ jobs:
check:
name: Tag matches the version
runs-on: ubuntu-latest
+ timeout-minutes: 10
outputs:
tag: ${{ steps.tag.outputs.tag }}
version: ${{ steps.tag.outputs.version }}
steps:
- - uses: actions/checkout@v7
+ # The tag a person typed is checked before anything uses it: as the
+ # ref to check out, in a script, or in a file name.
+ - name: The tag is a version tag
+ env:
+ TAG: ${{ inputs.tag || github.ref_name }}
+ run: |
+ if ! [[ "$TAG" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$ ]]; then
+ echo "::error::'$TAG' is not a version tag (v1.2.3, or v1.2.3-alpha.1)"
+ exit 1
+ fi
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ inputs.tag || github.ref }}
+ # The tag comes in through the environment, never pasted into the
+ # script: a value in ${{ }} is spliced into the shell's text before
+ # bash reads it, so a tag typed into the form by hand could have run
+ # as a command.
- id: tag
+ env:
+ TAG: ${{ inputs.tag || github.ref_name }}
run: |
- tag="${{ inputs.tag || github.ref_name }}"
+ tag="$TAG"
version=$(sed -n 's/^version = "\(.*\)"/\1/p' src/Cargo.toml | head -1)
# The tag is the version, or the version with a pre-release
# suffix: v0.2.1 and v0.2.1-alpha.22 both belong to 0.2.1.
@@ -95,11 +120,12 @@ jobs:
name: Publish release
needs: [check, build, phones]
runs-on: ubuntu-latest
+ timeout-minutes: 30
permissions:
contents: write
steps:
# The bundles only: the Store's packages are the store job's.
- - uses: actions/download-artifact@v8
+ - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: bundles
pattern: Concat-*
@@ -107,7 +133,7 @@ jobs:
# The tree the release is cut from: the notes are read out of its
# history, and the manifest out of its model table.
- - uses: actions/checkout@v7
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ needs.check.outputs.tag }}
fetch-depth: 0
@@ -118,11 +144,14 @@ jobs:
# fetched from. An updater or a package script reads this instead of
# guessing at asset names. See scripts/models.py.
- name: Write the manifest
+ env:
+ VERSION: ${{ needs.check.outputs.version }}
+ TAG: ${{ needs.check.outputs.tag }}
run: |
set -euo pipefail
python3 source/scripts/models.py release-manifest \
- --version "${{ needs.check.outputs.version }}" \
- --tag "${{ needs.check.outputs.tag }}" \
+ --version "$VERSION" \
+ --tag "$TAG" \
--bundles bundles \
--out bundles/manifest.json
cat bundles/manifest.json
@@ -134,22 +163,25 @@ jobs:
ls -l
sha256sum -- * > SHA256SUMS
- # The notes are the tag's section of CHANGELOG.md, or the commits
- # since the previous tag where it has none; see the script.
+ # The notes are the commits since the previous tag; see the script.
- name: Write the notes
id: notes
+ env:
+ TAG: ${{ needs.check.outputs.tag }}
run: |
set -euo pipefail
cd source
- tag="${{ needs.check.outputs.tag }}"
+ tag="$TAG"
previous=$(git describe --tags --abbrev=0 --match 'v*' "$tag^" 2>/dev/null || git rev-list --max-parents=0 HEAD | tail -1)
.github/release-notes.sh "$previous" "$tag" "${tag#v}" > "$RUNNER_TEMP/notes.md"
cat "$RUNNER_TEMP/notes.md"
- - uses: softprops/action-gh-release@v3
+ # A draft: a person publishes it. See the top of this file.
+ - uses: softprops/action-gh-release@efb35369e0ad2afab669f228072c1b0d510eae64 # v3.0.3
with:
tag_name: ${{ needs.check.outputs.tag }}
name: Concat ${{ needs.check.outputs.tag }}
+ draft: true
prerelease: ${{ contains(needs.check.outputs.tag, '-') }}
body_path: ${{ runner.temp }}/notes.md
fail_on_unmatched_files: true
@@ -178,8 +210,9 @@ jobs:
needs: [check, publish]
if: ${{ !contains(needs.check.outputs.tag, '-') && vars.MSSTORE_PRODUCT_ID != '' }}
runs-on: windows-2022
+ timeout-minutes: 30
steps:
- - uses: actions/download-artifact@v8
+ - uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
path: store
pattern: Store-windows-*
@@ -199,7 +232,7 @@ jobs:
& $makeappx.FullName bundle /o /bv "$env:VERSION.0" /d store /p "Concat-$env:VERSION-windows.msixbundle"
if ($LASTEXITCODE -ne 0) { throw "makeappx failed with $LASTEXITCODE" }
- - uses: microsoft/setup-msstore-cli@v1
+ - uses: microsoft/setup-msstore-cli@3b2ec9136230357ce9abc02b9d056626fb248d3f # v1
- name: Submit
shell: pwsh
diff --git a/scripts/ci/deps.lock b/scripts/ci/deps.lock
new file mode 100644
index 00000000..6ebd1950
--- /dev/null
+++ b/scripts/ci/deps.lock
@@ -0,0 +1,59 @@
+# The prebuilt dependencies CI downloads, each pinned to one file and the
+# sha256 of its bytes. scripts/ci/fetch.sh is the only thing that reads
+# this, and it refuses a file whose digest is not the one written here.
+#
+# name url sha256
+#
+# To bump one: change the URL, run `scripts/ci/fetch.sh NAME /tmp/x`, and
+# put the digest it reports in place of the old one. Where a version is
+# also named elsewhere, change it there too; the comment above each group
+# says where.
+
+# FFmpeg 8.1, BtbN's GPL shared builds. A month's last autobuild, not
+# `latest`: BtbN keeps those, where the daily ones are deleted after a
+# fortnight, and `latest` changes under a build every day.
+ffmpeg-linux64 https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-09-30-13-08/ffmpeg-n8.1.3-9-g29e619e767-linux64-gpl-shared-8.1.tar.xz 76649c19a7a4615b367251ba4a8adc643f65831278a00e7a0c38398f7f06484b
+ffmpeg-linuxarm64 https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-09-30-13-08/ffmpeg-n8.1.3-9-g29e619e767-linuxarm64-gpl-shared-8.1.tar.xz d5b52fcb4f8d86d997bb75f6c4b383ca0be3f4cd9efd94ab3b4f27b22984f91c
+ffmpeg-win64 https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-09-30-13-08/ffmpeg-n8.1.3-9-g29e619e767-win64-gpl-shared-8.1.zip dfe81c3aa0a546ee81980b1b824486dd0fe0a1d452f00cb55135379a74be7bcf
+ffmpeg-winarm64 https://github.com/BtbN/FFmpeg-Builds/releases/download/autobuild-2026-09-30-13-08/ffmpeg-n8.1.3-9-g29e619e767-winarm64-gpl-shared-8.1.zip 31f35037625a8084085248fb47de2958678601ee9ab54e087234693616359c4b
+
+# ONNX Runtime, Microsoft's own builds. 1.28.0 is the version ort
+# 2.0.0-rc.13 is built against (ORT_VERSION in ci.yml and build-app.yml);
+# 1.23.2 is the last Microsoft made for Intel Macs (ORT_INTEL_MAC_VERSION).
+# The Windows ones are the CPU builds; build-app.yml says why.
+onnxruntime-linux-x64 https://github.com/microsoft/onnxruntime/releases/download/v1.28.0/onnxruntime-linux-x64-1.28.0.tgz a3e1b79d7bb1bf09696ce675f49e4064e6c81f6202b8225624fff0e93f8d6407
+onnxruntime-linux-aarch64 https://github.com/microsoft/onnxruntime/releases/download/v1.28.0/onnxruntime-linux-aarch64-1.28.0.tgz e15ff8b5d85afe6c144d97c6fd432254bf76a219daaf17658087d6ecb3e8f0bb
+onnxruntime-win-x64 https://github.com/microsoft/onnxruntime/releases/download/v1.28.0/onnxruntime-win-x64-1.28.0.zip abef733dacbe2f571547a7150b479b5cb9cc0df22f96c24983a42cadb1b4f8bc
+onnxruntime-win-arm64 https://github.com/microsoft/onnxruntime/releases/download/v1.28.0/onnxruntime-win-arm64-1.28.0.zip cbe4547463ece092b505c3581376ed5896d22b5429f39d5e645e425ecdd369ad
+onnxruntime-osx-x86_64 https://github.com/microsoft/onnxruntime/releases/download/v1.23.2/onnxruntime-osx-x86_64-1.23.2.tgz d10359e16347b57d9959f7e80a225a5b4a66ed7d7e007274a15cae86836485a6
+
+# sherpa-onnx's prebuilt libraries. The version is the one src/Cargo.lock
+# resolves for sherpa-onnx-sys; the workflows check the two agree.
+sherpa-onnx-linux-x64-static-lib https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.7/sherpa-onnx-v1.13.7-linux-x64-static-lib.tar.bz2 d1be7a69ac2b30120058d8302e624239a3064085383cfa47994a14fdc44c32d6
+sherpa-onnx-linux-aarch64-static-lib https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.7/sherpa-onnx-v1.13.7-linux-aarch64-static-lib.tar.bz2 743b4debabab2e9f654e393da7263bd9795aabed60ae80c83023054083415d14
+sherpa-onnx-osx-arm64-static-lib https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.7/sherpa-onnx-v1.13.7-osx-arm64-static-lib.tar.bz2 126daa2e8c09a4c5d54dc985722c43bd22f598adc56445905b377454b1b27e38
+sherpa-onnx-osx-x64-static-lib https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.7/sherpa-onnx-v1.13.7-osx-x64-static-lib.tar.bz2 8d8db6199af0119b16f6e2b01c5548b90c4392a462388dd59491e8c471283cca
+sherpa-onnx-win-x64-shared-MT-Release-lib https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.7/sherpa-onnx-v1.13.7-win-x64-shared-MT-Release-lib.tar.bz2 7a362a9f339a2937e7f8806e904f3e8a031a7774982eea08bca8336b2c257529
+sherpa-onnx-win-arm64-shared-MT-Release-lib https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.7/sherpa-onnx-v1.13.7-win-arm64-shared-MT-Release-lib.tar.bz2 2afa23a479fe15729e1ea9a9ed242454b1d9fccf749103742071684baa91a232
+
+# The Linux packagers: nfpm (NFPM_VERSION in build-app.yml) for the .deb
+# and .rpm, and appimagetool with the AppImage runtime it embeds, both a
+# named release rather than `continuous`. The runtime is handed to
+# appimagetool with --runtime-file, so it fetches nothing of its own.
+nfpm-x86_64 https://github.com/goreleaser/nfpm/releases/download/v2.47.0/nfpm_2.47.0_Linux_x86_64.tar.gz 0660ca602b2d2d2ae4781a06c692b3eeb9d437ffea05b831d76e41f4a3188783
+nfpm-arm64 https://github.com/goreleaser/nfpm/releases/download/v2.47.0/nfpm_2.47.0_Linux_arm64.tar.gz 1c0f5f2999b9a974bfb04fdb0cc3306096de530ac5dbb25d739cc5f5219c919c
+appimagetool-x86_64 https://github.com/AppImage/appimagetool/releases/download/1.9.1/appimagetool-x86_64.AppImage ed4ce84f0d9caff66f50bcca6ff6f35aae54ce8135408b3fa33abfc3cb384eb0
+appimagetool-aarch64 https://github.com/AppImage/appimagetool/releases/download/1.9.1/appimagetool-aarch64.AppImage f0837e7448a0c1e4e650a93bb3e85802546e60654ef287576f46c71c126a9158
+appimage-runtime-x86_64 https://github.com/AppImage/type2-runtime/releases/download/20251108/runtime-x86_64 2fca8b443c92510f1483a883f60061ad09b46b978b2631c807cd873a47ec260d
+appimage-runtime-aarch64 https://github.com/AppImage/type2-runtime/releases/download/20251108/runtime-aarch64 00cbdfcf917cc6c0ff6d3347d59e0ca1f7f45a6df1a428a0d6d8a78664d87444
+
+# LLVM for Windows: libclang for bindgen, clang-cl on arm64, and
+# llvm-objdump for scripts/ci/isa-scan.sh.
+llvm-win64 https://github.com/llvm/llvm-project/releases/download/llvmorg-21.1.0/LLVM-21.1.0-win64.exe e751be8019b7500b100456c915b4b3c339d8d88b81b3aad49796d39f700e7ef3
+llvm-woa64 https://github.com/llvm/llvm-project/releases/download/llvmorg-21.1.0/LLVM-21.1.0-woa64.exe 3f181aecc83d0eb40fb966ce18b97a3602871dbff83fc91745e7e0fd6661a7f7
+
+# Intel's Software Development Emulator, which the smoke tests run the
+# x86_64 bundles under posing as an Ivy Bridge (AVX, no AVX2): the CPU the
+# bundles promise to start on. See scripts/ci/smoke-*.
+sde-win https://downloadmirror.intel.com/924984/sde-external-10.13.1-2026-07-28-win.tar.xz 74e626ede09b0baa5011fc9e51b58627ea92c3fc0bae5fd7db34b490f335f651
+sde-lin https://downloadmirror.intel.com/924984/sde-external-10.13.1-2026-07-28-lin.tar.xz 94e97d623fec54385686e1e7ba65ebc9941748c05ee451423948334892bf2b50
diff --git a/scripts/ci/fetch.sh b/scripts/ci/fetch.sh
new file mode 100755
index 00000000..b067cdd1
--- /dev/null
+++ b/scripts/ci/fetch.sh
@@ -0,0 +1,79 @@
+#!/usr/bin/env bash
+# Downloads one of the prebuilt dependencies scripts/ci/deps.lock names and
+# checks it against the digest written there, so a build only ever links
+# or ships bytes somebody has looked at once. A file whose digest does not
+# match is deleted and the script fails, naming both digests: a release
+# that was replaced upstream, or a download that went wrong, stops the
+# build rather than ending up in a bundle.
+#
+# scripts/ci/fetch.sh NAME OUT download NAME to the file OUT
+# scripts/ci/fetch.sh --url NAME print the URL deps.lock has for NAME
+#
+# OUT already there with the right digest is kept, so a rerun on a warm
+# machine downloads nothing. Runs on Linux, macOS and Git's bash on
+# Windows: sha256sum where there is one, shasum where there is not.
+#
+# To bump a dependency, change its row and put the new file's digest in
+# it; scripts/ci/fetch.sh prints the digest it got when they differ.
+set -euo pipefail
+
+here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+lock="$here/deps.lock"
+
+usage() {
+ echo "usage: $0 NAME OUT | $0 --url NAME" >&2
+ exit 2
+}
+
+# The row for a name: its URL and digest, space-separated.
+row() {
+ awk -v name="$1" '
+ /^[[:space:]]*(#|$)/ { next }
+ $1 == name { print $2, $3; found = 1; exit }
+ END { exit !found }
+ ' "$lock"
+}
+
+digest() {
+ if command -v sha256sum >/dev/null 2>&1; then
+ sha256sum "$1" | awk '{ print $1 }'
+ else
+ shasum -a 256 "$1" | awk '{ print $1 }'
+ fi
+}
+
+[ $# -eq 2 ] || usage
+
+if [ "$1" = "--url" ]; then
+ found=$(row "$2") || { echo "fetch.sh: no row named $2 in $lock" >&2; exit 1; }
+ echo "${found%% *}"
+ exit 0
+fi
+
+name=$1
+out=$2
+found=$(row "$name") || { echo "fetch.sh: no row named $name in $lock" >&2; exit 1; }
+url=${found%% *}
+want=${found##* }
+
+if [ -f "$out" ] && [ "$(digest "$out")" = "$want" ]; then
+ echo "fetch.sh: $name is already at $out"
+ exit 0
+fi
+
+echo "fetch.sh: $name from $url"
+mkdir -p "$(dirname "$out")"
+curl -fsSL --retry 5 --retry-all-errors --connect-timeout 30 -o "$out" "$url"
+
+got=$(digest "$out")
+if [ "$got" != "$want" ]; then
+ rm -f "$out"
+ {
+ echo "fetch.sh: $name does not match deps.lock"
+ echo " url: $url"
+ echo " expected: $want"
+ echo " got: $got"
+ } >&2
+ exit 1
+fi
+echo "fetch.sh: $name verified (sha256 $got)"
diff --git a/scripts/ci/isa-allow.txt b/scripts/ci/isa-allow.txt
new file mode 100644
index 00000000..d0ff2fc3
--- /dev/null
+++ b/scripts/ci/isa-allow.txt
@@ -0,0 +1,63 @@
+# What scripts/ci/isa-scan.sh lets each shipped x86_64 file hold beyond
+# the AVX baseline the bundles promise. One row per file name pattern (a
+# shell glob, first match wins); then either `any`, for a library that
+# picks its kernels by asking the CPU at run time and so carries code an
+# older CPU never reaches, or five limits - avx2, fma, bmi, mulx/rorx,
+# avx512; see scripts/ci/isa-scan.sh for what each counts - each a count
+# (fail past it), `~` and a count (warn past it), or `-` (no limit). A file
+# with no row may hold none at all.
+#
+# pattern avx2 fma bmi mulx/rorx avx512
+
+# FFmpeg: every codec, filter and scaler picks its SIMD by av_get_cpu_flags.
+avcodec-*.dll any
+avdevice-*.dll any
+avfilter-*.dll any
+avformat-*.dll any
+avutil-*.dll any
+postproc-*.dll any
+swresample-*.dll any
+swscale-*.dll any
+libavcodec.so* any
+libavdevice.so* any
+libavfilter.so* any
+libavformat.so* any
+libavutil.so* any
+libpostproc.so* any
+libswresample.so* any
+libswscale.so* any
+
+# Microsoft's ONNX Runtime: MLAS chooses its kernels by CPUID. Its general
+# code is built for the baseline (1.28.0 holds no BMI instructions at all).
+onnxruntime.dll any
+onnxruntime_providers_shared.dll any
+libonnxruntime.so* any
+
+# sherpa-onnx's Windows DLLs: built with the static MSVC runtime, whose
+# maths functions carry FMA3 twins chosen at run time.
+sherpa-onnx-c-api.dll any
+sherpa-onnx-cxx-api.dll any
+
+# The MSVC runtime, shipped beside the binary: memcpy and friends pick
+# their AVX2 paths at run time.
+vcruntime140*.dll any
+msvcp140*.dll any
+concrt140.dll any
+
+# Concat itself. It holds dispatched code of its own - memchr, aho-corasick
+# and ring choose AVX2 or BMI2 paths at run time (ring's hand-written
+# cryptography is where mulx and rorx come from), and Skia carries its
+# Haswell and Skylake-X stages beside the baseline ones - so the vector
+# classes are not limited. The compiler-made BMI column is the one that
+# tells: pyke's static ONNX Runtime, linked into concat.exe in 0.2.5 and
+# 0.2.6 (#313), brought some 2,700 of them. On Linux the binary also links
+# sherpa-onnx statically, with an ONNX Runtime of its own inside.
+#
+# PROVISIONAL: these BMI limits are estimates, not measurements, so for
+# now they warn rather than fail; the emulated Ivy Bridge in the smoke
+# tests is what fails a build that would crash. The first run on CI
+# prints each binary's counts. Once it has, set the limit a margin above
+# the count shown, without the `~`, so the next library compiled for
+# Haswell fails the build.
+concat.exe - - ~1000 - -
+concat - - ~1000 - -
diff --git a/scripts/ci/isa-scan.sh b/scripts/ci/isa-scan.sh
new file mode 100755
index 00000000..b74d2768
--- /dev/null
+++ b/scripts/ci/isa-scan.sh
@@ -0,0 +1,183 @@
+#!/usr/bin/env bash
+# Scans x86_64 binaries for instructions an Ivy Bridge does not have, so a
+# bundle that would die at launch with an illegal instruction on a 2012
+# machine (#193, #313) fails its build instead.
+#
+# scripts/ci/isa-scan.sh FILE_OR_FOLDER...
+#
+# Every PE (.exe, .dll) and ELF file found is disassembled with
+# llvm-objdump, and each instruction from a set the x86_64 baseline the
+# bundles promise (AVX, nothing newer; see GGML_* in build-app.yml) does not
+# include is counted, by class:
+#
+# avx2 256-bit integer vector work, permutes, broadcasts, gathers
+# fma fused multiply-add (FMA3)
+# bmi the BMI1 and BMI2 bit operations a compiler emits for ordinary
+# shifts and masks when told the CPU has them (shlx, shrx,
+# sarx, bzhi, andn, bextr, blsi, blsr, blsmsk, pdep, pext):
+# the sign of code compiled for Haswell as a whole rather
+# than one dispatched function. #313's crash was a shlx.
+# mulx/rorx the two BMI2 instructions hand-written cryptography uses on
+# its fast paths (ring's, Go's), behind a CPU check; counted
+# apart so they do not drown the column above
+# avx512 anything touching a zmm or mask register
+#
+# Not every one of those is a fault waiting to happen. A library that asks
+# the CPU what it has at run time and picks its kernels by the answer
+# carries AVX2 code it never runs on an older machine: FFmpeg's codecs and
+# scalers, ONNX Runtime's MLAS, sherpa-onnx. And a statically linked binary
+# carries such code too - memchr, aho-corasick, ring and Skia all dispatch
+# this way - so a count of zero is not what to expect from concat itself.
+# scripts/ci/isa-allow.txt says, per file, how much of each class is
+# expected: `any` for a library that dispatches (not even disassembled),
+# or a limit per class for a binary that holds some dispatched code among
+# its own - a number, past which the scan fails; `~` and a number, past
+# which it warns; or `-`, no limit. A file with no row may hold none.
+#
+# What the count catches is the size of the problem #313 was: pyke's
+# static ONNX Runtime 1.28.0 for Windows, compiled for Haswell, holds some
+# 2,700 compiler-made BMI instructions (1,559 shlx, 783 shrx, 273 sarx);
+# Microsoft's onnxruntime.dll of the same version holds none. What the
+# count cannot do is say whether an instruction runs, which is why the
+# build also launches the bundle under Intel's emulator posing as an Ivy
+# Bridge (scripts/ci/smoke-*). The two together are the check.
+#
+# llvm-objdump is found as $LLVM_OBJDUMP, as llvm-objdump on the path, or as
+# the newest llvm-objdump-<version> on the path, which is how Ubuntu installs
+# it. Exits 1 when a file is over a hard limit, 2 on a usage error.
+set -euo pipefail
+
+here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+allow="$here/isa-allow.txt"
+
+if [ $# -eq 0 ]; then
+ echo "usage: $0 FILE_OR_FOLDER..." >&2
+ exit 2
+fi
+
+objdump=${LLVM_OBJDUMP:-}
+if [ -z "$objdump" ]; then
+ if command -v llvm-objdump >/dev/null 2>&1; then
+ objdump=llvm-objdump
+ else
+ objdump=$(compgen -c llvm-objdump- | sort -t- -k3 -n | tail -1 || true)
+ fi
+fi
+if [ -z "$objdump" ] || ! command -v "$objdump" >/dev/null 2>&1; then
+ echo "isa-scan: no llvm-objdump; set LLVM_OBJDUMP or install LLVM" >&2
+ exit 2
+fi
+
+# The allowance for a file, as "avx2 fma bmi mulx/rorx avx512" limits, or
+# `any`: the first row whose pattern matches its name, or all zero.
+allowance() {
+ local base=$1 pattern rest
+ while read -r pattern rest; do
+ case "$pattern" in '' | '#'*) continue ;; esac
+ # shellcheck disable=SC2254 # the pattern is a glob on purpose
+ case "$base" in $pattern) echo "${rest%%#*}"; return ;; esac
+ done < "$allow"
+ echo 0 0 0 0 0
+}
+
+# Whether a file is an x86_64 PE or ELF, by its magic bytes and machine
+# field, without needing `file` (Git's bash on Windows has none).
+is_x86_64() {
+ local f=$1 magic
+ magic=$(od -An -tx1 -N4 "$f" 2>/dev/null | tr -d ' \n')
+ case "$magic" in
+ 7f454c46)
+ # ELF: e_machine at 18, 0x3e is x86-64.
+ [ "$(od -An -tx1 -j18 -N1 "$f" | tr -d ' \n')" = 3e ]
+ ;;
+ 4d5a*)
+ # PE: the header's offset at 0x3c, then "PE\0\0" and the machine,
+ # 0x8664 for x86-64.
+ local off
+ off=$(od -An -tu4 -j60 -N4 "$f" | tr -d ' \n')
+ [ "$(od -An -tx1 -j$((off + 4)) -N2 "$f" | tr -d ' \n')" = 6486 ]
+ ;;
+ *) return 1 ;;
+ esac
+}
+
+# Counts per class for one file, as "avx2 fma bmi mulx/rorx avx512", and
+# the ten most frequent offending mnemonics after them on separate lines.
+count() {
+ "$objdump" -d --no-show-raw-insn "$1" 2>/dev/null | awk '
+ # The mnemonic is the first word after the address; AT&T syntax,
+ # which is llvm-objdump'"'"'s default for x86.
+ /^ *[0-9a-f]+:/ {
+ line = $0
+ sub(/^ *[0-9a-f]+:[ \t]*/, "", line)
+ split(line, w, /[ \t]+/)
+ m = w[1]
+ if (m == "" || m ~ /^(rep|lock|notrack|data16|cs|ds)$/) m = w[2]
+ cls = ""
+ if (line ~ /%zmm|%k[0-7]/) cls = "avx512"
+ else if (m ~ /^vf(n)?m(add|sub|addsub|subadd)(132|213|231)/) cls = "fma"
+ else if (m ~ /^(andn|bextr|blsi|blsmsk|blsr|bzhi|pdep|pext|sarx|shlx|shrx)[lq]?$/) cls = "bmi"
+ else if (m ~ /^(mulx|rorx)[lq]?$/) cls = "asm"
+ else if (m ~ /^(vpbroadcast[bwdq]|vbroadcasti128|vperm2i128|vpermd|vpermq|vpermps|vpermpd|vinserti128|vextracti128|vpmaskmov[dq]|vpsllv[dq]|vpsrlv[dq]|vpsravd|vpgather|vgather|vpblendd)/) cls = "avx2"
+ else if (m ~ /^vp/ && line ~ /%ymm/ && m !~ /^(vpermil|vptest|vperm2f128)/) cls = "avx2"
+ if (cls != "") { n[cls]++; seen[m]++ }
+ }
+ END {
+ printf "%d %d %d %d %d\n", n["avx2"], n["fma"], n["bmi"], n["asm"], n["avx512"]
+ for (k in seen) printf "%d %s\n", seen[k], k
+ }' | { read -r head; echo "$head"; sort -rn | head -10; }
+}
+
+files=()
+for arg in "$@"; do
+ if [ -d "$arg" ]; then
+ while IFS= read -r -d '' f; do files+=("$f"); done \
+ < <(find "$arg" -type f \( -name '*.exe' -o -name '*.dll' -o -name '*.so' -o -name '*.so.*' -o -perm -u+x \) -print0)
+ else
+ files+=("$arg")
+ fi
+done
+
+status=0
+row() { printf '%-36s %10s %10s %10s %10s %10s %s\n' "$@"; }
+row file avx2 fma bmi mulx/rorx avx512 verdict
+for f in "${files[@]}"; do
+ is_x86_64 "$f" || continue
+ base=$(basename "$f")
+ read -r -a limits <<< "$(allowance "$base")"
+ if [ "${limits[0]}" = any ]; then
+ row "$base" - - - - - "dispatches at run time; not scanned"
+ continue
+ fi
+ result=$(count "$f")
+ read -r -a counts <<< "$(head -1 <<< "$result")"
+ verdict=ok
+ shown=()
+ for i in 0 1 2 3 4; do
+ limit=${limits[$i]:-0}
+ shown+=("${counts[$i]}/$limit")
+ case "$limit" in
+ -) ;;
+ "~"*)
+ if [ "${counts[$i]}" -gt "${limit#"~"}" ] && [ "$verdict" = ok ]; then verdict=WARN; fi
+ ;;
+ *)
+ if [ "${counts[$i]}" -gt "$limit" ]; then verdict=OVER; status=1; fi
+ ;;
+ esac
+ done
+ row "$base" "${shown[@]}" "$verdict"
+ if [ "$verdict" != ok ]; then
+ tail -n +2 <<< "$result" | sed 's/^/ /'
+ fi
+ if [ "$verdict" = WARN ] && [ -n "${GITHUB_ACTIONS:-}" ]; then
+ echo "::warning::isa-scan: $base holds more post-Ivy Bridge instructions than scripts/ci/isa-allow.txt expects (${shown[*]}); see that file"
+ fi
+done
+
+if [ "$status" -ne 0 ]; then
+ echo
+ echo "isa-scan: a file holds more post-Ivy Bridge instructions than scripts/ci/isa-allow.txt allows it." >&2
+ echo "A library compiled for Haswell (AVX2, FMA, BMI2) as a whole crashes the app on older CPUs; see #313." >&2
+fi
+exit "$status"
diff --git a/scripts/ci/smoke-linux.sh b/scripts/ci/smoke-linux.sh
new file mode 100644
index 00000000..5e2c32ba
--- /dev/null
+++ b/scripts/ci/smoke-linux.sh
@@ -0,0 +1,157 @@
+#!/usr/bin/env bash
+# Launches a built Linux bundle the way a person would, and fails if it
+# does not start. Nothing else in the build ever runs what it packages:
+# a library left out of lib/, a dependency the .deb forgot to name, or an
+# instruction an older CPU lacks all used to reach a release unnoticed.
+#
+# smoke-linux.sh [--sde SDE64] DIR
+# The staged folder (DIR/concat beside DIR/lib), on this machine:
+# every library resolves from the folder alone; `concat --version`
+# exits 0; `concat --smoke` (start-up through the first frame of the
+# window, then out) exits 0 under Xvfb, and does not so much as try
+# to open the NVIDIA driver's CUDA or NVENC libraries, which nothing
+# should touch before a person asks for a hardware export. With
+# --sde, `--version` again under Intel's emulator posing as an Ivy
+# Bridge (AVX, no AVX2), the oldest CPU the x86_64 bundle promises.
+#
+# smoke-linux.sh --container IMAGE DEB APPIMAGE
+# The packages, in a clean IMAGE (ubuntu:22.04, debian:12) through
+# docker: the .deb installed by apt, which pulls in exactly the
+# dependencies it declares and nothing else, then `--version` and
+# `--smoke` from the installed package and from the AppImage.
+#
+# Runs in CI after staging (build-app.yml), and locally against a folder
+# or packages built the same way. Needs xvfb-run; strace if it is there.
+set -euo pipefail
+
+here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+
+say() { printf '\n== %s\n' "$*"; }
+fail() {
+ if [ -n "${GITHUB_ACTIONS:-}" ]; then echo "::error::$*"; else echo "smoke: $*" >&2; fi
+ exit 1
+}
+
+# A fresh home for each run, so the start-up is a first start-up: no
+# settings, no recent projects, nothing cached from a run before.
+fresh_home() {
+ local home
+ home=$(mktemp -d)
+ mkdir -p "$home/.config" "$home/.cache" "$home/.local/share"
+ echo "$home"
+}
+
+# Runs a command with a time limit, prints what it wrote, and fails with
+# what it is told when the command does not exit 0.
+expect_ok() {
+ local what=$1 secs=$2 out status=0
+ shift 2
+ out=$(timeout --kill-after=10 "$secs" "$@" 2>&1) || status=$?
+ printf '%s\n' "$out" | tail -n 60
+ if [ "$status" -eq 124 ] || [ "$status" -eq 137 ]; then
+ fail "$what: still running after ${secs}s"
+ elif [ "$status" -ne 0 ]; then
+ fail "$what: exited with $status"
+ fi
+}
+
+host() {
+ local sde="" dir
+ if [ "${1:-}" = "--sde" ]; then sde=$2; shift 2; fi
+ [ $# -eq 1 ] || { echo "usage: $0 [--sde SDE64] DIR" >&2; exit 2; }
+ dir=$(cd "$1" && pwd)
+ local bin="$dir/concat"
+ [ -x "$bin" ] || fail "$bin is not there"
+
+ # The build's own environment points at the FFmpeg it linked; the
+ # folder has to stand without it.
+ unset LD_LIBRARY_PATH
+
+ say "every library resolves from the folder"
+ local report
+ report=$(ldd "$bin" || true)
+ if grep -i 'not found' <<< "$report"; then
+ echo "$report"
+ fail "the binary does not resolve every library from its own folder"
+ fi
+
+ say "concat --version"
+ HOME=$(fresh_home) expect_ok "concat --version" 60 "$bin" --version
+
+ say "concat --smoke"
+ local trace
+ trace=$(mktemp)
+ local tracer=()
+ if command -v strace >/dev/null 2>&1; then
+ tracer=(strace -f -qq -e "trace=open,openat" -o "$trace")
+ else
+ echo "no strace: the start-up is run but its file opens are not checked"
+ fi
+ HOME=$(fresh_home) expect_ok "concat --smoke" 180 \
+ xvfb-run -a -s "-screen 0 1920x1080x24" "${tracer[@]}" "$bin" --smoke
+ if [ ${#tracer[@]} -gt 0 ]; then
+ local cuda
+ cuda=$(grep -E 'lib(cuda|nvcuvid|nvidia-encode)\.so' "$trace" || true)
+ if [ -n "$cuda" ]; then
+ echo "$cuda"
+ fail "concat --smoke tried to load the NVIDIA driver's CUDA or NVENC libraries at start-up"
+ fi
+ echo "no CUDA or NVENC library opened at start-up"
+ fi
+
+ if [ -n "$sde" ]; then
+ say "concat --version on an emulated Ivy Bridge"
+ # The emulator stops the program at the first instruction the chip
+ # it poses as does not have, and says which.
+ HOME=$(fresh_home) expect_ok "concat --version under SDE -ivb" 600 "$sde" -ivb -- "$bin" --version
+ fi
+}
+
+container() {
+ [ $# -eq 3 ] || { echo "usage: $0 --container IMAGE DEB APPIMAGE" >&2; exit 2; }
+ local image=$1 deb appimage
+ deb=$(cd "$(dirname "$2")" && pwd)/$(basename "$2")
+ appimage=$(cd "$(dirname "$3")" && pwd)/$(basename "$3")
+ [ -f "$deb" ] || fail "$deb is not there"
+ [ -f "$appimage" ] || fail "$appimage is not there"
+ say "in a clean $image"
+ docker run --rm \
+ -v "$here/smoke-linux.sh:/ci/smoke-linux.sh:ro" \
+ -v "$deb:/pkg/concat.deb:ro" \
+ -v "$appimage:/pkg/concat.AppImage:ro" \
+ "$image" bash /ci/smoke-linux.sh --inside /pkg/concat.deb /pkg/concat.AppImage
+}
+
+# Inside the container. Xvfb and Mesa's software drivers stand in for
+# the desktop's display and GPU, and one font for the desktop's fonts;
+# everything else the program needs comes from the package's own
+# dependencies, so a library it forgot to name is missing here too.
+inside() {
+ local deb=$1 appimage=$2
+ export DEBIAN_FRONTEND=noninteractive
+ say "installing the .deb and what it depends on"
+ apt-get update -qq
+ apt-get install -y -qq --no-install-recommends \
+ "$deb" xvfb xauth libgl1-mesa-dri mesa-vulkan-drivers fonts-dejavu-core > /dev/null
+
+ say "concat --version (the package)"
+ HOME=$(fresh_home) expect_ok "concat --version from the .deb" 60 concat --version
+ say "concat --smoke (the package)"
+ HOME=$(fresh_home) expect_ok "concat --smoke from the .deb" 180 \
+ xvfb-run -a -s "-screen 0 1920x1080x24" concat --smoke
+
+ # No FUSE in a container: the AppImage unpacks itself and runs.
+ export APPIMAGE_EXTRACT_AND_RUN=1
+ say "concat --version (the AppImage)"
+ HOME=$(fresh_home) expect_ok "concat --version from the AppImage" 120 "$appimage" --version
+ say "concat --smoke (the AppImage)"
+ HOME=$(fresh_home) expect_ok "concat --smoke from the AppImage" 240 \
+ xvfb-run -a -s "-screen 0 1920x1080x24" "$appimage" --smoke
+}
+
+case "${1:-}" in
+ --container) shift; container "$@" ;;
+ --inside) shift; inside "$@" ;;
+ *) host "$@" ;;
+esac
+say "ok"
diff --git a/scripts/ci/smoke-macos.sh b/scripts/ci/smoke-macos.sh
new file mode 100644
index 00000000..bb616f4d
--- /dev/null
+++ b/scripts/ci/smoke-macos.sh
@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+# Launches the built macOS bundle the way a person would, and fails if it
+# does not start: nothing else in the build ever runs what it packages.
+#
+# smoke-macos.sh Concat.app
+#
+# The signature holds over the whole bundle (an ad-hoc one, which is what
+# a fork's build gets, passes and is reported); no binary in it loads a
+# library from Homebrew or anywhere else outside the bundle; `concat
+# --version` exits 0; and `concat --smoke` - start-up through the first
+# frame of the window, then out - exits 0. Runs in CI after signing
+# (build-app.yml), and locally against a bundle built the same way.
+set -euo pipefail
+
+say() { printf '\n== %s\n' "$*"; }
+note() {
+ if [ -n "${GITHUB_ACTIONS:-}" ]; then echo "::warning::$*"; else echo "smoke: $*"; fi
+}
+fail() {
+ if [ -n "${GITHUB_ACTIONS:-}" ]; then echo "::error::$*"; else echo "smoke: $*" >&2; fi
+ exit 1
+}
+
+# A time limit without coreutils: perl's alarm, which the exec keeps.
+# A run that overstays it dies of SIGALRM, status 142.
+expect_ok() {
+ local what=$1 secs=$2 out status=0
+ shift 2
+ out=$(perl -e 'alarm shift; exec @ARGV or die "exec: $!"' "$secs" "$@" 2>&1) || status=$?
+ printf '%s\n' "$out" | tail -n 60
+ if [ "$status" -eq 142 ]; then
+ fail "$what: still running after ${secs}s"
+ elif [ "$status" -ne 0 ]; then
+ fail "$what: exited with $status"
+ fi
+}
+
+[ $# -eq 1 ] || { echo "usage: $0 Concat.app" >&2; exit 2; }
+app=$1
+bin="$app/Contents/MacOS/concat"
+[ -x "$bin" ] || fail "$bin is not there"
+
+say "the signature"
+codesign --verify --deep --strict --verbose=2 "$app" || fail "the bundle's signature does not verify"
+details=$(codesign -dv "$app" 2>&1 || true)
+if grep -q 'Signature=adhoc' <<< "$details"; then
+ note "the bundle is ad-hoc signed: it opens, with Gatekeeper's warning, and is not notarised"
+else
+ grep -E '^(Authority|TeamIdentifier)=' <<< "$details" || true
+fi
+
+say "nothing loads from outside the bundle"
+leaks=""
+while IFS= read -r -d '' f; do
+ if [[ $(file -b "$f") == *Mach-O* ]]; then
+ found=$(otool -L "$f" | tail -n +2 | grep -E '/opt/homebrew|/usr/local|/opt/local' || true)
+ if [ -n "$found" ]; then leaks+="$f:"$'\n'"$found"$'\n'; fi
+ fi
+done < <(find "$app/Contents" -type f -print0)
+if [ -n "$leaks" ]; then
+ printf '%s' "$leaks"
+ fail "a binary in the bundle loads a library from outside it"
+fi
+echo "every load path is the system's or the bundle's"
+
+say "concat --version"
+HOME=$(mktemp -d) expect_ok "concat --version" 60 "$bin" --version
+
+say "concat --smoke"
+HOME=$(mktemp -d) expect_ok "concat --smoke" 180 "$bin" --smoke
+
+say "ok"
diff --git a/scripts/ci/smoke-windows.ps1 b/scripts/ci/smoke-windows.ps1
new file mode 100644
index 00000000..bf053236
--- /dev/null
+++ b/scripts/ci/smoke-windows.ps1
@@ -0,0 +1,161 @@
+# Launches the staged Windows bundle the way a person would, and fails if
+# it does not start. Nothing else in the build ever ran what it packaged,
+# and 0.2.6 shipped installers whose program would not open: a DLL that
+# was not in the folder, an old DirectML.dll found in System32 instead of
+# a new one (#303), an instruction an Ivy Bridge does not have (#313).
+#
+# scripts/ci/smoke-windows.ps1 -Bundle stage\Concat-<version>-windows-<arch> [-Sde path\to\sde.exe]
+#
+# 1. Every DLL any program or library in the folder imports is either in
+# the folder or one Windows itself provides. A DLL that only happens to
+# be on this machine - the Visual C++ runtime, DirectML, ONNX Runtime,
+# which Windows 11 carries a copy of its own - counts as missing.
+# 2. `concat.exe --version` exits 0 with PATH cut down to Windows' own
+# folders, so every DLL comes from the folder or from System32 - as on
+# a machine with nothing else installed. It touches no GPU: it proves
+# the program and every DLL it loads at start load and initialise.
+# 3. `concat.exe --smoke` exits 0 the same way: the start-up runs through
+# the first frame of the window, then the program leaves.
+# 4. With -Sde, `--version` again under Intel's emulator posing as an Ivy
+# Bridge (AVX, nothing newer), which stops the program at the first
+# instruction that chip does not have. x86_64 only.
+#
+# concat.exe is a GUI program: it has no console, so it is judged by its
+# exit code and not by what it prints. Windows' error dialogs are turned
+# off for the programs this starts, so a missing DLL is an exit code here
+# rather than a dialog waiting for a click on a runner nobody is watching.
+#Requires -Version 7
+param(
+ [Parameter(Mandatory = $true)][string]$Bundle,
+ [string]$Sde = ""
+)
+$ErrorActionPreference = "Stop"
+
+$bundle = (Resolve-Path $Bundle).Path
+$exe = Join-Path $bundle "concat.exe"
+if (-not (Test-Path $exe)) { throw "$exe is not there" }
+
+function Say($text) { Write-Host ""; Write-Host "== $text" }
+function Fail($text) {
+ if ($env:GITHUB_ACTIONS) { Write-Host "::error::$text" }
+ throw $text
+}
+
+# No error dialogs for this process or the ones it starts: a child takes
+# its error mode from its parent. SEM_FAILCRITICALERRORS (the loader's
+# "the code execution cannot proceed") and SEM_NOGPFAULTERRORBOX (the
+# crash dialog).
+Add-Type -Namespace Smoke -Name Native -MemberDefinition @"
+[DllImport("kernel32.dll")] public static extern uint SetErrorMode(uint mode);
+"@
+[void][Smoke.Native]::SetErrorMode(0x0001 -bor 0x0002)
+
+# What a status a program died with means, for the ones a broken bundle
+# dies with.
+$meanings = @{
+ [int]0xC0000135 = "STATUS_DLL_NOT_FOUND: a DLL it imports is nowhere to be found"
+ [int]0xC0000139 = "STATUS_ENTRYPOINT_NOT_FOUND: a DLL it found is not the one it was built against"
+ [int]0xC000007B = "STATUS_INVALID_IMAGE_FORMAT: a DLL for another architecture, or a damaged one"
+ [int]0xC000001D = "STATUS_ILLEGAL_INSTRUCTION: an instruction this CPU does not have"
+ [int]0xC0000005 = "STATUS_ACCESS_VIOLATION: a crash"
+ [int]0xC0000409 = "STATUS_STACK_BUFFER_OVERRUN: a fail-fast, which is how a Rust panic aborts"
+ [int]0xC0E90002 = "a code integrity block (Smart App Control) on an unsigned or untrusted file"
+}
+
+# Starts a program with the given PATH and waits for it, up to a limit;
+# fails unless it exits 0. Its output is shown when there is any.
+function Expect-Ok($what, $file, [string[]]$arguments, $path, $seconds) {
+ $info = New-Object System.Diagnostics.ProcessStartInfo
+ $info.FileName = $file
+ foreach ($a in $arguments) { [void]$info.ArgumentList.Add($a) }
+ $info.UseShellExecute = $false
+ $info.RedirectStandardOutput = $true
+ $info.RedirectStandardError = $true
+ # Not the bundle: the program has to find its DLLs beside itself, not
+ # in the folder it was started from.
+ $info.WorkingDirectory = [IO.Path]::GetTempPath()
+ $info.Environment["PATH"] = $path
+ $process = [System.Diagnostics.Process]::Start($info)
+ $stdout = $process.StandardOutput.ReadToEndAsync()
+ $stderr = $process.StandardError.ReadToEndAsync()
+ if (-not $process.WaitForExit($seconds * 1000)) {
+ $process.Kill($true)
+ Fail "${what}: still running after $seconds s"
+ }
+ $process.WaitForExit()
+ $out = ($stdout.Result + $stderr.Result).Trim()
+ if ($out) { $out -split "`n" | Select-Object -Last 60 | ForEach-Object { Write-Host " $_" } }
+ $code = $process.ExitCode
+ if ($code -ne 0) {
+ $meaning = $meanings[[int]$code]
+ Fail ("${what}: exited with 0x{0:X8}{1}" -f $code, $(if ($meaning) { " - $meaning" } else { "" }))
+ }
+ Write-Host " exited 0"
+}
+
+# ── 1. Every import is in the folder or Windows' own ───────────────────
+Say "every DLL the bundle imports is in it, or is Windows' own"
+$vswhere = "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe"
+$dumpbin = & $vswhere -latest -products * -find "VC\Tools\MSVC\**\bin\Host*\*\dumpbin.exe" | Select-Object -First 1
+if (-not $dumpbin) { throw "dumpbin.exe is not in the Visual Studio installation" }
+
+# DLLs a clean Windows does not have, or has in a version of its own that
+# is not the one the bundle was built against. One of these resolved from
+# System32 on the runner is a DLL missing on somebody's machine.
+$mustShip = @(
+ "vcruntime140.dll", "vcruntime140_1.dll", "msvcp140.dll", "msvcp140_1.dll",
+ "msvcp140_2.dll", "msvcp140_atomic_wait.dll", "msvcp140_codecvt_ids.dll",
+ "concrt140.dll", "vccorlib140.dll", "vcomp140.dll",
+ "directml.dll", "onnxruntime.dll", "onnxruntime_providers_shared.dll"
+)
+$system = Join-Path $env:SystemRoot "System32"
+$inBundle = @{}
+Get-ChildItem $bundle -Recurse -File | ForEach-Object { $inBundle[$_.Name.ToLowerInvariant()] = $true }
+$missing = @()
+foreach ($pe in Get-ChildItem $bundle -Recurse -File -Include *.exe, *.dll) {
+ $lines = & $dumpbin /nologo /dependents $pe.FullName
+ if ($LASTEXITCODE -ne 0) { throw "dumpbin failed on $($pe.Name)" }
+ # The names are the indented lines under "Image has the following
+ # (delay load) dependencies:", up to the summary.
+ $reading = $false
+ foreach ($line in $lines) {
+ if ($line -match "Image has the following .*dependencies") { $reading = $true; continue }
+ if ($line -match "^\s*Summary") { $reading = $false; continue }
+ if (-not $reading) { continue }
+ $dll = $line.Trim()
+ if (-not $dll -or $dll -notmatch "\.dll$") { continue }
+ $name = $dll.ToLowerInvariant()
+ if ($inBundle.ContainsKey($name)) { continue }
+ # API sets: names the loader resolves to whatever implements them.
+ if ($name -match "^(api|ext)-ms-") { continue }
+ if ($mustShip -contains $name) { $missing += "$($pe.Name) -> $dll (must ship with the program)"; continue }
+ if (-not (Test-Path (Join-Path $system $dll))) { $missing += "$($pe.Name) -> $dll (not in the folder or in System32)" }
+ }
+}
+if ($missing.Count -gt 0) {
+ $missing | ForEach-Object { Write-Host " $_" }
+ Fail "the bundle imports DLLs it does not carry"
+}
+Write-Host " every import resolves to the folder or to Windows"
+
+# Windows' own folders and nothing else.
+$bare = @($system, $env:SystemRoot, (Join-Path $system "Wbem"), (Join-Path $system "WindowsPowerShell\v1.0")) -join ";"
+
+# ── 2. --version ───────────────────────────────────────────────────────
+Say "concat.exe --version, with only Windows on the path"
+Expect-Ok "concat.exe --version" $exe @("--version") $bare 60
+
+# ── 3. --smoke ─────────────────────────────────────────────────────────
+Say "concat.exe --smoke, with only Windows on the path"
+Expect-Ok "concat.exe --smoke" $exe @("--smoke") $bare 180
+
+# ── 4. On an emulated Ivy Bridge ───────────────────────────────────────
+if ($Sde) {
+ Say "concat.exe --version on an emulated Ivy Bridge"
+ $sdeExe = (Resolve-Path $Sde).Path
+ # The emulator's own folder too: it loads its pieces from beside itself.
+ $path = (Split-Path $sdeExe) + ";" + $bare
+ Expect-Ok "concat.exe --version under SDE -ivb" $sdeExe @("-ivb", "--", $exe, "--version") $path 600
+}
+
+Say "ok"
diff --git a/src/.config/nextest.toml b/src/.config/nextest.toml
new file mode 100644
index 00000000..24181fc1
--- /dev/null
+++ b/src/.config/nextest.toml
@@ -0,0 +1,40 @@
+# cargo-nextest's settings for this workspace. CI runs the tests through
+# nextest rather than `cargo test` for three things `cargo test` does not
+# give it: every test in a process of its own, so a process-wide setting
+# one test changes (the hardware decode preference, say) never reaches
+# another; a timeout per test, so a test that hangs is killed and named
+# rather than holding the whole step until the runner gives up on it; and
+# a JUnit report the job keeps when it fails.
+#
+# Locally `cargo nextest run` uses the default profile, which is nextest's
+# own defaults; `--profile ci` is what CI runs.
+
+[profile.ci]
+# Every test runs, whatever fails first: a red run should list everything
+# that is red, not the first thing.
+fail-fast = false
+# A test is reported as slow after a minute and killed after five, which
+# is far past what any test takes on Linux (the longest, the export
+# scenarios, finish in under one). Killed, it is reported as a timeout
+# under its own name.
+slow-timeout = { period = "60s", terminate-after = 5 }
+# Failures again at the end, after the rest of the output has scrolled
+# past them.
+final-status-level = "fail"
+failure-output = "immediate-final"
+
+[profile.ci.junit]
+# Written to target/nextest/ci/junit.xml, which the CI job uploads when it
+# fails. The output of a failed test is kept in the report; a passing
+# one's is not.
+path = "junit.xml"
+store-success-output = false
+store-failure-output = true
+
+# The scripted editing session exports some thirty times, and on Windows'
+# software adapter each export is slower than on Linux's lavapipe: it is
+# given longer before it counts as stuck. CI also runs it on its own with
+# its output shown, so a hang names the edit it stopped at; see ci.yml.
+[[profile.ci.overrides]]
+filter = "test(=every_edit_still_exports)"
+slow-timeout = { period = "60s", terminate-after = 12 }
diff --git a/src/crates/concat-host/tests/export.rs b/src/crates/concat-host/tests/export.rs
index d08af09f..eaef5b7e 100644
--- a/src/crates/concat-host/tests/export.rs
+++ b/src/crates/concat-host/tests/export.rs
@@ -563,6 +563,12 @@ impl Studio {
fn export_at(&mut self, label: &str, rate: Option<(i64, i64)>) -> Exported {
self.count += 1;
let label = format!("{:02} {label}", self.count).replace('/', "-");
+ // Each export names itself before it starts and again once the
+ // file is written, so a run that hangs (every_edit_still_exports
+ // on Windows CI, at the time of writing) says which edit and which
+ // half: the render, or the reopen and read-back after it. Shown
+ // with --nocapture, or under a failure.
+ eprintln!("{label}: exporting");
let settings = self.session.settings();
let spec = ExportSpec {
output: self
@@ -593,6 +599,7 @@ impl Studio {
let cancel = AtomicBool::new(false);
let written = export::run(&request, &cancel, |_| {})
.unwrap_or_else(|error| panic!("{label}: the export failed: {error}"));
+ eprintln!("{label}: written, reading it back");
self.session.save(None).expect("saves the project");
let reopened = Session::open(self.session.path(), self.session.settings())
@@ -1781,18 +1788,56 @@ fn the_edges_export_too() {
/// Hardware decode preferred, the way the app always sets it: the
/// export decodes its sources on the platform's device where there is one
-/// and reads the same picture and sound back. The preference is the
-/// process's, so the other scenarios running alongside share it for the
-/// moment, and must not mind.
+/// and reads the same picture and sound back.
+///
+/// A platform can name a device the machine cannot open: Windows' CI
+/// runner has WARP, a software Direct3D adapter with no video decoder, so
+/// D3D11VA falls back to the CPU there, as it should, and the reader never
+/// reports the device. The device is tried first on its own, by name and
+/// without touching the preference; where it does not decode, the test
+/// skips and says so, unless the job says the GPU tests must run
+/// (`CONCAT_REQUIRE_GPU`, as cards.rs reads it), in which case it fails.
+///
+/// The preference is the process's. Under nextest every test is a process
+/// of its own and nothing else sees it; under `cargo test` the scenarios
+/// running alongside share it while this one runs, and the guard puts back
+/// whatever it was before, so none of them is left with it after.
#[test]
fn an_export_decodes_on_the_hardware_when_preferred() {
- struct Preferred;
- impl Drop for Preferred {
+ let scratch = Scratch::new("hardware");
+ // HEVC, not the H.264 the other scenarios use: 8-bit H.264 is the one
+ // stream the preference leaves on the CPU, which decodes it faster
+ // (concat_media::hardware::hardware_wins).
+ let sources = Sources::make_in(scratch.path(), VideoCodec::Hevc);
+
+ if let Some(device) = HwDevice::platform_default() {
+ let probe = DecodeOptions {
+ hardware: concat_media::HwPolicy::Device(device),
+ ..DecodeOptions::default()
+ };
+ let mut reader = Decoder::open(&sources.peek, &probe).expect("opens");
+ reader.next_frame().expect("decodes").expect("a frame");
+ if reader.hardware() != Some(device) {
+ let why = format!(
+ "{} is this platform's device, but it does not decode HEVC on this machine \
+ (a software adapter, or no video decoder)",
+ device.label()
+ );
+ if std::env::var_os("CONCAT_REQUIRE_GPU").is_some() {
+ panic!("{why}, and CONCAT_REQUIRE_GPU is set");
+ }
+ eprintln!("{why}, and CONCAT_REQUIRE_GPU is not set; skipping");
+ return;
+ }
+ }
+
+ struct Restore(Option<HwDevice>);
+ impl Drop for Restore {
fn drop(&mut self) {
- concat_media::set_hardware_decode(false);
+ concat_media::hardware::set_hardware_device(self.0);
}
}
- let _preferred = Preferred;
+ let _restore = Restore(concat_media::hardware::hardware_device());
concat_media::set_hardware_decode(true);
assert_eq!(
concat_media::hardware_decode(),
@@ -1800,11 +1845,6 @@ fn an_export_decodes_on_the_hardware_when_preferred() {
"on where the platform has a device, off where it has none"
);
- let scratch = Scratch::new("hardware");
- // HEVC, not the H.264 the other scenarios use: 8-bit H.264 is the one
- // stream the preference leaves on the CPU, which decodes it faster
- // (concat_media::hardware::hardware_wins).
- let sources = Sources::make_in(scratch.path(), VideoCodec::Hevc);
// A reader opened the way the engine opens them follows the preference
// onto the device, on a machine that has one.
let mut reader = Decoder::open(&sources.peek, &DecodeOptions::default()).expect("opens");
diff --git a/src/crates/concat-speech/Cargo.toml b/src/crates/concat-speech/Cargo.toml
index f04cc400..700db998 100644
--- a/src/crates/concat-speech/Cargo.toml
+++ b/src/crates/concat-speech/Cargo.toml
@@ -73,8 +73,9 @@ whisper-rs = { version = "0.16", features = ["metal"] }
# runtime in place of the plain one.
ort = { workspace = true, optional = true, features = ["coreml"] }
-# DirectML, so Chatterbox runs on any Direct3D 12 GPU; the same runtime
-# build concat-vision's cutouts use.
+# DirectML, so Chatterbox runs on any Direct3D 12 GPU where the runtime
+# has the provider. The shipped Windows bundle links Microsoft's CPU build
+# instead, so there it runs on the CPU; concat-vision's manifest says why.
[target.'cfg(target_os = "windows")'.dependencies]
ort = { workspace = true, optional = true, features = ["directml"] }
diff --git a/src/crates/concat-vision/Cargo.toml b/src/crates/concat-vision/Cargo.toml
index cdaae5d5..6b864989 100644
--- a/src/crates/concat-vision/Cargo.toml
+++ b/src/crates/concat-vision/Cargo.toml
@@ -26,6 +26,16 @@ ort = { workspace = true, optional = true }
[target.'cfg(target_vendor = "apple")'.dependencies]
ort = { workspace = true, optional = true, features = ["coreml"] }
+# DirectML on Windows asks for the provider; whether it is there is the
+# runtime's to say. The bundles and CI link Microsoft's CPU build of ONNX
+# Runtime dynamically instead of the static DirectML one ort-sys would
+# download - that one is compiled for Haswell and kept the app from
+# starting on older CPUs (#313), and it imports a DirectML.dll many
+# machines only have an old copy of (#303). See build-app.yml's ONNX
+# Runtime (Windows) step. With the CPU build the provider fails to
+# register and ort carries on with the CPU one. The feature stays because
+# a build with no ORT_LIB_LOCATION (a developer's) downloads by feature,
+# and pyke publishes no Windows build without one.
[target.'cfg(target_os = "windows")'.dependencies]
ort = { workspace = true, optional = true, features = ["directml"] }
diff --git a/src/rust-toolchain.toml b/src/rust-toolchain.toml
index 36c87b3f..1d511a0f 100644
--- a/src/rust-toolchain.toml
+++ b/src/rust-toolchain.toml
@@ -1,6 +1,9 @@
-# Pinned so a checkout in two years builds the same way it does today.
+# Pinned so a checkout in two years builds the same way it does today, to
+# the patch release: "1.93" alone is whichever 1.93.x is newest on the day,
+# so two runs a week apart could build with different compilers. CI reads
+# the channel from here (the workflows' "The pinned toolchain" steps).
[toolchain]
-channel = "1.93"
+channel = "1.93.1"
components = ["rustfmt", "clippy"]
# The engine crates that carry no native library build for the web too, and
# CI checks that they keep doing so; see README.md.