Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

Link FFmpeg and compile whisper in, and move the host's Rust into the engine

Jareerbc2673a96ff8
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 5b0dcfd4..4af3b6da 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -1,19 +1,20 @@
name: Build App
-# Compiles the Concat desktop app for macOS, Windows and Linux.
+# Tests the engine and builds Concat - the Slint window, `engine/crates/concat`
+# - for macOS, Windows and Linux.
#
-# Every build stages the static FFmpeg pair into desktop/src-tauri/ffmpeg/
-# first - the release guard in build.rs refuses to build without it, because a
-# packaged app without its FFmpeg ships broken (no PATH fallback from Finder).
+# The engine links FFmpeg and compiles whisper.cpp in, so a build needs the
+# FFmpeg 7+ development libraries (headers and import libraries), cmake and a
+# C++ toolchain - nothing is staged beside the binary any more. Homebrew's
+# FFmpeg serves on macOS; Windows and Linux use a BtbN `shared` build, which
+# ships headers, import libraries and the runtime libraries together.
#
-# - Pushes to main: build, and when every job is green, automatically publish
-# a pre-release tagged v<version>-alpha.<n> - the version comes from
-# tauri.conf.json, and <n> increments past the highest existing alpha tag
-# for that version. A red build publishes nothing.
-# - Tags starting with v (v0.2.0, ...): publish the installers under that tag
-# name. Tags containing a hyphen are marked as pre-releases. This is the
-# path for stable releases; alphas take care of themselves.
-# - Manual runs: build and upload the bundles as artifacts only.
+# - Pushes to main and manual runs: test, build, and upload the binaries as
+# artifacts.
+# - Releases: none yet. The Tauri app in desktop/ used to be packaged and
+# published from here; it is deprecated and no longer built. Automatic
+# alpha releases resume once the Slint window is packaged for each
+# platform.
on:
workflow_dispatch:
push:
@@ -24,6 +25,11 @@ concurrency:
group: build-${{ github.ref }}
cancel-in-progress: true
+env:
+ # BtbN's autobuilds: GPL (x264 in), shared, FFmpeg 8.1.
+ FFMPEG_LINUX: https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-linux64-gpl-shared-8.1.tar.xz
+ FFMPEG_WINDOWS: https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-win64-gpl-shared-8.1.zip
+
jobs:
engine-tests:
name: Engine tests (ubuntu)
@@ -34,28 +40,38 @@ jobs:
- uses: Swatinem/rust-cache@v2
with:
workspaces: engine
- # The concat crate builds Slint with its Skia renderer, which links
- # fontconfig and freetype on Linux; winit and glutin dlopen the rest at
- # run time, so only these headers are needed to compile the workspace.
- - name: Slint build dependencies
+
+ # Slint's Skia renderer links fontconfig and freetype; cpal links
+ # ALSA; bindgen (for the FFmpeg bindings) needs libclang; whisper.cpp
+ # is built by cmake.
+ - name: System libraries
run: |
sudo apt-get update
- sudo apt-get install -y libfontconfig1-dev libfreetype6-dev libxkbcommon-dev libgl1-mesa-dev
+ sudo apt-get install -y \
+ libfontconfig1-dev libfreetype6-dev libxkbcommon-dev libgl1-mesa-dev \
+ libasound2-dev libclang-dev cmake
+
+ # Ubuntu's packaged FFmpeg is older than the 7 the engine needs, so the
+ # libraries come from BtbN, and FFMPEG_DIR tells the bindings where.
+ - name: FFmpeg development libraries
+ run: |
+ set -euo pipefail
+ curl -fsSL --retry 5 --retry-all-errors -o ffmpeg.tar.xz "$FFMPEG_LINUX"
+ mkdir -p ffmpeg-dev
+ tar -xf ffmpeg.tar.xz -C ffmpeg-dev --strip-components=1
+ echo "FFMPEG_DIR=$PWD/ffmpeg-dev" >> "$GITHUB_ENV"
+ echo "LD_LIBRARY_PATH=$PWD/ffmpeg-dev/lib" >> "$GITHUB_ENV"
+
- run: cargo test --workspace
working-directory: engine
# -D warnings covers rustc lints too (missing_docs among them), so a
# public item without documentation fails here, not in review.
- run: cargo clippy --workspace --all-targets -- -D warnings
working-directory: engine
- # The committed TypeScript IPC types must match the Rust source of
- # truth: regenerate the engine-owned bindings and fail on any drift.
- - name: IPC type freshness (engine)
- run: |
- scripts/generate-ipc-types.sh engine
- git diff --exit-code desktop/src/lib/generated
app:
name: App (${{ matrix.platform }})
+ needs: [engine-tests]
strategy:
fail-fast: false
matrix:
@@ -64,271 +80,80 @@ jobs:
os: macos-latest
- platform: windows
os: windows-latest
- # 22.04, not latest: bundles link the build machine's glibc and
- # webkit, so the oldest supported runner is what keeps the deb and
- # AppImage runnable on older distros.
+ # 22.04, not latest: the binary links the build machine's glibc,
+ # so the oldest supported runner is what keeps it runnable on
+ # older distributions.
- platform: linux
os: ubuntu-22.04
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
-
- # Tauri's Linux prerequisites, plus libfuse2 for the AppImage tooling.
- - name: Linux system libraries
- if: matrix.platform == 'linux'
- run: |
- sudo apt-get update
- sudo apt-get install -y \
- libwebkit2gtk-4.1-dev \
- build-essential curl wget file \
- libxdo-dev libssl-dev \
- libayatana-appindicator3-dev librsvg2-dev \
- libgtk-3-dev libfuse2 \
- libasound2-dev
-
- - uses: actions/setup-node@v4
- with:
- node-version: 22
- cache: npm
- cache-dependency-path: desktop/package-lock.json
-
- uses: dtolnay/rust-toolchain@stable
-
- uses: Swatinem/rust-cache@v2
with:
- workspaces: desktop/src-tauri
-
- # Static builds, so the pair runs on machines with nothing installed.
- # macos-latest is Apple Silicon, hence the arm64 build.
- - name: Stage bundled FFmpeg (macOS)
- if: matrix.platform == 'macos'
- run: |
- set -euo pipefail
- mkdir -p desktop/src-tauri/ffmpeg ffdl
- curl -fsSL --retry 5 --retry-all-errors -o ffdl/ffmpeg.zip \
- "https://ffmpeg.martin-riedl.de/redirect/latest/macos/arm64/release/ffmpeg.zip"
- curl -fsSL --retry 5 --retry-all-errors -o ffdl/ffprobe.zip \
- "https://ffmpeg.martin-riedl.de/redirect/latest/macos/arm64/release/ffprobe.zip"
- unzip -o ffdl/ffmpeg.zip -d ffdl/ffmpeg-out
- unzip -o ffdl/ffprobe.zip -d ffdl/ffprobe-out
- mv "$(find ffdl/ffmpeg-out -type f -name ffmpeg | head -1)" desktop/src-tauri/ffmpeg/ffmpeg
- mv "$(find ffdl/ffprobe-out -type f -name ffprobe | head -1)" desktop/src-tauri/ffmpeg/ffprobe
- chmod +x desktop/src-tauri/ffmpeg/ffmpeg desktop/src-tauri/ffmpeg/ffprobe
- desktop/src-tauri/ffmpeg/ffmpeg -version | head -1
- desktop/src-tauri/ffmpeg/ffprobe -version | head -1
+ workspaces: engine
- # Same static-build service as macOS, linux/amd64 flavour.
- - name: Stage bundled FFmpeg (Linux)
+ - name: System libraries (Linux)
if: matrix.platform == 'linux'
run: |
- set -euo pipefail
- mkdir -p desktop/src-tauri/ffmpeg ffdl
- curl -fsSL --retry 5 --retry-all-errors -o ffdl/ffmpeg.zip \
- "https://ffmpeg.martin-riedl.de/redirect/latest/linux/amd64/release/ffmpeg.zip"
- curl -fsSL --retry 5 --retry-all-errors -o ffdl/ffprobe.zip \
- "https://ffmpeg.martin-riedl.de/redirect/latest/linux/amd64/release/ffprobe.zip"
- unzip -o ffdl/ffmpeg.zip -d ffdl/ffmpeg-out
- unzip -o ffdl/ffprobe.zip -d ffdl/ffprobe-out
- mv "$(find ffdl/ffmpeg-out -type f -name ffmpeg | head -1)" desktop/src-tauri/ffmpeg/ffmpeg
- mv "$(find ffdl/ffprobe-out -type f -name ffprobe | head -1)" desktop/src-tauri/ffmpeg/ffprobe
- chmod +x desktop/src-tauri/ffmpeg/ffmpeg desktop/src-tauri/ffmpeg/ffprobe
- desktop/src-tauri/ffmpeg/ffmpeg -version | head -1
- desktop/src-tauri/ffmpeg/ffprobe -version | head -1
-
- # The transcriber ships too: users never install a binary (see desktop
- # decision 0010). Built from source where no official binary exists,
- # cached by version so it costs one cold build per platform.
- - name: Cache whisper-cli (unix)
- if: matrix.platform != 'windows'
- uses: actions/cache@v4
- with:
- path: whisper-stage
- key: whisper-cli-v1.8.2-${{ matrix.platform }}
+ sudo apt-get update
+ sudo apt-get install -y \
+ libfontconfig1-dev libfreetype6-dev libxkbcommon-dev libgl1-mesa-dev \
+ libasound2-dev libclang-dev cmake
- - name: Stage bundled whisper-cli (unix)
- if: matrix.platform != 'windows'
+ - name: FFmpeg development libraries (Linux)
+ if: matrix.platform == 'linux'
run: |
set -euo pipefail
- if [ ! -x whisper-stage/whisper-cli ]; then
- git clone --depth 1 --branch v1.8.2 https://github.com/ggml-org/whisper.cpp.git wbuild
- cmake -S wbuild -B wbuild/build -DCMAKE_BUILD_TYPE=Release \
- -DBUILD_SHARED_LIBS=OFF -DGGML_METAL_EMBED_LIBRARY=ON -DWHISPER_BUILD_TESTS=OFF
- cmake --build wbuild/build --config Release -j 4 --target whisper-cli
- mkdir -p whisper-stage
- cp wbuild/build/bin/whisper-cli whisper-stage/whisper-cli
- fi
- mkdir -p desktop/src-tauri/whisper
- cp whisper-stage/whisper-cli desktop/src-tauri/whisper/whisper-cli
- chmod +x desktop/src-tauri/whisper/whisper-cli
- desktop/src-tauri/whisper/whisper-cli --help >/dev/null 2>&1 || true
+ curl -fsSL --retry 5 --retry-all-errors -o ffmpeg.tar.xz "$FFMPEG_LINUX"
+ mkdir -p ffmpeg-dev
+ tar -xf ffmpeg.tar.xz -C ffmpeg-dev --strip-components=1
+ echo "FFMPEG_DIR=$PWD/ffmpeg-dev" >> "$GITHUB_ENV"
+ echo "LD_LIBRARY_PATH=$PWD/ffmpeg-dev/lib" >> "$GITHUB_ENV"
- - name: Stage bundled whisper-cli (Windows)
- if: matrix.platform == 'windows'
- shell: pwsh
- run: |
- $ErrorActionPreference = "Stop"
- New-Item -ItemType Directory -Force desktop/src-tauri/whisper | Out-Null
- curl.exe -fsSL --retry 5 --retry-all-errors -o whisper-win.zip `
- "https://github.com/ggml-org/whisper.cpp/releases/download/v1.8.2/whisper-bin-x64.zip"
- Expand-Archive whisper-win.zip -DestinationPath wdl
- Copy-Item (Get-ChildItem wdl -Recurse -Filter whisper-cli.exe | Select-Object -First 1).FullName desktop/src-tauri/whisper/
- # The official build is dynamic; its DLLs ride along in the same
- # resource folder, which is also the executable's search path.
- Get-ChildItem wdl -Recurse -Filter *.dll | ForEach-Object {
- Copy-Item $_.FullName desktop/src-tauri/whisper/
- }
+ - name: FFmpeg development libraries (macOS)
+ if: matrix.platform == 'macos'
+ run: brew install ffmpeg cmake
- - name: Stage bundled FFmpeg (Windows)
+ # bindgen needs libclang on Windows too; LLVM from chocolatey provides
+ # it. The BtbN zip unpacks to one top-level folder.
+ - name: FFmpeg development libraries (Windows)
if: matrix.platform == 'windows'
shell: pwsh
run: |
$ErrorActionPreference = "Stop"
- New-Item -ItemType Directory -Force desktop/src-tauri/ffmpeg | Out-Null
- curl.exe -fsSL --retry 5 --retry-all-errors -o ffmpeg-win.zip `
- "https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-n8.1-latest-win64-gpl-8.1.zip"
+ choco install llvm -y --no-progress
+ curl.exe -fsSL --retry 5 --retry-all-errors -o ffmpeg-win.zip "$env:FFMPEG_WINDOWS"
Expand-Archive ffmpeg-win.zip -DestinationPath ffdl
- Copy-Item (Get-ChildItem ffdl -Recurse -Filter ffmpeg.exe | Select-Object -First 1).FullName desktop/src-tauri/ffmpeg/
- Copy-Item (Get-ChildItem ffdl -Recurse -Filter ffprobe.exe | Select-Object -First 1).FullName desktop/src-tauri/ffmpeg/
- & desktop/src-tauri/ffmpeg/ffmpeg.exe -version | Select-Object -First 1
- & desktop/src-tauri/ffmpeg/ffprobe.exe -version | Select-Object -First 1
-
- - run: npm ci
- working-directory: desktop
-
- - run: npm run typecheck
- working-directory: desktop
-
- - run: npm run lint
- working-directory: desktop
-
- - run: npm test
- working-directory: desktop
-
- # Host lints on one platform: the host compiles on all three anyway,
- # and clippy findings do not vary by OS enough to pay for three runs.
- - name: Host clippy (macOS)
- if: matrix.platform == 'macos'
- run: cargo clippy --all-targets -- -D warnings
- working-directory: desktop/src-tauri
-
- # The host half of the IPC type freshness gate (EditorView and
- # friends). Here rather than in engine-tests because compiling the
- # host needs Tauri's system libraries, which this job already has.
- - name: IPC type freshness (host)
- if: matrix.platform == 'macos'
- run: |
- scripts/generate-ipc-types.sh host
- git diff --exit-code desktop/src/lib/generated
-
- # Full release build - LTO and codegen-units=1 make this the slow step.
- - run: npm run app:build
- working-directory: desktop
+ $dir = (Get-ChildItem ffdl -Directory | Select-Object -First 1).FullName
+ "FFMPEG_DIR=$dir" | Out-File -FilePath $env:GITHUB_ENV -Append
+ "LIBCLANG_PATH=C:\Program Files\LLVM\bin" | Out-File -FilePath $env:GITHUB_ENV -Append
+ "$dir\bin" | Out-File -FilePath $env:GITHUB_PATH -Append
+
+ # The shipping profile: fat LTO, panic=abort, stripped. See the
+ # workspace Cargo.toml for what each knob costs.
+ - run: cargo build --profile app -p concat
+ working-directory: engine
- # The runtime looks for the pair in the app's resource directory; a
- # bundle that lost them (or their execute bit) would ship broken while
- # this workflow reports green.
- - name: Verify FFmpeg made it into the bundle (macOS)
- if: matrix.platform == 'macos'
+ # The binary, plus the FFmpeg runtime libraries it loads on the
+ # platforms where those are not the system's.
+ - name: Stage the build (unix)
+ if: matrix.platform != 'windows'
run: |
set -euo pipefail
- app="$(find desktop/src-tauri/target/release/bundle/macos -maxdepth 1 -name '*.app' | head -1)"
- test -n "$app" || { echo "no .app produced"; exit 1; }
- for binary in ffmpeg ffprobe; do
- staged="$app/Contents/Resources/ffmpeg/$binary"
- test -x "$staged" || { echo "$staged missing or not executable"; exit 1; }
- "$staged" -version | head -1
- done
-
- - name: Verify FFmpeg made it into the bundle (Linux)
- if: matrix.platform == 'linux'
+ mkdir -p stage
+ cp engine/target/app/concat stage/
+ if [ -d ffmpeg-dev/lib ]; then mkdir -p stage/lib && cp -a ffmpeg-dev/lib/*.so* stage/lib/; fi
+ - name: Stage the build (Windows)
+ if: matrix.platform == 'windows'
+ shell: pwsh
run: |
- set -euo pipefail
- deb="$(find desktop/src-tauri/target/release/bundle/deb -name '*.deb' | head -1)"
- test -n "$deb" || { echo "no .deb produced"; exit 1; }
- dpkg-deb -x "$deb" unpacked
- # Tauri's deb puts resources straight under usr/lib/<product>/ -
- # there is no literal "resources" path component - so match the
- # binary inside its own ffmpeg/ folder wherever that landed.
- for binary in ffmpeg ffprobe; do
- staged="$(find unpacked -type f -path "*/ffmpeg/$binary" | head -1)"
- test -n "$staged" && test -x "$staged" || { echo "$binary missing from the deb or not executable"; exit 1; }
- "$staged" -version | head -1
- done
+ New-Item -ItemType Directory -Force stage | Out-Null
+ Copy-Item engine/target/app/concat.exe stage/
+ Copy-Item "$env:FFMPEG_DIR\bin\*.dll" stage/
- uses: actions/upload-artifact@v4
with:
name: Concat-${{ matrix.platform }}
- path: |
- desktop/src-tauri/target/release/bundle/dmg/*.dmg
- desktop/src-tauri/target/release/bundle/msi/*.msi
- desktop/src-tauri/target/release/bundle/nsis/*.exe
- desktop/src-tauri/target/release/bundle/deb/*.deb
- desktop/src-tauri/target/release/bundle/rpm/*.rpm
- desktop/src-tauri/target/release/bundle/appimage/*.AppImage
+ path: stage
if-no-files-found: error
-
- release:
- name: Publish release
- if: startsWith(github.ref, 'refs/tags/v') || (github.event_name == 'push' && github.ref == 'refs/heads/main')
- needs: [engine-tests, app]
- runs-on: ubuntu-latest
- permissions:
- contents: write
- steps:
- # fetch-depth 0 so every existing tag is present locally - the next
- # alpha number is computed from them.
- - uses: actions/checkout@v4
- with:
- fetch-depth: 0
-
- - uses: actions/download-artifact@v4
- with:
- path: bundles
- merge-multiple: true
-
- - name: List what is being published
- run: find bundles -type f | sort
-
- # A pushed tag releases under its own name. A green main build mints the
- # next alpha tag itself: v<version>-alpha.<n>, where <n> is one past the
- # highest alpha already tagged for this version. action-gh-release
- # creates the tag while publishing, and because it authenticates with
- # GITHUB_TOKEN, that tag does not re-trigger this workflow.
- - name: Decide the tag
- id: tag
- run: |
- set -euo pipefail
- if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
- echo "name=$GITHUB_REF_NAME" >> "$GITHUB_OUTPUT"
- exit 0
- fi
- version="$(jq -r .version desktop/src-tauri/tauri.conf.json)"
- highest=0
- for tag in $(git tag --list "v${version}-alpha.*"); do
- n="${tag##*.}"
- if [[ "$n" =~ ^[0-9]+$ ]] && (( n > highest )); then highest="$n"; fi
- done
- echo "name=v${version}-alpha.$((highest + 1))" >> "$GITHUB_OUTPUT"
-
- - uses: softprops/action-gh-release@v2
- with:
- tag_name: ${{ steps.tag.outputs.name }}
- target_commitish: ${{ github.sha }}
- name: Concat ${{ steps.tag.outputs.name }}
- # A hyphen in the tag (v0.2.0-alpha.3) marks a pre-release.
- prerelease: ${{ contains(steps.tag.outputs.name, '-') }}
- generate_release_notes: true
- body: |
- Concat is AGPL-3.0-or-later, with a plugin exception for Concat API
- plugins ([LICENSE-EXCEPTIONS.md](https://github.com/jub0t/Concat/blob/main/LICENSE-EXCEPTIONS.md)).
- The installers bundle GPL-built FFmpeg binaries, invoked as separate
- programs, and statically link GPL-3.0 espeak-ng — sources and licenses
- in [THIRD_PARTY_NOTICES.md](https://github.com/jub0t/Concat/blob/main/THIRD_PARTY_NOTICES.md).
- fail_on_unmatched_files: true
- files: |
- bundles/**/*.dmg
- bundles/**/*.msi
- bundles/**/*.exe
- bundles/**/*.deb
- bundles/**/*.rpm
- bundles/**/*.AppImage
diff --git a/.github/workflows/nix.yml b/.github/workflows/nix.yml
index a9651085..4649dfd4 100644
--- a/.github/workflows/nix.yml
+++ b/.github/workflows/nix.yml
@@ -1,6 +1,7 @@
-# Validates the Nix flake (#7) on anything the Nix build actually consumes -
-# the flake itself, the engine, the desktop app and its lockfiles - so the
-# check shows up beside the other builds instead of only on flake edits.
+# Validates the Nix flake (#7). The flake still packages the deprecated Tauri
+# app in desktop/, which no longer builds against the engine - so this runs
+# only when the flake or that tree changes, until the flake packages the
+# Slint window instead.
name: Nix
on:
@@ -10,14 +11,12 @@ on:
- "flake.nix"
- "flake.lock"
- ".github/workflows/nix.yml"
- - "engine/**"
- "desktop/**"
pull_request:
paths:
- "flake.nix"
- "flake.lock"
- ".github/workflows/nix.yml"
- - "engine/**"
- "desktop/**"
jobs:
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
index 641eda4e..b5fb8ecc 100644
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -1,322 +1,448 @@
# Concat Architecture
-A study map of the system as it stands (August 2026): what each layer owns, how
-they talk, where the sharp edges are, and what deserves attention next. File
+A study map of the system as it stands in September 2026, written just after
+two large moves: the user interface left Tauri + React for Slint, and the
+engine stopped spawning `ffmpeg`, `ffprobe` and `whisper-cli` in favour of
+linking FFmpeg and compiling whisper.cpp in. What each layer owns, how they
+talk, where the sharp edges are, and what deserves attention next. File
references are current as of this writing.
-> **September 2026:** the UI layer described below is being replaced. The
-> Slint editor window now lives in the engine workspace as the `concat` crate
-> (`engine/crates/concat`), ported from the wc-ui-rnd research tree, and
-> `desktop/` — both the React UI and the Tauri host — is deprecated. The
-> engine doctrine in §1 is unchanged and is the whole reason the swap is
-> possible: the engine owns the project, the render path and the document, so
-> the UI is the replaceable part. Everything from §2 on describes the Tauri
-> app, which still ships until `concat` is wired to the engine. The IPC
-> surface in §5 and the chains mirror in §6.1 go away with it: a Rust window
-> calls the engine crates directly, in-process, and has no reason to
-> re-author FFmpeg filter strings in a second language.
-
---
-## 1. The three layers and the one rule
+## 1. The shape of it, and the one rule
```mermaid
flowchart LR
- subgraph UI["desktop/src — React UI"]
- APP["App.tsx<br/>layout, selection, dialogs"]
- HOOKS["useEngineSession<br/>useEngineTruth<br/>usePlaybackBridge"]
+ subgraph WINDOW["engine/crates/concat — the window (Slint)"]
+ UI["ui/*.slint<br/>panes, dialogs, primitives"]
+ MAIN["src/main.rs<br/>state, bindings, demo data"]
end
- subgraph HOST["desktop/src-tauri — Tauri host"]
- CMD["39 #[tauri::command]<br/>editor_api · playback · transcribe · templates"]
- SESSION["Session<br/>(one Mutex&lt;Option&lt;Session&gt;&gt;)"]
+ subgraph HOST["concat-host · concat-speech"]
+ SESSION[Session]
+ PLAY[Playback]
+ MON[Monitor]
+ EXP[Exporter]
+ PROJ["projects · templates · media caches"]
+ SPEECH["Transcriber · Speech"]
end
- subgraph ENGINE["engine/crates — the product"]
- CORE[concat-core]
- MEDIA[concat-media]
+ subgraph ENGINE["the engine"]
PROJECT[concat-project]
- RENDER[concat-render]
EXPORT[concat-export]
+ RENDER[concat-render]
+ MEDIA[concat-media]
+ CORE[concat-core]
end
- UI -- "invoke() — JSON + ArrayBuffer" --> HOST
- HOST -- events --> UI
+ MAIN --> HOST
HOST --> ENGINE
- ENGINE -.-> FF["ffmpeg / ffprobe<br/>subprocesses"]
+ MEDIA -. links .-> FF["libavformat · libavcodec<br/>libavfilter · libswscale · libswresample"]
+ SPEECH -. compiles in .-> W["whisper.cpp · sherpa-onnx"]
```
**The rule (engine doctrine):** everything important lives in the engine
-crates. The Tauri host is plumbing; the UI renders state and sends commands.
-The engine owns the project model, the undo history, the render path, the
-document format. When a feature is being designed, the first question is
-"which engine crate does this belong to" — TS holds only what the UI
-genuinely owns (layout, selection, in-flight gestures).
-
-The engine crates keep a strict dependency arrow (see `engine/README.md`):
-`core` depends on nothing, everything else depends inward, and `concat-project`
-knows nothing about rendering. The desktop crate consumes the engine as path
-dependencies — deliberately *not* one workspace, so the engine builds, tests
-and versions alone.
-
-| Crate | LOC | Owns |
-|---|---|---|
-| `concat-core` | ~1.4k | Rational time, frame model, arena handles, timeline model. Zero dependencies. |
-| `concat-media` | ~3.0k | Everything FFmpeg: probe, subprocess decode/encode, reader pool with byte-budgeted LRU frame cache, waveform peaks. Optional `ffi` feature links FFmpeg for exact seeks. |
-| `concat-project` | ~3.3k | The document: model, every edit command (`commands.rs`, ~1.3k), undo `Editor`, concat.json round-trip. |
-| `concat-render` | ~1.3k | Compositing. `CpuCompositor` is the reference; `WgpuCompositor` behind the `gpu` feature, falls back cleanly when no GPU. |
-| `concat-export` | ~2.2k | Timeline → file: flatten, filtergraph builders (`chains.rs`), the frame-by-frame render loop. |
-| `concat-cli` | ~0.2k | Probe/render vertical slice for testing the engine without the app. |
+crates. The window renders state and issues commands; the host layer is
+plumbing between the two. The engine owns the project model, the undo
+history, the render path, the document format, and now the codecs. When a
+feature is being designed, the first question is still "which engine crate
+does this belong to".
+
+What changed since the Tauri days is that the rule no longer has to survive a
+wire. There is no IPC, no JSON boundary, no second language: the window calls
+Rust functions in-process. Two of the previous architecture's largest debts -
+the hand-maintained TypeScript mirror of the FFmpeg filter catalogue, and the
+UI re-deriving "what is on screen" in parallel with the engine's flattener -
+ceased to exist with the language they were written in. The remaining risk is
+the opposite one: with no boundary, editing logic can leak into the window
+without anyone noticing. The test for a change is whether it could be driven
+from `concat-cli` without a window; if not, it is in the wrong place.
+
+The dependency arrows point one way:
+
+```
+concat (window) → concat-speech → concat-host → {export, project, media} → core
+ → render → core
+```
+
+`concat-core` depends on nothing. `concat-project` knows nothing about
+rendering. `concat-media` is the only crate that knows FFmpeg exists.
+
+| Crate | Lines | Tests | Owns |
+|---|---|---|---|
+| `concat-core` | 1.6k | 31 | Rational time, frame model, arena handles, timeline model. Zero dependencies. |
+| `concat-media` | 3.4k | 39 | Linked FFmpeg: probe, video decode with filter graphs and frame pacing, audio decode through a graph, H.264 encode, AAC mixing through libavfilter, muxing, waveform peaks, the reader pool with its byte-budgeted frame cache, JPEG stills. |
+| `concat-project` | 3.3k | 41 | The document: model, every edit command, undo `Editor`, `concat.json` round-trip. |
+| `concat-render` | 1.3k | 28 | Compositing. `CpuCompositor` is the reference; `WgpuCompositor` behind `gpu`. |
+| `concat-export` | 2.5k | 32 | Timeline → file: flatten, filtergraph builders (`chains.rs`), the frame-by-frame render loop, the paused monitor's true frame. |
+| `concat-host` | 3.4k | 22 | Sessions, project folders and recents, media caches beside the project, the monitor's reader pool, the export slot, audio playback, templates, one-at-a-time job slots, app directories. |
+| `concat-speech` | 1.2k | 6 | Transcription (whisper.cpp in-process) and text to speech (Kokoro via sherpa-onnx), with model downloads. |
+| `concat-cli` | 0.2k | 3 | Probe/render vertical slice for testing the engine without the app. |
+| `concat` | 5.0k Rust + 14.9k Slint | 0 | The window. Every pane, dialog and primitive, and - today - a demo model that stands in for the engine. |
---
-## 2. The edit loop — how a keystroke becomes truth
+## 2. The edit loop
-The engine owns the project; the UI never mutates its own copy of the model.
-Every edit round-trips:
+The engine owns the project; the window never keeps a model of its own. Every
+edit round-trips, now as a function call:
```mermaid
sequenceDiagram
- participant UI as React component
- participant Q as useEngineSession queue
- participant H as editor_api.rs
+ participant W as window (main.rs)
+ participant S as concat_host::Session
participant E as Editor (concat-project)
- UI->>Q: dispatch({op:"splitClips", ...})
- Note over Q: commands serialise through one<br/>promise chain, seeded with editorOpen<br/>so nothing races the session install
- Q->>H: invoke("editor_apply", command)
- H->>E: editor.apply(command)
- E-->>H: Outcome + new project state
- H-->>Q: EditorView (the whole truth)
- Q->>UI: setView(next) — UI re-renders from engine state
+ W->>S: session.apply(Command::SplitClips { .. })
+ S->>E: editor.apply(command)
+ E-->>S: Outcome (minted ids) + new Project
+ S-->>W: EditorView { project, can_undo, can_redo, settings, created_id }
+ W->>W: republish the Slint models from the Project
```
-Key properties:
-
-- **One queue.** Commands serialise so responses can never land out of order
- and overwrite newer state with older (`useEngineSession.ts`).
-- **The echo layer.** Drags don't send a command per pixel. The UI previews
- the gesture locally ("echo"), using arithmetic mirrored from the engine,
- and commits **one** command per gesture on release — so undo undoes the
- drag, not a pixel of it. The echo clears only after the engine's state
- arrives, so nothing flashes back mid-flight.
-- **Undo lives in the engine** — a bounded history of full `Project` clones
+Key properties, unchanged in substance from the Tauri architecture:
+
+- **One session, one thread.** `Session` is a plain struct owned by the
+ window's event-loop thread. There is no mutex around the edit because there
+ is no second caller; long work (saves, probes, renders, decodes) takes what
+ it needs *out* of the session first and runs on its own thread.
+- **Gestures commit once.** A drag must not send a command per pixel. The
+ Tauri UI previewed gestures locally ("echo") and committed one command on
+ release, so undo undoes the drag. The Slint window's demo model does the
+ same thing in its `Gesture` state machine; that shape survives the wiring.
+- **Undo lives in the engine** - a bounded history of full `Project` clones
(depth 200), not command inversion.
-- **Autosave** debounces 1.5 s after any state change; the engine writes the
- document (temp file + rename, so a crash mid-save can't truncate the
- project).
-- **Wire types are generated.** `EditorView`, `Command` and friends are
- ts-rs exports from the Rust types into `desktop/src/lib/generated/`;
- CI fails if the committed files drift from the Rust source of truth.
+- **Saving is one code path.** `Session::prepare_save` hands back the folder
+ and the document; `projects::save` writes to a temporary file and renames,
+ so a crash mid-save cannot truncate the project.
+- **The document format is frozen by the documents that exist.** Projects
+ saved as `wolfcut.json` still open; new ones are written as `concat.json`,
+ with the version key the app's current name. The reader never looks at
+ that key.
---
-## 3. Preview & playback — two clocks, one truth
+## 3. Media: linked, not spawned
+
+`concat-media` used to pipe raw RGBA out of an `ffmpeg` child process and
+parse `ffprobe`'s JSON. It now calls the libraries. What that bought, and
+what it cost:
+
+- **Every frame carries its real presentation timestamp**, so variable
+ frame-rate material no longer drifts through a pipe that had no
+ timestamps at all.
+- **Seeks are frame-accurate**: the container lands on the keyframe before
+ the target and the decoder discards up to it. The reader pool's "roll
+ forward or seek" policy is now guided by timestamps rather than by counting
+ frames ordinally.
+- **Rotation is honoured**: portrait phone footage is turned the way every
+ player turns it, by reading the display matrix off the codec parameters
+ (where FFmpeg 7 moved it) and inserting the same `transpose`/`hflip,vflip`
+ ffmpeg's own autorotate would.
+- **A build needs the FFmpeg development libraries.** `ffmpeg-the-third`
+ binds them at build time with bindgen, so a build machine needs headers,
+ import libraries and libclang. Homebrew serves on macOS; Windows and older
+ Linux distributions use a BtbN `shared` build via `FFMPEG_DIR`. This is the
+ price of the move and it is paid in CI configuration, not in code.
+
+### 3.1 The video decoder
+
+`decode::Decoder` does in one place what the subprocess did with flags:
-Audio and video take different paths, deliberately (desktop decisions 0005,
-0009 — see §7):
+```
+container seek (keyframe) → discard frames before `start`
+ → [pacer: pick the source frame on screen at the next output instant]
+ → libavfilter graph: [rotate] → scale → [effect chain → guard scale] → format=rgba
+ → copy rows out of the padded buffer into a Frame
+```
+
+The graph is built lazily from the first decoded frame's real format and
+size, and rebuilt if a later frame differs. The clip's effect chain is a
+plain FFmpeg filter string, exactly the string the export mix uses for
+audio, validated so it cannot escape its slot. Pacing (`at_rate`) duplicates
+and drops source frames so exactly one comes out per output instant; without
+it the decoder yields every source frame with its timestamp, which is what
+the reader pool wants. `looping` repeats the last frame forever, which is how
+a still image behaves like footage.
+
+### 3.2 The audio decoder
+
+`samples::AudioDecoder` is the one door for sound-as-numbers. Callers say
+what window of the file they want, which filters to run it through, and the
+rate, layout and format they want back; libavfilter's `atrim`, the filters,
+`aresample` and `aformat` do the rest in one graph. Three callers, three
+shapes:
+
+| Caller | Rate | Layout | Format | Filters |
+|---|---|---|---|---|
+| waveform peaks | 48 kHz | mono | s16 | none |
+| playback cache | 48 kHz | stereo | s16 | speed stages + the clip's chain |
+| transcription | 16 kHz | mono | f32 | none |
+
+Because the playback cache and the export mix run the clip's speed and chain
+through the same filter strings, what the preview mixes is what the export
+renders.
+
+### 3.3 Encode, mix, mux
+
+- `encode::Encoder` is libx264 through libavcodec, RGBA → yuv420p through
+ libswscale, into an MP4 with `+faststart`. `encode::jpeg` is the same
+ machinery with the MJPEG encoder, for posters.
+- `audio::mix_to_file` parses the same `filter_complex` string
+ `audio::mix_graph` always produced - the pure, unit-tested planner is
+ untouched - into a libavfilter graph with one `abuffer` per clip bound to
+ the `[N:a]` labels, an `aformat` landing on the AAC encoder's format, and
+ an `abuffersink` sized to the encoder's frame. Inputs are fed round-robin
+ and the sink drained after every push, so no input runs ahead by more than
+ a frame.
+- `audio::mux` copies the encoded streams into the final container,
+ stopping at the shorter one.
+
+### 3.4 The reader pool
+
+Unchanged in purpose: one warm decoder per (file, size, chain), a
+byte-budgeted LRU of decoded frames in front, "roll forward if the target is
+close ahead, seek otherwise". The two backends it used to switch between
+(linked when available, pipe otherwise) collapsed into one.
+
+---
+
+## 4. Preview and playback: two clocks, one truth
```mermaid
flowchart TB
- subgraph AUDIO["Audio — engine-owned (playback.rs)"]
- DEC["decode worker pool<br/>clip span → WAV in cache/audio"]
+ subgraph AUDIO["Audio — concat_host::playback"]
+ DEC["decode workers<br/>clip span → WAV in cache/audio"]
MMAP["mmap'd PCM"]
CPAL["one cpal callback mixes everything"]
- CLK["device sample counter →<br/>AtomicU64 position_micros"]
+ CLK["device sample counter →<br/>position(), 30 Hz position events"]
DEC --> MMAP --> CPAL --> CLK
end
- subgraph VIDEO["Video — pulled by the UI"]
- TRUTH["useEngineTruth pull loop<br/>one request in flight, snapped to frame grid,<br/>decode-ahead clamped to 1–3 frames"]
- PF["preview_frame → raw RGBA<br/>(tauri ipc::Response = ArrayBuffer, no JSON)"]
- CANVAS["putImageData on the monitor canvas"]
- TRUTH --> PF --> CANVAS
+ subgraph VIDEO["Video — concat_host::preview::Monitor"]
+ PULL["window pulls one frame per instant<br/>prefetch warms the next few"]
+ PF["preview_frame → raw RGBA"]
+ IMG["slint::Image from the buffer"]
+ PULL --> PF --> IMG
end
- CLK -- "transport events" --> TRUTH
+ CLK -- "the window follows" --> PULL
```
-- **The audio device is the only clock.** The transport thread publishes
- position; the UI interpolates between events. Video chases audio, never
- the other way around.
-- **The monitor draws the engine's real composited frame** — the same
- pixels the exporter would produce — at `previewQuality` (default 0.5)
- to keep the per-frame RGBA copy over IPC affordable.
-- **Effects preview in two grades:** live approximations (CSS filters /
- canvas ops in `lib/effects.ts`) during interaction, engine truth for the
- monitor. The export is "only ever a higher-quality version of what you
- already saw."
-- All 15 byte-shipping commands use `tauri::ipc::Response` (ArrayBuffer),
- so no base64 or number-array serialisation on the hot paths. The one
- exception: `write_artwork(bytes: Vec<u8>)` serialises inbound as a JSON
- number array.
+- **The audio device is the only clock.** `Playback::position()` reads the
+ mix callback's sample counter; `PlaybackEvents::position` pushes it at
+ 30 Hz while playing. Video chases audio, never the other way around.
+- **The monitor shows the engine's real composited frame** - the same pixels
+ the exporter would produce - at a preview size the window chooses.
+- **Nothing here knows about a window.** `Playback` reports through a trait
+ object; `Monitor` returns bytes. The window decides which thread runs what,
+ and hands results back to the Slint event loop with
+ `slint::invoke_from_event_loop`. That seam is where the wiring in §7 goes.
---
-## 4. Export
+## 5. Export
-Hybrid pipeline in `concat-export`:
+Hybrid pipeline in `concat-export`, unchanged in shape:
-- **Picture:** flattened timeline → per-frame composition in-process
- (CPU or wgpu) → raw frames piped into one `ffmpeg` encode subprocess.
-- **Audio:** one FFmpeg filtergraph mixes every audible clip in a single
- pass.
-- **Transitions are lowered before either path runs** — a cross-fade
- becomes overlapping clips with opacity ramps on doubled track indices,
- so the compositor never knows transitions exist.
-- Filter strings are byte-pinned by tests on **both** sides of the mirror
- (see §6, the chains mirror).
+- **Picture:** flattened timeline → per-frame composition in-process (CPU or
+ wgpu) → the linked H.264 encoder.
+- **Audio:** one filtergraph mixes every audible clip in a single pass.
+- **Transitions are lowered before either path runs** - a cross-fade becomes
+ overlapping clips with opacity ramps and fade filters - so the compositor
+ never knows transitions exist.
+- The host's `export::request` flattens the open session and adds the
+ destination and quality; `export::run` reports progress through a callback
+ and stops on a flag; `Exporter` is the one-at-a-time slot.
---
-## 5. IPC surface (39 commands)
-
-| Group | Commands |
-|---|---|
-| Editor session | `editor_open/apply/undo/redo/save/state/close` |
-| Projects & recents | `create_project`, `open_project`, `recent_projects`, `forget_project`, `project_preview` |
-| Media | `probe_media`, `read_media_bytes` (capped), `extract_peaks`, `extract_filmstrip`, `read_artwork`, `write_artwork` |
-| Preview & transport | `preview_frame`, `preview_prefetch`, `transport_play/pause/seek`, `audio_set_clips` |
-| Export | `export_project`, `cancel_export` |
-| Templates | 5 commands |
-| Transcription | 7 commands |
+## 6. Speech
+
+- **Transcription** loads a ggml Whisper model through `whisper-rs`, keeps it
+ loaded until another is asked for, feeds it 16 kHz mono floats straight
+ from `AudioDecoder`, and reads segments back in centiseconds. On macOS the
+ Metal build runs the encoder on the GPU. Cancellation is whisper's abort
+ callback reading the job's flag.
+- **Text to speech** is Kokoro through sherpa-onnx's official bindings,
+ statically linked from prebuilt libraries the sys crate downloads at build
+ time. Narration lands as a WAV in the project's `audio/` folder - not
+ `cache/`, because a clip on the timeline points at it.
+- Both download models on demand into the app's data directory
+ (`app.concat.editor` under the platform's application-support folder) and
+ refuse a second concurrent run through `SingleFlight`.
---
-## 6. Footguns and technical debt
-
-The honest list, ordered by how much they matter. Nothing here is on fire;
-several are traps waiting for a contributor who doesn't know the invariant.
-
-### 6.1 The chains mirror (highest-leverage debt)
-
-`desktop/src/lib/effects.ts` + `filters.ts` are **hand-maintained TS mirrors**
-of `engine/crates/concat-export/src/chains.rs`. Both sides emit FFmpeg filter
-strings; the only thing keeping them identical is a corpus of pinned-string
-tests on each side (`chains.rs:523+` vs `effects.test.ts` / `filters.test.ts`).
-This is the largest deliberate violation of "the engine owns it": the TS side
-exists so previews can be computed without IPC, but every new effect must be
-written twice and pinned twice. It also means `usePlaybackBridge` builds a
-filter string *in TS* and ships it to Rust — the UI is composing engine
-commands in the engine's own language. Worth studying: could the engine emit
-the preview approximation (CSS filter string, canvas op list) alongside the
-chain, making TS a renderer of engine-provided looks instead of a second
-author?
-
-### 6.2 Lock discipline in `playback.rs`
-
-`playback.rs` uses `.lock().unwrap()` on shared state ~14 times across four
-kinds of threads (mix callback, transport, decode workers, cache sweep). One
-panic in any decode worker poisons the mutex and cascades unwraps through the
-others. Meanwhile `editor_api.rs` converts poison to a `String` error at
-`:120,188` but silently ignores it at `:266` — three different poison policies
-in one codebase. Pick one (probably: poison = clear the poisoned state and
-carry on, since every cache here is rebuildable).
-
-### 6.3 Unbounded growth
-
-- `redo: Vec<Project>` in the engine editor has **no cap** (undo is capped at
- 200) — each entry is a deep clone of the whole project
- (`concat-project/src/editor.rs:28`).
-- `desktop/src/lib/assets.ts` — the `peaks`/`strips`/`stripFrames` maps grow
- per media id with no eviction, and no `ImageBitmap.close()` is ever called;
- GPU-backed bitmaps wait for GC. A long session with many imports leaks
- VRAM-adjacent memory.
-
-### 6.4 Swallowed errors
-
-Twenty `catch(() => undefined)`-shaped sites. Most are defensible
-(fire-and-forget artwork writes), but four are in the engine-session queue
-itself (`useEngineSession.ts:101,118,129,206`) — a failed **undo/redo**
-drains silently. Host side: `export.rs:38` drops progress-event send
-failures; `playback.rs:251` drops audio-error emission failures. A grep for
-`catch(() =>` is a good periodic audit.
-
-### 6.5 Trust boundaries
-
-- `Preview.tsx:869` feeds `preview_frame` bytes straight into
- `new ImageData(bytes, w, h)` — a short buffer throws inside the draw loop
- rather than failing gracefully.
-- Counterexample done right: `peaks_bytes` validates cached bytes with
- `plausible_peaks` before trusting them (`lib.rs:233`).
-
-### 6.6 Default builds don't test the real paths
-
-`cargo test` at default features exercises **neither** the wgpu compositor
-(`gpu` off) **nor** the FFI decoder (`ffi` off) — the two code paths preview
-and scrubbing actually rely on in a bundled build. CI should run at least one
-job with `--features gpu,ffi`.
-
-### 6.7 Frontend hotspots
-
-- `App.tsx` holds **30 `useState`s**; `<Preview>` takes ~46 props,
- `<TimelinePanel>` ~39, `<MediaBin>` ~28. The hooks
- (`useEngineSession/Truth/PlaybackBridge`) are the right pattern — more of
- App.tsx's state wants to migrate into them (selection + tool state is the
- obvious next hook).
-- The five biggest files: `TimelinePanel.tsx` 1786, `App.tsx` 1583,
- `Preview.tsx` 1429, `MediaBin.tsx` 1072, `lib/effects.ts` 694.
-- `lib/monitor.ts` re-derives "what is on screen / what exports" in parallel
- with the engine's flattener — same mirror concern as §6.1 in miniature.
-
-### 6.8 Panics in host runtime paths
-
-Thread spawns `.expect()` in `playback.rs:271,289,314,442`;
-`editor_api.rs:190` `.expect("just set")`. Engine `.expect()`s exist in
-`decode.rs`, `encode.rs`, `pool.rs`, `time.rs:97` ("rational overflowed
-i64") — most are genuine invariants, but each is a crash-not-error if wrong.
-
-### 6.9 Housekeeping
-
-- **Zero TODO/FIXME/HACK comments** in the entire codebase — invariants live
- in prose comments instead. Keep it that way.
-- `desktop/src-tauri/src/export.rs` is a 42-line shim left over from moving
- export into the engine; fold it into `lib.rs` or leave it as the seam.
+## 7. The window, and the wiring that is not done
----
+`engine/crates/concat` is the Slint tree ported from the wc-ui-rnd research
+repository: 14.9k lines of `.slint` across the workspace (dockable seats
+holding four views), the timeline (lanes, tabs, track headers, tray), the
+inspector, the media bin, the launch screen, the export and settings sheets,
+menus, tooltips, toasts, and the primitives under them. Fonts and effect
+previews are embedded in the binary. It builds under the workspace's edition
+and lints and it runs.
+
+What drives it today is `src/main.rs`: 5k lines binding every callback of
+the `Editor` global to a **demo model** - `Studio`, `TimelineDoc`,
+`ClipDoc`, `Library`, `ExportState`, `SettingsState` - with fake media,
+synthetic waveforms and simulated downloads. That model exists so the UI
+could be built and judged before the engine was attached. It is the thing to
+replace, and replacing it is the single largest open item in the codebase.
-## 7. The lost decision records
+The mapping is direct, which is why it was worth porting the window first:
-Code comments still cite `docs/decisions/…` paths that were deleted in commit
-`36168ac` ("docs: drop the decision logs"). Four ids are referenced from
-surviving code (`ffi.rs:9`, `gpu.rs:4`, `playback.rs:3`). The titles, for
-recovery via `git show` if wanted:
+| Demo state in `main.rs` | Becomes |
+|---|---|
+| `TimelineDoc` / `ClipDoc` / `TrackDoc` | `concat_project::Project` read through `Session::project()`; edits become `Command`s |
+| `Library` (bin rows, counts, filter) | `Project::media` plus `concat_host::media::probe` on import |
+| `wave_path` from a seeded synthesiser | `concat_host::media::peaks` → the same SVG path builder |
+| the empty preview stage | `Monitor::frame` → `slint::Image` |
+| `playing` / `playhead` ticked by a timer | `Playback` and its clock |
+| `ExportState`'s simulated progress | `Exporter` + `export::run` on a thread, progress through `invoke_from_event_loop` |
+| `demo_transcribers` / `demo_voices` | `Transcriber::status` / `Speech::status`, downloads with real progress |
+| `StartState` / `demo_recents` | `projects::create` / `open` / `list` / `forget` |
+| `Dock` layout, gestures, selection, tool, zoom | stay in the window: this is what the UI genuinely owns |
+
+Recommended order, each step leaving a runnable app: projects and recents →
+media import with real probes and peaks → the timeline reading `Project`
+and writing `Command`s → the monitor → playback → export → speech →
+templates. Threading pattern for all of it: `std::thread::spawn` the work,
+`slint::invoke_from_event_loop` the result; never block the event loop on a
+decoder.
-- *engine:* 0002 ffmpeg-over-a-pipe · 0003 arena-handles-not-pointers ·
- 0004 cpu-compositor-first · 0005 license-mpl-2 · 0006
- transitions-in-the-exporter · 0007 engine-owns-the-project · 0008
- templates-are-documents-plus-slots · 0009 the-engine-owns-the-render-path
-- *desktop:* 0001 tauri-over-electron · 0002 the-hot-surfaces-are-not-the-dom
- · 0003 csp · 0004 audio-preview-in-the-webview (superseded) · 0005
- audio-playback-in-the-engine · 0006 multiple-timelines · 0007
- preview-approximation-layer · 0008 bundled-ffmpeg-and-releases · 0009
- the-monitor-streams-the-true-frame · 0010 the-app-installs-its-own-tools
+---
-Either restore them or strip the dangling references — a comment pointing at
-a deleted document is worse than no comment.
+## 8. Footguns and technical debt
+
+The honest list, ordered by how much they matter. Nothing here is on fire,
+but several are traps for a contributor who does not know the invariant.
+
+### 8.1 The window is a demo (highest priority)
+
+Everything in §7. Until the wiring lands, the app cannot open a real
+project. The host and engine are complete enough to support every feature
+the Tauri app had; the gap is entirely in `main.rs`.
+
+### 8.2 Title clips have no rasteriser
+
+The Tauri UI rasterised text clips to PNGs in the webview (`rasterize.ts`)
+and shipped them into the export as image clips - `ExportRequest` still
+expects titles that way. Nothing in the Rust tree draws text into a `Frame`.
+This needs a text rasteriser (a `cosmic-text`/`fontdue`-shaped crate against
+the embedded Inter and Synonym faces) in the engine, so titles render
+identically in the monitor and the export. Until then, text clips are
+timeline objects with no picture.
+
+### 8.3 The decoder's pacing changed semantics slightly
+
+The subprocess applied `-r` (output pacing) after `-vf` (the effect chain),
+so a `fade` in a chain counted *source* frames. `Decoder` runs the chain on
+each *paced* output frame, so fades count output frames - which is what
+`resolve_transitions` assumes when it computes `nb_frames` at the output
+rate, and arguably more correct. Nobody has yet compared an exported dissolve
+frame-for-frame against the old build. Do that before trusting it.
+
+### 8.4 Lock discipline in `playback.rs`
+
+Carried over unchanged: 28 `unwrap`/`expect` sites across four kinds of
+threads (mix callback, transport, decode workers, cache sweep). One panic in
+a decode worker poisons a mutex and cascades. Pick one poison policy
+(probably: clear the poisoned state and carry on, since every cache here is
+rebuildable) and apply it once.
+
+### 8.5 Unbounded growth
+
+- `redo: Vec<Project>` in `concat-project`'s editor has **no cap** (undo is
+ capped at 200); each entry is a deep clone of the whole project.
+- The reader pool's cache is bounded; the window's own image cache (once
+ wired) needs a bound too, or a long session of filmstrips accumulates
+ textures. Filmstrips are now raw RGBA `Frame`s with no disk cache (the old
+ JPEG cache went with the subprocess); posters are still cached as JPEG.
+
+### 8.6 The mix has no end-to-end test
+
+`mix_graph` is pinned by a dozen unit tests, and the export loop has tests
+that render synthetic video; `mix_to_file` and `mux` are exercised by no
+test with real audio. The old subprocess path was covered by
+`desktop/src-tauri/tests/export_integration.rs`, which died with the host.
+Write the replacement against a generated tone before anything else touches
+the audio path.
+
+### 8.7 Build-environment fragility
+
+- FFmpeg 7 is the floor (the display matrix moved to `coded_side_data` in
+ 7.0; the code reads it there). Ubuntu 24.04 ships 6.1, hence the BtbN
+ download in CI. A contributor with a distribution FFmpeg older than 7 gets
+ a compile error deep in the bindings, not a message.
+- `whisper-rs-sys` builds whisper.cpp with cmake at build time; `sherpa-onnx-
+ sys` downloads prebuilt static libraries at build time. A clean build needs
+ the network twice (three times counting `skia-bindings`) and several
+ minutes. The workspace lock resolves 746 crates.
+- The CI in `.github/workflows/build.yml` was rewritten for all of this and
+ has not yet run green on all three platforms. Windows in particular
+ (libclang via chocolatey, `FFMPEG_DIR`, MSVC for whisper.cpp) is untested.
+
+### 8.8 Dead code still in the repository
+
+- `desktop/` no longer builds: its host Rust moved into `concat-host` and
+ `concat-speech`, the media API it used is gone, and the Tauri commands were
+ its only reason to exist. It stays as a reference for UI behaviour until
+ the Slint window reproduces it, then it should be deleted along with
+ `scripts/generate-ipc-types.sh`, the `types` features of `concat-project`
+ and `concat-export` (ts-rs exports for a TypeScript that no longer exists),
+ and the `desktop/src/lib/...` references in `chains.rs`'s comments.
+- `flake.nix` packages the Tauri app and therefore does not build either.
+- `ExportClip`, `ExportRequest` and `PreviewFrameRequest` are wire types
+ shaped for JSON; with no wire, `concat-export` could take the engine's
+ `Project` and settings directly and drop the flattening-through-a-DTO.
+
+### 8.9 Panics in runtime paths
+
+Engine `.expect()`s exist in `decode.rs`, `encode.rs`, `pool.rs` and
+`time.rs` ("rational overflowed i64"). Most are genuine invariants
+("the graph has an input" right after building it); each is a crash-not-error
+if wrong. The host's `templates.rs` and `media.rs` counts are dominated by
+tests, which is fine.
+
+### 8.10 Housekeeping
+
+- **Zero TODO/FIXME/HACK comments** in the tree. Invariants live in prose
+ comments instead; keep it that way.
+- Comments in `concat/ui/*.slint` still cite `web/` (the React reference in
+ the wc-ui-rnd repository) and `desktop/` (the Tauri app). They are history,
+ not pointers; rewrite them as the files are touched.
+- Code comments still cite `docs/decisions/…` paths deleted long ago. Either
+ restore the records or strip the references.
+- `cargo fmt --check` is not clean across the older crates and is not
+ enforced in CI; the new crates are formatted. Decide whether to enforce it.
---
-## 8. Test coverage map
+## 9. Test coverage map
| Area | Tests | Gaps |
|---|---|---|
-| Engine crates | 179 (core 31, media 48, project 41, render 28, export 32, cli 3) | `gpu`/`ffi` features untested at defaults (§6.6) |
-| Desktop Rust | 15 | **Zero** for `lib.rs` (all 25 commands), `editor_api.rs` — the session lifecycle that produced three of this week's bug reports |
-| Frontend lib | 98 across 9 files | solid |
-| Frontend components/hooks | **0** | `useEngineSession` (queue, echo, autosave) is the highest-value target; the big components need at least smoke renders |
+| Engine crates | 174 (core 31, media 39, project 41, render 28, export 32, cli 3) | `mix_to_file`/`mux` untested end to end (§8.6); `gpu` feature untested at defaults |
+| Host | 22 | `Playback` (threads, cpal) has unit tests only for the WAV walk and the sweep; no test drives the decode workers |
+| Speech | 6 | Model downloads and whisper itself are network- and model-bound; test with `tiny.en` behind an opt-in feature |
+| Window | 0 | Nothing yet; once wired, drive `Session` from a headless harness and snapshot the Slint models |
+
+The real-media tests generate their own fixtures with the linked encoder
+(a 90-frame colour ramp, a 60-frame strip) so they run anywhere the engine
+builds. There is no corpus of real-world files; portrait phone footage,
+variable frame rate, and odd containers are the cases most likely to bite.
---
-## 9. Where to improve first (opinionated)
+## 10. Where to improve first (opinionated)
-1. **Session-lifecycle tests for `editor_api.rs`** — open/apply/close against
- temp project folders. The "fresh project unopenable" and "import before
- open" bugs both lived here; both were testable without a UI.
-2. **One poison policy** in `playback.rs` (§6.2) — mechanical, removes the
+1. **Wire the window** (§7), in the order given, committing after each step.
+2. **A tone-based mix test** (§8.6) before touching the audio path again.
+3. **A text rasteriser in the engine** (§8.2) - the one feature the Rust
+ tree cannot express today.
+4. **Delete `desktop/`** and the ts-rs vestiges (§8.8) once the window covers
+ its features; it is a liability every day it looks like code.
+5. **One poison policy** in `playback.rs` (§8.4) - mechanical, removes the
cascade failure mode.
-3. **Cap the redo stack and evict `assets.ts` bitmaps** (§6.3) — small
- patches, close real leaks.
-4. **Surface the swallowed undo/redo failures** (§6.4) — they already have a
- toast pathway.
-5. **A `--features gpu,ffi` CI job** (§6.6).
-6. **The chains mirror** (§6.1) — the big one; a design conversation, not a
- patch. Engine-emitted preview looks would delete ~700 lines of mirrored
- TS and the entire pinned-string coupling.
-7. **Decompose App.tsx state into hooks** (§6.7) — as touched, not as a
- rewrite.
+6. **Cap the redo stack** (§8.5) - a small patch, a real leak.
+7. **Green CI on all three platforms** (§8.7), then package the window and
+ restore automatic alpha releases.
+8. **Frame-for-frame compare an exported dissolve** against a build from
+ before the FFI move (§8.3).
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f35e4550..732b5bff 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,10 @@ behaviour.
## Unreleased
+- The engine links FFmpeg and compiles whisper.cpp in: nothing spawns an
+ `ffmpeg`, `ffprobe` or `whisper-cli` process any more, so a build is one
+ binary with no tools to stage beside it. Seeks are frame-accurate and
+ every decoded frame carries its real timestamp.
- The editor window is being rebuilt in Slint as the `concat` crate inside
the engine workspace: a native Rust window with no web view, embedding its
own fonts and effect previews. It is not what releases ship yet — the
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 3b2fd1de..5602f0f3 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -31,8 +31,9 @@ nix develop # shell with ffmpeg, whisper and the Rust/Node toolchains
npm run app # run the editor
```
-Without Nix you will need Rust (see `rust-version` in `engine/Cargo.toml`)
-and `ffmpeg`/`ffprobe` on `PATH`. Then:
+Without Nix you will need Rust (see `rust-version` in `engine/Cargo.toml`),
+the FFmpeg 7+ development libraries (`brew install ffmpeg`; see
+`engine/README.md` for Windows and Linux), cmake and a C++ compiler. Then:
```sh
cd engine && cargo run -p concat
diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md
index 7ff91b52..5decc6a1 100644
--- a/THIRD_PARTY_NOTICES.md
+++ b/THIRD_PARTY_NOTICES.md
@@ -2,18 +2,27 @@
## FFmpeg
-Concat bundles unmodified `ffmpeg` and `ffprobe` binaries and invokes them as
-separate child processes; the app does not link against them.
+Concat links FFmpeg's libraries - libavformat, libavcodec, libavfilter,
+libswscale and libswresample - through the `ffmpeg-the-third` crate. The
+Slint app spawns no `ffmpeg` or `ffprobe` process.
-- **Windows**: static GPL builds from the BtbN autobuild project
- (`ffmpeg-n8.1-latest-win64-gpl`), built with x264.
- Source and build scripts: https://github.com/BtbN/FFmpeg-Builds
-- **macOS**: static builds by Martin Riedl.
- Source and build info: https://ffmpeg.martin-riedl.de
+FFmpeg is licensed under the LGPL-2.1-or-later; builds that include x264
+(which the H.264 export uses) are GPL-2.0-or-later. Concat's own sources are
+AGPL-3.0-or-later, and section 13 of the GPL-3.0 and AGPL-3.0 expressly
+permits linking the two, so a distributed build may be conveyed on those
+terms. Which FFmpeg a binary carries depends on the machine that built it:
+Homebrew's on macOS, a BtbN `shared` build (https://github.com/BtbN/FFmpeg-Builds)
+on Windows and in CI. FFmpeg source code: https://ffmpeg.org/download.html
-FFmpeg is licensed under the LGPL-2.1-or-later, with the bundled builds
-compiled as GPL-2.0-or-later (they include x264). FFmpeg source code:
-https://ffmpeg.org/download.html
+The deprecated Tauri app in `desktop/` still bundles `ffmpeg` and `ffprobe`
+binaries as separate programs; that arrangement goes away with it.
+
+## whisper.cpp
+
+Transcription compiles whisper.cpp (https://github.com/ggml-org/whisper.cpp,
+MIT) and ggml into the app through the `whisper-rs` crate. Whisper models
+are downloaded on demand from https://huggingface.co/ggerganov/whisper.cpp
+(MIT) and never bundled.
## Slint — used under GPL-3.0-only
@@ -67,12 +76,6 @@ Neither licence permits selling the fonts on their own, and the OFL requires
that Inter's copyright notice and licence travel with any redistribution. Both
are satisfied by shipping the `fonts/` directory as it stands.
-## Whisper models (optional download)
-
-Auto-captions can download ggml Whisper models from
-https://huggingface.co/ggerganov/whisper.cpp (MIT). Models are fetched on
-demand and never bundled.
-
## sherpa-onnx and Kokoro voices
Text to speech links the sherpa-onnx runtime statically
diff --git a/desktop/README.md b/desktop/README.md
index 0dfce30b..fba3352a 100644
--- a/desktop/README.md
+++ b/desktop/README.md
@@ -7,9 +7,10 @@
> [!WARNING]
> **Deprecated.** Concat is moving from Tauri + React to Slint. The
> replacement is the `concat` crate in [`../engine/crates/concat`](../engine/crates/concat),
-> and that is where UI work goes now. This tree stays only until the Slint
-> window can do everything it does, because it is still what releases ship:
-> bug fixes are welcome, features are not.
+> and that is where UI work goes now. This tree no longer builds: its host
+> Rust moved into `engine/crates/concat-host` and `concat-speech`, and the
+> engine links FFmpeg instead of spawning it. It stays only as a reference
+> for the UI behaviour the Slint window is reproducing, and will be deleted.
The editor front end. Tauri v2 + React 19 + Tailwind v4, talking to the Rust
engine in [`../engine`](../engine).
diff --git a/desktop/src-tauri/tauri.conf.json b/desktop/src-tauri/tauri.conf.json
index a898e3fd..2b5d72f2 100644
--- a/desktop/src-tauri/tauri.conf.json
+++ b/desktop/src-tauri/tauri.conf.json
@@ -2,7 +2,7 @@
"$schema": "https://schema.tauri.app/config/2",
"productName": "WolfCut",
"version": "0.2.0",
- "identifier": "dev.relay.editor",
+ "identifier": "app.concat.editor",
"build": {
"beforeDevCommand": "npm run dev",
"devUrl": "http://localhost:1420",
diff --git a/engine/Cargo.lock b/engine/Cargo.lock
index 6ae92d21..1674d782 100644
--- a/engine/Cargo.lock
+++ b/engine/Cargo.lock
@@ -107,8 +107,8 @@ dependencies = [
"accesskit_consumer",
"hashbrown 0.16.1",
"static_assertions",
- "windows",
- "windows-core",
+ "windows 0.62.2",
+ "windows-core 0.62.2",
]
[[package]]
@@ -132,6 +132,17 @@ version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
+[[package]]
+name = "aes"
+version = "0.8.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
+dependencies = [
+ "cfg-if",
+ "cipher",
+ "cpufeatures",
+]
+
[[package]]
name = "ahash"
version = "0.8.12"
@@ -178,6 +189,28 @@ version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
+[[package]]
+name = "alsa"
+version = "0.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ed7572b7ba83a31e20d1b48970ee402d2e3e0537dcfe0a3ff4d6eb7508617d43"
+dependencies = [
+ "alsa-sys",
+ "bitflags 2.13.1",
+ "cfg-if",
+ "libc",
+]
+
+[[package]]
+name = "alsa-sys"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "db8fee663d06c4e303404ef5f40488a53e062f89ba8bfed81f42325aafad1527"
+dependencies = [
+ "libc",
+ "pkg-config",
+]
+
[[package]]
name = "android-activity"
version = "0.6.1"
@@ -187,7 +220,7 @@ dependencies = [
"android-properties",
"bitflags 2.13.1",
"cc",
- "jni",
+ "jni 0.22.4",
"libc",
"log",
"ndk",
@@ -284,6 +317,9 @@ name = "arbitrary"
version = "1.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1"
+dependencies = [
+ "derive_arbitrary",
+]
[[package]]
name = "arg_enum_proc_macro"
@@ -657,6 +693,15 @@ dependencies = [
"no_std_io2",
]
+[[package]]
+name = "block-buffer"
+version = "0.10.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
+dependencies = [
+ "generic-array",
+]
+
[[package]]
name = "block2"
version = "0.5.1"
@@ -736,6 +781,12 @@ dependencies = [
"syn 3.0.4",
]
+[[package]]
+name = "byteorder"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
+
[[package]]
name = "byteorder-lite"
version = "0.1.0"
@@ -748,6 +799,44 @@ version = "1.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
+[[package]]
+name = "bzip2"
+version = "0.4.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bdb116a6ef3f6c3698828873ad02c3014b3c85cadb88496095628e3ef1e347f8"
+dependencies = [
+ "bzip2-sys",
+ "libc",
+]
+
+[[package]]
+name = "bzip2"
+version = "0.5.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "49ecfb22d906f800d4fe833b6282cf4dc1c298f5057ca0b5445e5c209735ca47"
+dependencies = [
+ "bzip2-sys",
+]
+
+[[package]]
+name = "bzip2"
+version = "0.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c"
+dependencies = [
+ "libbz2-rs-sys",
+]
+
+[[package]]
+name = "bzip2-sys"
+version = "0.1.13+1.0.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14"
+dependencies = [
+ "cc",
+ "pkg-config",
+]
+
[[package]]
name = "calloop"
version = "0.13.0"
@@ -799,12 +888,6 @@ dependencies = [
"wayland-client",
]
-[[package]]
-name = "camino"
-version = "1.2.5"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bb1307f12aa967b5a58416e87b3653360e0fd614a016b6e970db08fecbb1b80d"
-
[[package]]
name = "cc"
version = "1.4.4"
@@ -817,6 +900,12 @@ dependencies = [
"shlex 2.0.1",
]
+[[package]]
+name = "cesu8"
+version = "1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6d43a04d8753f35258c91f8ec639f792891f748a1edbd759cf1dcea3382ad83c"
+
[[package]]
name = "cexpr"
version = "0.6.0"
@@ -860,6 +949,26 @@ dependencies = [
"windows-link",
]
+[[package]]
+name = "cipher"
+version = "0.4.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad"
+dependencies = [
+ "crypto-common",
+ "inout",
+]
+
+[[package]]
+name = "clang"
+version = "2.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b2ed30b30e1e17966b55ddd0b9dd2758fd824f0a6da6875c8abb17e6aba47c28"
+dependencies = [
+ "clang-sys",
+ "libc",
+]
+
[[package]]
name = "clang-sys"
version = "1.9.1"
@@ -929,6 +1038,15 @@ dependencies = [
"hashbrown 0.16.1",
]
+[[package]]
+name = "cmake"
+version = "0.1.58"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678"
+dependencies = [
+ "cc",
+]
+
[[package]]
name = "codespan-reporting"
version = "0.13.1"
@@ -966,6 +1084,7 @@ dependencies = [
name = "concat"
version = "0.2.0"
dependencies = [
+ "concat-media",
"slint",
"slint-build",
]
@@ -997,12 +1116,26 @@ dependencies = [
]
[[package]]
-name = "concat-media"
+name = "concat-host"
version = "0.2.0"
dependencies = [
"concat-core",
- "rusty_ffmpeg",
+ "concat-export",
+ "concat-media",
+ "concat-project",
+ "cpal",
+ "memmap2",
+ "serde",
"serde_json",
+]
+
+[[package]]
+name = "concat-media"
+version = "0.2.0"
+dependencies = [
+ "concat-core",
+ "ffmpeg-the-third",
+ "libc",
"thiserror 2.0.20",
]
@@ -1025,6 +1158,21 @@ dependencies = [
"wgpu 30.0.1",
]
+[[package]]
+name = "concat-speech"
+version = "0.2.0"
+dependencies = [
+ "bzip2 0.6.1",
+ "concat-host",
+ "concat-media",
+ "serde",
+ "serde_json",
+ "sherpa-onnx",
+ "tar",
+ "ureq",
+ "whisper-rs",
+]
+
[[package]]
name = "concurrent-queue"
version = "2.5.0"
@@ -1055,6 +1203,12 @@ dependencies = [
"syn 3.0.4",
]
+[[package]]
+name = "constant_time_eq"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7c74b8349d32d297c9134b8c88677813a227df8f779daa29bfc29c183fe3dca6"
+
[[package]]
name = "convert_case"
version = "0.10.0"
@@ -1127,12 +1281,76 @@ dependencies = [
"libm",
]
+[[package]]
+name = "coreaudio-rs"
+version = "0.13.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1aae284fbaf7d27aa0e292f7677dfbe26503b0d555026f702940805a630eac17"
+dependencies = [
+ "bitflags 1.3.2",
+ "libc",
+ "objc2-audio-toolbox",
+ "objc2-core-audio",
+ "objc2-core-audio-types",
+ "objc2-core-foundation",
+]
+
[[package]]
name = "countme"
version = "3.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7704b5fdd17b18ae31c4c1da5a2e0305a2bf17b5249300a9ee9ed7b72114c636"
+[[package]]
+name = "cpal"
+version = "0.16.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cbd307f43cc2a697e2d1f8bc7a1d824b5269e052209e28883e5bc04d095aaa3f"
+dependencies = [
+ "alsa",
+ "coreaudio-rs",
+ "dasp_sample",
+ "jni 0.21.1",
+ "js-sys",
+ "libc",
+ "mach2",
+ "ndk",
+ "ndk-context",
+ "num-derive",
+ "num-traits",
+ "objc2-audio-toolbox",
+ "objc2-core-audio",
+ "objc2-core-audio-types",
+ "wasm-bindgen",
+ "wasm-bindgen-futures",
+ "web-sys",
+ "windows 0.54.0",
+]
+
+[[package]]
+name = "cpufeatures"
+version = "0.2.17"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
+dependencies = [
+ "libc",
+]
+
+[[package]]
+name = "crc"
+version = "3.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d"
+dependencies = [
+ "crc-catalog",
+]
+
+[[package]]
+name = "crc-catalog"
+version = "2.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853"
+
[[package]]
name = "crc32fast"
version = "1.5.1"
@@ -1188,6 +1406,16 @@ version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
+[[package]]
+name = "crypto-common"
+version = "0.1.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
+dependencies = [
+ "generic-array",
+ "typenum",
+]
+
[[package]]
name = "ctor"
version = "0.10.1"
@@ -1203,12 +1431,41 @@ version = "1.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f27ae1dd37df86211c42e150270f82743308803d90a6f6e6651cd730d5e1732f"
+[[package]]
+name = "dasp_sample"
+version = "0.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0c87e182de0887fd5361989c677c4e8f5000cd9491d6d563161a8f3a5519fc7f"
+
[[package]]
name = "data-url"
version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be1e0bca6c3637f992fc1cc7cbc52a78c1ef6db076dbf1059c4323d6a2048376"
+[[package]]
+name = "deflate64"
+version = "0.1.12"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ac6b926516df9c60bfa16e107b21086399f8285a44ca9711344b9e553c5146e2"
+
+[[package]]
+name = "deranged"
+version = "0.5.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
+
+[[package]]
+name = "derive_arbitrary"
+version = "1.4.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1e567bd82dcff979e4b03460c307b3cdc9e96fde3d73bed1496d2bc75d9dd62a"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.119",
+]
+
[[package]]
name = "derive_more"
version = "2.1.1"
@@ -1243,6 +1500,17 @@ dependencies = [
"syn 3.0.4",
]
+[[package]]
+name = "digest"
+version = "0.10.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
+dependencies = [
+ "block-buffer",
+ "crypto-common",
+ "subtle",
+]
+
[[package]]
name = "dispatch"
version = "0.2.0"
@@ -1509,6 +1777,31 @@ dependencies = [
"wgpu 29.0.4",
]
+[[package]]
+name = "ffmpeg-sys-the-third"
+version = "6.0.0+ffmpeg-9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8beaa5d31a4dee3c5a93caf6c0dd3e9d49fb9849c96354f8d619ec6b062036f2"
+dependencies = [
+ "bindgen",
+ "cc",
+ "clang",
+ "libc",
+ "pkg-config",
+ "vcpkg",
+]
+
+[[package]]
+name = "ffmpeg-the-third"
+version = "6.0.0+ffmpeg-9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cf29dd4474510215c218b491acf6c06d27c42cf3579abdc20936290a3817ca31"
+dependencies = [
+ "bitflags 2.13.1",
+ "ffmpeg-sys-the-third",
+ "libc",
+]
+
[[package]]
name = "field-offset"
version = "0.3.6"
@@ -1628,8 +1921,8 @@ dependencies = [
"read-fonts 0.39.2",
"roxmltree",
"smallvec",
- "windows",
- "windows-core",
+ "windows 0.62.2",
+ "windows-core 0.62.2",
"yeslogic-fontconfig-sys",
]
@@ -1669,6 +1962,12 @@ dependencies = [
"percent-encoding",
]
+[[package]]
+name = "fs_extra"
+version = "1.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c"
+
[[package]]
name = "futures"
version = "0.3.34"
@@ -1792,6 +2091,16 @@ dependencies = [
"libc",
]
+[[package]]
+name = "generic-array"
+version = "0.14.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
+dependencies = [
+ "typenum",
+ "version_check",
+]
+
[[package]]
name = "gethostname"
version = "1.1.0"
@@ -1811,6 +2120,17 @@ dependencies = [
"unicode-width",
]
+[[package]]
+name = "getrandom"
+version = "0.2.17"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
+dependencies = [
+ "cfg-if",
+ "libc",
+ "wasi",
+]
+
[[package]]
name = "getrandom"
version = "0.3.4"
@@ -1818,9 +2138,11 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
dependencies = [
"cfg-if",
+ "js-sys",
"libc",
"r-efi 5.3.0",
"wasip2",
+ "wasm-bindgen",
]
[[package]]
@@ -1950,7 +2272,7 @@ dependencies = [
"log",
"presser",
"thiserror 2.0.20",
- "windows",
+ "windows 0.62.2",
]
[[package]]
@@ -2068,6 +2390,15 @@ version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dfa686283ad6dd069f105e5ab091b04c62850d3e4cf5d67debad1933f55023df"
+[[package]]
+name = "hmac"
+version = "0.12.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e"
+dependencies = [
+ "digest",
+]
+
[[package]]
name = "htmlparser"
version = "0.2.1"
@@ -2172,7 +2503,7 @@ dependencies = [
"wasm-bindgen",
"web-sys",
"webbrowser",
- "windows",
+ "windows 0.62.2",
"winit",
"zbus",
]
@@ -2278,7 +2609,7 @@ dependencies = [
"web-sys",
"web-time",
"wgpu 29.0.4",
- "windows",
+ "windows 0.62.2",
]
[[package]]
@@ -2351,8 +2682,8 @@ dependencies = [
"unicode-segmentation",
"vtable",
"wgpu 29.0.4",
- "windows",
- "windows-core",
+ "windows 0.62.2",
+ "windows-core 0.62.2",
"write-fonts",
]
@@ -2388,7 +2719,7 @@ dependencies = [
"js-sys",
"log",
"wasm-bindgen",
- "windows-core",
+ "windows-core 0.62.2",
]
[[package]]
@@ -2647,6 +2978,15 @@ dependencies = [
"hashbrown 0.17.1",
]
+[[package]]
+name = "inout"
+version = "0.1.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01"
+dependencies = [
+ "generic-array",
+]
+
[[package]]
name = "input"
version = "0.10.0"
@@ -2727,6 +3067,22 @@ version = "1.0.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
+[[package]]
+name = "jni"
+version = "0.21.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1a87aa2bb7d2af34197c04845522473242e1aa17c12f4935d5856491a7fb8c97"
+dependencies = [
+ "cesu8",
+ "cfg-if",
+ "combine",
+ "jni-sys 0.3.1",
+ "log",
+ "thiserror 1.0.69",
+ "walkdir",
+ "windows-sys 0.45.0",
+]
+
[[package]]
name = "jni"
version = "0.22.4"
@@ -2876,6 +3232,12 @@ version = "0.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7a79a3332a6609480d7d0c9eab957bca6b455b91bb84e66d19f5ff66294b85b8"
+[[package]]
+name = "libbz2-rs-sys"
+version = "0.2.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "34b357333733e8260735ba5894eb928c02ecc69c78715f01a8019e7fa7f2db4c"
+
[[package]]
name = "libc"
version = "0.2.189"
@@ -3046,6 +3408,36 @@ dependencies = [
"num-traits",
]
+[[package]]
+name = "lzma-rs"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "297e814c836ae64db86b36cf2a557ba54368d03f6afcd7d947c266692f71115e"
+dependencies = [
+ "byteorder",
+ "crc",
+]
+
+[[package]]
+name = "lzma-sys"
+version = "0.1.20"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5fda04ab3764e6cde78b9974eec4f779acaba7c4e84b36eca3cf77c581b85d27"
+dependencies = [
+ "cc",
+ "libc",
+ "pkg-config",
+]
+
+[[package]]
+name = "mach2"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d640282b302c0bb0a2a8e0233ead9035e3bed871f0b7e81fe4a1ec829765db44"
+dependencies = [
+ "libc",
+]
+
[[package]]
name = "maybe-rayon"
version = "0.1.1"
@@ -3307,6 +3699,12 @@ dependencies = [
"num-traits",
]
+[[package]]
+name = "num-conv"
+version = "0.2.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
+
[[package]]
name = "num-derive"
version = "0.4.2"
@@ -3432,6 +3830,21 @@ dependencies = [
"objc2-quartz-core 0.3.2",
]
+[[package]]
+name = "objc2-audio-toolbox"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6948501a91121d6399b79abaa33a8aa4ea7857fe019f341b8c23ad6e81b79b08"
+dependencies = [
+ "bitflags 2.13.1",
+ "libc",
+ "objc2 0.6.4",
+ "objc2-core-audio",
+ "objc2-core-audio-types",
+ "objc2-core-foundation",
+ "objc2-foundation 0.3.2",
+]
+
[[package]]
name = "objc2-cloud-kit"
version = "0.2.2"
@@ -3451,20 +3864,42 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "73ad74d880bb43877038da939b7427bba67e9dd42004a18b809ba7d87cee241c"
dependencies = [
- "bitflags 2.13.1",
+ "bitflags 2.13.1",
+ "objc2 0.6.4",
+ "objc2-foundation 0.3.2",
+]
+
+[[package]]
+name = "objc2-contacts"
+version = "0.2.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a5ff520e9c33812fd374d8deecef01d4a840e7b41862d849513de77e44aa4889"
+dependencies = [
+ "block2 0.5.1",
+ "objc2 0.5.2",
+ "objc2-foundation 0.2.2",
+]
+
+[[package]]
+name = "objc2-core-audio"
+version = "0.3.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e1eebcea8b0dbff5f7c8504f3107c68fc061a3eb44932051c8cf8a68d969c3b2"
+dependencies = [
+ "dispatch2",
"objc2 0.6.4",
- "objc2-foundation 0.3.2",
+ "objc2-core-audio-types",
+ "objc2-core-foundation",
]
[[package]]
-name = "objc2-contacts"
-version = "0.2.2"
+name = "objc2-core-audio-types"
+version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a5ff520e9c33812fd374d8deecef01d4a840e7b41862d849513de77e44aa4889"
+checksum = "5a89f2ec274a0cf4a32642b2991e8b351a404d290da87bb6a9a9d8632490bd1c"
dependencies = [
- "block2 0.5.1",
- "objc2 0.5.2",
- "objc2-foundation 0.2.2",
+ "bitflags 2.13.1",
+ "objc2 0.6.4",
]
[[package]]
@@ -3881,6 +4316,16 @@ version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4"
+[[package]]
+name = "pbkdf2"
+version = "0.12.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2"
+dependencies = [
+ "digest",
+ "hmac",
+]
+
[[package]]
name = "percent-encoding"
version = "2.3.2"
@@ -4068,6 +4513,12 @@ dependencies = [
"zerovec",
]
+[[package]]
+name = "powerfmt"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"
+
[[package]]
name = "ppv-lite86"
version = "0.2.21"
@@ -4470,6 +4921,20 @@ dependencies = [
"bytemuck",
]
+[[package]]
+name = "ring"
+version = "0.17.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
+dependencies = [
+ "cc",
+ "cfg-if",
+ "getrandom 0.2.17",
+ "libc",
+ "untrusted",
+ "windows-sys 0.52.0",
+]
+
[[package]]
name = "rowan"
version = "0.16.1"
@@ -4552,23 +5017,46 @@ dependencies = [
]
[[package]]
-name = "rustversion"
-version = "1.0.23"
+name = "rustls"
+version = "0.23.43"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
+checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06"
+dependencies = [
+ "log",
+ "once_cell",
+ "ring",
+ "rustls-pki-types",
+ "rustls-webpki",
+ "subtle",
+ "zeroize",
+]
[[package]]
-name = "rusty_ffmpeg"
-version = "0.17.0+ffmpeg.8.1"
+name = "rustls-pki-types"
+version = "1.15.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "f2af9fcc780549d620ddedd4d9f26d62c523f221c0db882ea75e8a5ffe2fff9a"
+checksum = "2f4925028c7eb5d1fcdaf196971378ed9d2c1c4efc7dc5d011256f76c99c0a96"
dependencies = [
- "bindgen",
- "camino",
- "once_cell",
- "pkg-config",
+ "zeroize",
+]
+
+[[package]]
+name = "rustls-webpki"
+version = "0.103.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2"
+dependencies = [
+ "ring",
+ "rustls-pki-types",
+ "untrusted",
]
+[[package]]
+name = "rustversion"
+version = "1.0.23"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
+
[[package]]
name = "rustybuzz"
version = "0.20.1"
@@ -4696,6 +5184,40 @@ dependencies = [
"serde_core",
]
+[[package]]
+name = "sha1"
+version = "0.10.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8"
+dependencies = [
+ "cfg-if",
+ "cpufeatures",
+ "digest",
+]
+
+[[package]]
+name = "sherpa-onnx"
+version = "1.13.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3b4ef11f6b23c916a8e36c3cf65201f796d0e6aa30182a18d4f798d0b7f62547"
+dependencies = [
+ "serde",
+ "serde_json",
+ "sherpa-onnx-sys",
+]
+
+[[package]]
+name = "sherpa-onnx-sys"
+version = "1.13.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "566aad07d0924a7ed897cc035cc67ad71cdfddb959717dbf445031d5d4b58d4d"
+dependencies = [
+ "bzip2 0.4.4",
+ "tar",
+ "ureq",
+ "zip",
+]
+
[[package]]
name = "shlex"
version = "1.3.0"
@@ -4789,7 +5311,7 @@ checksum = "9f512ac418a64194842dd05566320805dad1c957c521039db2486fd6368865bc"
dependencies = [
"bitflags 2.13.1",
"skia-bindings",
- "windows",
+ "windows 0.62.2",
]
[[package]]
@@ -5082,6 +5604,12 @@ dependencies = [
"syn 2.0.119",
]
+[[package]]
+name = "subtle"
+version = "2.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
+
[[package]]
name = "svgtypes"
version = "0.16.1"
@@ -5262,6 +5790,25 @@ dependencies = [
"zune-jpeg",
]
+[[package]]
+name = "time"
+version = "0.3.55"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134"
+dependencies = [
+ "deranged",
+ "num-conv",
+ "powerfmt",
+ "serde_core",
+ "time-core",
+]
+
+[[package]]
+name = "time-core"
+version = "0.1.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109"
+
[[package]]
name = "tiny-skia"
version = "0.11.4"
@@ -5477,6 +6024,12 @@ dependencies = [
"serde",
]
+[[package]]
+name = "typenum"
+version = "1.20.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
+
[[package]]
name = "udev"
version = "0.9.3"
@@ -5572,12 +6125,34 @@ version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853"
+[[package]]
+name = "untrusted"
+version = "0.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
+
[[package]]
name = "unty"
version = "0.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6d49784317cd0d1ee7ec5c716dd598ec5b4483ea832a2dced265471cc0f690ae"
+[[package]]
+name = "ureq"
+version = "2.12.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "02d1a66277ed75f640d608235660df48c8e3c19f3b4edb6a263315626cc3c01d"
+dependencies = [
+ "base64",
+ "flate2",
+ "log",
+ "once_cell",
+ "rustls",
+ "rustls-pki-types",
+ "url",
+ "webpki-roots 0.26.11",
+]
+
[[package]]
name = "url"
version = "2.5.8"
@@ -5652,6 +6227,12 @@ dependencies = [
"wasm-bindgen",
]
+[[package]]
+name = "vcpkg"
+version = "0.2.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426"
+
[[package]]
name = "version_check"
version = "0.9.5"
@@ -5691,6 +6272,12 @@ dependencies = [
"winapi-util",
]
+[[package]]
+name = "wasi"
+version = "0.11.1+wasi-snapshot-preview1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
+
[[package]]
name = "wasip2"
version = "1.0.4+wasi-0.2.12"
@@ -5916,7 +6503,7 @@ version = "1.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "62c35be770821a214dbc362fc26908c853e776c0004294d0b10b8a6bad582f94"
dependencies = [
- "jni",
+ "jni 0.22.4",
"log",
"ndk-context",
"objc2 0.6.4",
@@ -5926,6 +6513,24 @@ dependencies = [
"web-sys",
]
+[[package]]
+name = "webpki-roots"
+version = "0.26.11"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"
+dependencies = [
+ "webpki-roots 1.0.9",
+]
+
+[[package]]
+name = "webpki-roots"
+version = "1.0.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "7dcd9d09a39985f5344844e66b0c530a33843579125f23e21e9f0f220850f22a"
+dependencies = [
+ "rustls-pki-types",
+]
+
[[package]]
name = "weezl"
version = "0.1.12"
@@ -6162,9 +6767,9 @@ dependencies = [
"web-sys",
"wgpu-naga-bridge 29.0.4",
"wgpu-types 29.0.4",
- "windows",
- "windows-core",
- "windows-result",
+ "windows 0.62.2",
+ "windows-core 0.62.2",
+ "windows-result 0.4.1",
]
[[package]]
@@ -6218,9 +6823,9 @@ dependencies = [
"web-sys",
"wgpu-naga-bridge 30.0.1",
"wgpu-types 30.0.1",
- "windows",
- "windows-core",
- "windows-result",
+ "windows 0.62.2",
+ "windows-core 0.62.2",
+ "windows-result 0.4.1",
]
[[package]]
@@ -6273,6 +6878,28 @@ dependencies = [
"web-sys",
]
+[[package]]
+name = "whisper-rs"
+version = "0.16.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "2088172d00f936c348d6a72f488dc2660ab3f507263a195df308a3c2383229f6"
+dependencies = [
+ "whisper-rs-sys",
+]
+
+[[package]]
+name = "whisper-rs-sys"
+version = "0.15.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6986c0fe081241d391f09b9a071fbcbb59720c3563628c3c829057cf69f2a56f"
+dependencies = [
+ "bindgen",
+ "cfg-if",
+ "cmake",
+ "fs_extra",
+ "semver",
+]
+
[[package]]
name = "winapi-util"
version = "0.1.11"
@@ -6282,6 +6909,16 @@ dependencies = [
"windows-sys 0.61.2",
]
+[[package]]
+name = "windows"
+version = "0.54.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9252e5725dbed82865af151df558e754e4a3c2c30818359eb17465f1346a1b49"
+dependencies = [
+ "windows-core 0.54.0",
+ "windows-targets 0.52.6",
+]
+
[[package]]
name = "windows"
version = "0.62.2"
@@ -6289,7 +6926,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "527fadee13e0c05939a6a05d5bd6eec6cd2e3dbd648b9f8e447c6518133d8580"
dependencies = [
"windows-collections",
- "windows-core",
+ "windows-core 0.62.2",
"windows-future",
"windows-numerics",
]
@@ -6300,7 +6937,17 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "23b2d95af1a8a14a3c7367e1ed4fc9c20e0a26e79551b1454d72583c97cc6610"
dependencies = [
- "windows-core",
+ "windows-core 0.62.2",
+]
+
+[[package]]
+name = "windows-core"
+version = "0.54.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "12661b9c89351d684a50a8a643ce5f608e20243b9fb84687800163429f161d65"
+dependencies = [
+ "windows-result 0.1.2",
+ "windows-targets 0.52.6",
]
[[package]]
@@ -6312,7 +6959,7 @@ dependencies = [
"windows-implement",
"windows-interface",
"windows-link",
- "windows-result",
+ "windows-result 0.4.1",
"windows-strings",
]
@@ -6322,7 +6969,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e1d6f90251fe18a279739e78025bd6ddc52a7e22f921070ccdc67dde84c605cb"
dependencies = [
- "windows-core",
+ "windows-core 0.62.2",
"windows-link",
"windows-threading",
]
@@ -6361,10 +7008,19 @@ version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e2e40844ac143cdb44aead537bbf727de9b044e107a0f1220392177d15b0f26"
dependencies = [
- "windows-core",
+ "windows-core 0.62.2",
"windows-link",
]
+[[package]]
+name = "windows-result"
+version = "0.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5e383302e8ec8515204254685643de10811af0ed97ea37210dc26fb0032647f8"
+dependencies = [
+ "windows-targets 0.52.6",
+]
+
[[package]]
name = "windows-result"
version = "0.4.1"
@@ -6383,6 +7039,15 @@ dependencies = [
"windows-link",
]
+[[package]]
+name = "windows-sys"
+version = "0.45.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "75283be5efb2831d37ea142365f009c02ec203cd29a3ebecbc093d52315b66d0"
+dependencies = [
+ "windows-targets 0.42.2",
+]
+
[[package]]
name = "windows-sys"
version = "0.48.0"
@@ -6419,6 +7084,21 @@ dependencies = [
"windows-link",
]
+[[package]]
+name = "windows-targets"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8e5180c00cd44c9b1c88adb3693291f1cd93605ded80c250a75d472756b4d071"
+dependencies = [
+ "windows_aarch64_gnullvm 0.42.2",
+ "windows_aarch64_msvc 0.42.2",
+ "windows_i686_gnu 0.42.2",
+ "windows_i686_msvc 0.42.2",
+ "windows_x86_64_gnu 0.42.2",
+ "windows_x86_64_gnullvm 0.42.2",
+ "windows_x86_64_msvc 0.42.2",
+]
+
[[package]]
name = "windows-targets"
version = "0.48.5"
@@ -6459,6 +7139,12 @@ dependencies = [
"windows-link",
]
+[[package]]
+name = "windows_aarch64_gnullvm"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "597a5118570b68bc08d8d59125332c54f1ba9d9adeedeef5b99b02ba2b0698f8"
+
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.48.5"
@@ -6471,6 +7157,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
+[[package]]
+name = "windows_aarch64_msvc"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e08e8864a60f06ef0d0ff4ba04124db8b0fb3be5776a5cd47641e942e58c4d43"
+
[[package]]
name = "windows_aarch64_msvc"
version = "0.48.5"
@@ -6483,6 +7175,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
+[[package]]
+name = "windows_i686_gnu"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c61d927d8da41da96a81f029489353e68739737d3beca43145c8afec9a31a84f"
+
[[package]]
name = "windows_i686_gnu"
version = "0.48.5"
@@ -6501,6 +7199,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
+[[package]]
+name = "windows_i686_msvc"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "44d840b6ec649f480a41c8d80f9c65108b92d89345dd94027bfe06ac444d1060"
+
[[package]]
name = "windows_i686_msvc"
version = "0.48.5"
@@ -6513,6 +7217,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
+[[package]]
+name = "windows_x86_64_gnu"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8de912b8b8feb55c064867cf047dda097f92d51efad5b491dfb98f6bbb70cb36"
+
[[package]]
name = "windows_x86_64_gnu"
version = "0.48.5"
@@ -6525,6 +7235,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
+[[package]]
+name = "windows_x86_64_gnullvm"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "26d41b46a36d453748aedef1486d5c7a85db22e56aff34643984ea85514e94a3"
+
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.48.5"
@@ -6537,6 +7253,12 @@ version = "0.52.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
+[[package]]
+name = "windows_x86_64_msvc"
+version = "0.42.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
+
[[package]]
name = "windows_x86_64_msvc"
version = "0.48.5"
@@ -6735,6 +7457,15 @@ version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec7a2a501ed189703dba8b08142f057e887dfc4b2cc4db2d343ac6376ba3e0b9"
+[[package]]
+name = "xz2"
+version = "0.1.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "388c44dc09d76f1536602ead6d325eb532f5c122f17782bd57fb47baeeb767e2"
+dependencies = [
+ "lzma-sys",
+]
+
[[package]]
name = "y4m"
version = "0.8.0"
@@ -6934,6 +7665,26 @@ dependencies = [
"synstructure",
]
+[[package]]
+name = "zeroize"
+version = "1.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
+dependencies = [
+ "zeroize_derive",
+]
+
+[[package]]
+name = "zeroize_derive"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3c50655cbb0fe3fc43170059e702f1ce5e19b84cec58dc87b037a09935c2f328"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.119",
+]
+
[[package]]
name = "zerotrie"
version = "0.2.5"
@@ -6969,6 +7720,36 @@ dependencies = [
"syn 3.0.4",
]
+[[package]]
+name = "zip"
+version = "2.4.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "fabe6324e908f85a1c52063ce7aa26b68dcb7eb6dbc83a2d148403c9bc3eba50"
+dependencies = [
+ "aes",
+ "arbitrary",
+ "bzip2 0.5.2",
+ "constant_time_eq",
+ "crc32fast",
+ "crossbeam-utils",
+ "deflate64",
+ "displaydoc",
+ "flate2",
+ "getrandom 0.3.4",
+ "hmac",
+ "indexmap",
+ "lzma-rs",
+ "memchr",
+ "pbkdf2",
+ "sha1",
+ "thiserror 2.0.20",
+ "time",
+ "xz2",
+ "zeroize",
+ "zopfli",
+ "zstd",
+]
+
[[package]]
name = "zlib-rs"
version = "0.6.7"
@@ -6981,6 +7762,46 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
+[[package]]
+name = "zopfli"
+version = "0.8.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249"
+dependencies = [
+ "bumpalo",
+ "crc32fast",
+ "log",
+ "simd-adler32",
+]
+
+[[package]]
+name = "zstd"
+version = "0.13.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a"
+dependencies = [
+ "zstd-safe",
+]
+
+[[package]]
+name = "zstd-safe"
+version = "7.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d"
+dependencies = [
+ "zstd-sys",
+]
+
+[[package]]
+name = "zstd-sys"
+version = "2.0.16+zstd.1.5.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748"
+dependencies = [
+ "cc",
+ "pkg-config",
+]
+
[[package]]
name = "zune-core"
version = "0.5.3"
diff --git a/engine/Cargo.toml b/engine/Cargo.toml
index f37a33d6..bfeed58a 100644
--- a/engine/Cargo.toml
+++ b/engine/Cargo.toml
@@ -11,8 +11,11 @@ license = "AGPL-3.0-or-later" # + plugin exception, see LICENSE-EXCEPTIONS.md
[workspace.dependencies]
concat-core = { path = "crates/concat-core" }
concat-media = { path = "crates/concat-media" }
+concat-project = { path = "crates/concat-project" }
concat-render = { path = "crates/concat-render" }
concat-export = { path = "crates/concat-export" }
+concat-host = { path = "crates/concat-host" }
+concat-speech = { path = "crates/concat-speech" }
serde_json = "1"
thiserror = "2"
diff --git a/engine/README.md b/engine/README.md
index c205a4d7..947aea60 100644
--- a/engine/README.md
+++ b/engine/README.md
@@ -7,10 +7,12 @@ The video engine behind Concat. Rust, no GC, no hidden control flow.
| Crate | What it is | Dependencies |
|---|---|---|
| `concat-core` | Time, arena, frames, timeline model. The vocabulary every other crate speaks. | **none** (std only) |
-| `concat-media` | Getting pixels in and out of files. FFmpeg lives here and nowhere else. | `concat-core`, serde_json |
+| `concat-media` | Getting pixels and samples in and out of files. Links FFmpeg (libav*), and is the only crate that knows it exists. | `concat-core`, ffmpeg-the-third |
| `concat-project` | The edit itself: document model, operations as commands, undo, concat.json IO. | serde, serde_json |
| `concat-render` | Turning a timeline plus a timestamp into one finished frame. | `concat-core` |
| `concat-cli` | A binary to drive the above. The vertical slice. | all of them |
+| `concat-host` | What the window needs that is not the edit: sessions, project folders, previews, playback, templates, job slots. | `concat-media`, `concat-project`, `concat-export`, cpal |
+| `concat-speech` | Transcription (whisper.cpp, in-process) and text to speech (Kokoro via sherpa-onnx). | `concat-host`, `concat-media`, whisper-rs, sherpa-onnx |
| `concat` | The editor window: every pane, dialog and primitive, in Slint. The app a user launches. | slint (engine wiring is next) |
The dependency arrows point one way: `{concat, cli} -> {media, render} -> core`.
@@ -34,7 +36,12 @@ cargo run -p concat # the editor window, debug
cargo build --profile app -p concat # the shipping binary: fat LTO, panic=abort, stripped
```
-Requires `ffmpeg` and `ffprobe` on `PATH`. The window builds with Slint's
+Nothing is spawned at run time: FFmpeg is linked, whisper.cpp and sherpa-onnx
+are compiled in. A build needs the FFmpeg 7+ development libraries (headers
+and import libraries) - `brew install ffmpeg` on macOS, a BtbN `shared` build
+unpacked and pointed at with `FFMPEG_DIR` on Windows and on Linux
+distributions whose packaged FFmpeg is older than 7 - plus cmake and a C++
+toolchain for whisper.cpp. The window builds with Slint's
Skia renderer by default; `--no-default-features --features wgpu` swaps in
FemtoVG over wgpu, and the two are meant to be compared, not chosen once.
On Linux, Skia needs the fontconfig and freetype headers at build time (see
diff --git a/engine/crates/concat-cli/src/main.rs b/engine/crates/concat-cli/src/main.rs
index 3400e01d..06952aac 100644
--- a/engine/crates/concat-cli/src/main.rs
+++ b/engine/crates/concat-cli/src/main.rs
@@ -13,7 +13,7 @@ use std::path::PathBuf;
use clap::{Parser, Subcommand};
use concat_core::time::{FrameRate, Rational};
use concat_core::timeline::{Clip, MediaRef, Timeline, Track, TrackKind};
-use concat_media::{EncodeOptions, FfmpegEncoder, FrameSink, ReaderPool};
+use concat_media::{EncodeOptions, Encoder, FrameSink, ReaderPool};
use concat_render::{Compositor, CpuCompositor, Layer, plan_frame};
#[derive(Parser)]
@@ -49,9 +49,12 @@ enum Command {
fn main() -> Result<(), Box<dyn Error>> {
match Cli::parse().command {
Command::Probe { path } => probe(&path),
- Command::Render { input, output, frames, fade } => {
- render(&input, &output, frames, fade)
- }
+ Command::Render {
+ input,
+ output,
+ frames,
+ fade,
+ } => render(&input, &output, frames, fade),
}
}
@@ -95,12 +98,7 @@ fn probe(path: &PathBuf) -> Result<(), Box<dyn Error>> {
/// not just one clip played start to finish. The shortcut this function
/// carried for its whole early life is gone; the pool it was waiting for
/// exists (`concat_media::pool`).
-fn render(
- input: &PathBuf,
- output: &PathBuf,
- frames: u64,
- fade: u64,
-) -> Result<(), Box<dyn Error>> {
+fn render(input: &PathBuf, output: &PathBuf, frames: u64, fade: u64) -> Result<(), Box<dyn Error>> {
let info = concat_media::probe(input)?;
let video = info.require_video()?;
let (width, height) = (video.width, video.height);
@@ -109,8 +107,7 @@ fn render(
let timeline = single_clip_timeline(input, width, height, rate, frames);
let mut pool = ReaderPool::with_defaults();
- let mut encoder =
- FfmpegEncoder::create(output, width, height, rate, &EncodeOptions::default())?;
+ let mut encoder = Encoder::create(output, width, height, rate, &EncodeOptions::default())?;
let mut compositor = CpuCompositor;
println!("rendering {frames} frames at {width}x{height} {rate}");
@@ -156,7 +153,10 @@ fn single_clip_timeline(
let track = timeline.add_track(Track::new("V1", TrackKind::Video));
let duration = rate.time_of_frame(frames as i64);
timeline
- .add_clip(track, Clip::new(MediaRef::new(input), Rational::ZERO, duration))
+ .add_clip(
+ track,
+ Clip::new(MediaRef::new(input), Rational::ZERO, duration),
+ )
.expect("the track was just added");
timeline
}
@@ -189,13 +189,8 @@ mod tests {
#[test]
fn the_timeline_covers_exactly_the_requested_frames() {
- let timeline = single_clip_timeline(
- &PathBuf::from("a.mp4"),
- 1920,
- 1080,
- FrameRate::NTSC_30,
- 90,
- );
+ let timeline =
+ single_clip_timeline(&PathBuf::from("a.mp4"), 1920, 1080, FrameRate::NTSC_30, 90);
assert_eq!(timeline.frame_count(), 90);
assert_eq!(timeline.clip_count(), 1);
}
diff --git a/engine/crates/concat-core/src/arena.rs b/engine/crates/concat-core/src/arena.rs
index 9f7bafe9..405df887 100644
--- a/engine/crates/concat-core/src/arena.rs
+++ b/engine/crates/concat-core/src/arena.rs
@@ -41,7 +41,11 @@ pub struct Id<T> {
impl<T> Id<T> {
fn new(index: u32, generation: u32) -> Self {
- Self { index, generation, marker: PhantomData }
+ Self {
+ index,
+ generation,
+ marker: PhantomData,
+ }
}
/// The slot this handle points at. Useful for stable display ordering; it
@@ -76,7 +80,10 @@ impl<T> Hash for Id<T> {
impl<T> fmt::Debug for Id<T> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
- let name = std::any::type_name::<T>().rsplit("::").next().unwrap_or("?");
+ let name = std::any::type_name::<T>()
+ .rsplit("::")
+ .next()
+ .unwrap_or("?");
write!(f, "{name}#{}v{}", self.index, self.generation)
}
}
@@ -97,12 +104,20 @@ pub struct Arena<T> {
impl<T> Arena<T> {
/// An empty arena.
pub fn new() -> Self {
- Self { slots: Vec::new(), free: Vec::new(), len: 0 }
+ Self {
+ slots: Vec::new(),
+ free: Vec::new(),
+ len: 0,
+ }
}
/// An empty arena with room for `capacity` values.
pub fn with_capacity(capacity: usize) -> Self {
- Self { slots: Vec::with_capacity(capacity), free: Vec::new(), len: 0 }
+ Self {
+ slots: Vec::with_capacity(capacity),
+ free: Vec::new(),
+ len: 0,
+ }
}
/// Stores `value` and returns a handle to it.
@@ -121,7 +136,10 @@ impl<T> Arena<T> {
Id::new(index, generation)
} else {
let index = u32::try_from(self.slots.len()).expect("arena exceeded u32::MAX slots");
- self.slots.push(Slot::Occupied { value, generation: 0 });
+ self.slots.push(Slot::Occupied {
+ value,
+ generation: 0,
+ });
self.len += 1;
Id::new(index, 0)
}
@@ -157,7 +175,12 @@ impl<T> Arena<T> {
_ => return None,
}
let next_generation = id.generation.wrapping_add(1);
- let previous = std::mem::replace(slot, Slot::Vacant { generation: next_generation });
+ let previous = std::mem::replace(
+ slot,
+ Slot::Vacant {
+ generation: next_generation,
+ },
+ );
self.free.push(id.index);
self.len -= 1;
match previous {
@@ -178,22 +201,28 @@ impl<T> Arena<T> {
/// Iterates over live values in slot order, paired with their handles.
pub fn iter(&self) -> impl Iterator<Item = (Id<T>, &T)> {
- self.slots.iter().enumerate().filter_map(|(index, slot)| match slot {
- Slot::Occupied { value, generation } => {
- Some((Id::new(index as u32, *generation), value))
- }
- Slot::Vacant { .. } => None,
- })
+ self.slots
+ .iter()
+ .enumerate()
+ .filter_map(|(index, slot)| match slot {
+ Slot::Occupied { value, generation } => {
+ Some((Id::new(index as u32, *generation), value))
+ }
+ Slot::Vacant { .. } => None,
+ })
}
/// Mutably iterates over live values in slot order, paired with handles.
pub fn iter_mut(&mut self) -> impl Iterator<Item = (Id<T>, &mut T)> {
- self.slots.iter_mut().enumerate().filter_map(|(index, slot)| match slot {
- Slot::Occupied { value, generation } => {
- Some((Id::new(index as u32, *generation), value))
- }
- Slot::Vacant { .. } => None,
- })
+ self.slots
+ .iter_mut()
+ .enumerate()
+ .filter_map(|(index, slot)| match slot {
+ Slot::Occupied { value, generation } => {
+ Some((Id::new(index as u32, *generation), value))
+ }
+ Slot::Vacant { .. } => None,
+ })
}
}
diff --git a/engine/crates/concat-core/src/frame.rs b/engine/crates/concat-core/src/frame.rs
index 17a2b9bf..0324149e 100644
--- a/engine/crates/concat-core/src/frame.rs
+++ b/engine/crates/concat-core/src/frame.rs
@@ -41,12 +41,20 @@ impl Frame {
for pixel in pixels.chunks_exact_mut(BYTES_PER_PIXEL) {
pixel[3] = 255;
}
- Self { width, height, pixels }
+ Self {
+ width,
+ height,
+ pixels,
+ }
}
/// A fully transparent frame.
pub fn transparent(width: u32, height: u32) -> Self {
- Self { width, height, pixels: vec![0u8; Self::byte_len(width, height)] }
+ Self {
+ width,
+ height,
+ pixels: vec![0u8; Self::byte_len(width, height)],
+ }
}
/// Wraps an existing buffer.
@@ -54,7 +62,11 @@ impl Frame {
/// Returns `None` if the buffer is not exactly `width * height * 4` bytes.
/// Decoders should use this rather than trusting their own arithmetic.
pub fn from_rgba(width: u32, height: u32, pixels: Vec<u8>) -> Option<Self> {
- (pixels.len() == Self::byte_len(width, height)).then_some(Self { width, height, pixels })
+ (pixels.len() == Self::byte_len(width, height)).then_some(Self {
+ width,
+ height,
+ pixels,
+ })
}
/// Width in pixels.
@@ -114,6 +126,27 @@ impl Frame {
}
}
+ /// Copies `source` into this frame with its top-left corner at (`x`, `y`),
+ /// clipping whatever falls outside. No blending: the source's pixels
+ /// replace what was there.
+ pub fn blit(&mut self, source: &Frame, x: u32, y: u32) {
+ let width = self.width;
+ let height = self.height;
+ let columns = source.width.min(width.saturating_sub(x)) as usize;
+ let rows = source.height.min(height.saturating_sub(y)) as usize;
+ if columns == 0 || rows == 0 {
+ return;
+ }
+ let source_row = source.width as usize * 4;
+ let target_row = width as usize * 4;
+ for row in 0..rows {
+ let from = row * source_row;
+ let to = (y as usize + row) * target_row + x as usize * 4;
+ self.pixels[to..to + columns * 4]
+ .copy_from_slice(&source.pixels[from..from + columns * 4]);
+ }
+ }
+
/// Fills the whole frame with one colour.
pub fn fill(&mut self, rgba: [u8; 4]) {
for pixel in self.pixels.chunks_exact_mut(BYTES_PER_PIXEL) {
@@ -123,7 +156,8 @@ impl Frame {
/// Iterates over rows, top to bottom.
pub fn rows(&self) -> impl Iterator<Item = &[u8]> {
- self.pixels.chunks_exact(self.width as usize * BYTES_PER_PIXEL)
+ self.pixels
+ .chunks_exact(self.width as usize * BYTES_PER_PIXEL)
}
}
diff --git a/engine/crates/concat-core/src/time.rs b/engine/crates/concat-core/src/time.rs
index 5670580f..068aab2a 100644
--- a/engine/crates/concat-core/src/time.rs
+++ b/engine/crates/concat-core/src/time.rs
@@ -215,7 +215,8 @@ impl PartialOrd for Rational {
impl Add for Rational {
type Output = Self;
fn add(self, rhs: Self) -> Self {
- let num = i128::from(self.num) * i128::from(rhs.den) + i128::from(rhs.num) * i128::from(self.den);
+ let num =
+ i128::from(self.num) * i128::from(rhs.den) + i128::from(rhs.num) * i128::from(self.den);
Self::reduce(num, i128::from(self.den) * i128::from(rhs.den))
}
}
@@ -251,7 +252,10 @@ impl Div for Rational {
impl Neg for Rational {
type Output = Self;
fn neg(self) -> Self {
- Self { num: -self.num, den: self.den }
+ Self {
+ num: -self.num,
+ den: self.den,
+ }
}
}
@@ -298,17 +302,26 @@ pub struct FrameRate(Rational);
impl FrameRate {
/// 23.976 fps (24000/1001).
- pub const FILM_NTSC: Self = Self(Rational { num: 24000, den: 1001 });
+ pub const FILM_NTSC: Self = Self(Rational {
+ num: 24000,
+ den: 1001,
+ });
/// 24 fps.
pub const FILM: Self = Self(Rational { num: 24, den: 1 });
/// 25 fps.
pub const PAL: Self = Self(Rational { num: 25, den: 1 });
/// 29.97 fps (30000/1001).
- pub const NTSC_30: Self = Self(Rational { num: 30000, den: 1001 });
+ pub const NTSC_30: Self = Self(Rational {
+ num: 30000,
+ den: 1001,
+ });
/// 30 fps.
pub const THIRTY: Self = Self(Rational { num: 30, den: 1 });
/// 59.94 fps (60000/1001).
- pub const NTSC_60: Self = Self(Rational { num: 60000, den: 1001 });
+ pub const NTSC_60: Self = Self(Rational {
+ num: 60000,
+ den: 1001,
+ });
/// 60 fps.
pub const SIXTY: Self = Self(Rational { num: 60, den: 1 });
@@ -317,7 +330,10 @@ impl FrameRate {
/// # Panics
/// If `fps` is zero or negative.
pub fn new(fps: Rational) -> Self {
- assert!(!fps.is_zero() && !fps.is_negative(), "frame rate must be positive");
+ assert!(
+ !fps.is_zero() && !fps.is_negative(),
+ "frame rate must be positive"
+ );
Self(fps)
}
@@ -376,7 +392,10 @@ impl TimeRange {
/// # Panics
/// If `duration` is negative.
pub fn new(start: Rational, duration: Rational) -> Self {
- assert!(!duration.is_negative(), "TimeRange duration must not be negative");
+ assert!(
+ !duration.is_negative(),
+ "TimeRange duration must not be negative"
+ );
Self { start, duration }
}
@@ -440,7 +459,10 @@ mod tests {
fn frame_boundaries_belong_to_the_frame_that_starts_there() {
let rate = FrameRate::THIRTY;
assert_eq!(rate.frame_at(rate.time_of_frame(7)), 7);
- assert_eq!(rate.frame_at(rate.time_of_frame(7) - Rational::new(1, 1_000_000)), 6);
+ assert_eq!(
+ rate.frame_at(rate.time_of_frame(7) - Rational::new(1, 1_000_000)),
+ 6
+ );
}
#[test]
@@ -453,7 +475,10 @@ mod tests {
#[test]
fn parses_ffmpeg_fractions() {
- assert_eq!(Rational::parse("30000/1001"), Some(FrameRate::NTSC_30.fps()));
+ assert_eq!(
+ Rational::parse("30000/1001"),
+ Some(FrameRate::NTSC_30.fps())
+ );
assert_eq!(Rational::parse("25"), Some(Rational::from_int(25)));
assert_eq!(Rational::parse("1.5"), Some(Rational::new(3, 2)));
assert_eq!(Rational::parse("-1.25"), Some(Rational::new(-5, 4)));
diff --git a/engine/crates/concat-core/src/timeline.rs b/engine/crates/concat-core/src/timeline.rs
index fc15b711..b6b3d471 100644
--- a/engine/crates/concat-core/src/timeline.rs
+++ b/engine/crates/concat-core/src/timeline.rs
@@ -105,8 +105,12 @@ pub struct Transform {
impl Transform {
/// Fitted, centred, unrotated.
- pub const IDENTITY: Transform =
- Transform { scale: 1.0, offset_x: 0.0, offset_y: 0.0, rotation: 0.0 };
+ pub const IDENTITY: Transform = Transform {
+ scale: 1.0,
+ offset_x: 0.0,
+ offset_y: 0.0,
+ rotation: 0.0,
+ };
/// True when applying this transform would change nothing.
pub fn is_identity(&self) -> bool {
@@ -198,7 +202,12 @@ pub struct Track {
impl Track {
/// An empty, enabled track.
pub fn new(name: impl Into<String>, kind: TrackKind) -> Self {
- Self { name: name.into(), kind, enabled: true, clips: Vec::new() }
+ Self {
+ name: name.into(),
+ kind,
+ enabled: true,
+ clips: Vec::new(),
+ }
}
/// The clips on this track, in insertion order.
@@ -263,7 +272,9 @@ impl Timeline {
/// Iterates over tracks bottom-most first.
pub fn tracks(&self) -> impl Iterator<Item = (TrackId, &Track)> {
- self.order.iter().filter_map(|&id| self.tracks.get(id).map(|track| (id, track)))
+ self.order
+ .iter()
+ .filter_map(|&id| self.tracks.get(id).map(|track| (id, track)))
}
/// Removes a track and every clip on it.
@@ -282,7 +293,11 @@ impl Timeline {
// otherwise a stale handle leaks an orphaned clip.
self.tracks.get(track)?;
let id = self.clips.insert(clip);
- self.tracks.get_mut(track).expect("checked above").clips.push(id);
+ self.tracks
+ .get_mut(track)
+ .expect("checked above")
+ .clips
+ .push(id);
Some(id)
}
@@ -326,7 +341,11 @@ impl Timeline {
/// Where the last clip ends. Zero for an empty timeline.
pub fn duration(&self) -> Rational {
- self.clips.iter().map(|(_, clip)| clip.range().end()).max().unwrap_or(Rational::ZERO)
+ self.clips
+ .iter()
+ .map(|(_, clip)| clip.range().end())
+ .max()
+ .unwrap_or(Rational::ZERO)
}
/// How many whole output frames the timeline runs for.
@@ -347,7 +366,10 @@ pub struct Project {
impl Project {
/// A new, empty project at the given output format.
pub fn new(name: impl Into<String>, timeline: Timeline) -> Self {
- Self { name: name.into(), timeline }
+ Self {
+ name: name.into(),
+ timeline,
+ }
}
}
@@ -363,7 +385,10 @@ mod tests {
let mut timeline = Timeline::hd();
let track = timeline.add_track(Track::new("V1", TrackKind::Video));
let clip = timeline
- .add_clip(track, Clip::new(MediaRef::new("a.mp4"), seconds(2), seconds(3)))
+ .add_clip(
+ track,
+ Clip::new(MediaRef::new("a.mp4"), seconds(2), seconds(3)),
+ )
.expect("track exists");
(timeline, track, clip)
}
@@ -376,7 +401,11 @@ mod tests {
assert_eq!(clip.source_time_at(seconds(2)), Some(Rational::ZERO));
assert_eq!(clip.source_time_at(seconds(4)), Some(seconds(2)));
assert_eq!(clip.source_time_at(seconds(1)), None, "before the clip");
- assert_eq!(clip.source_time_at(seconds(5)), None, "the end is exclusive");
+ assert_eq!(
+ clip.source_time_at(seconds(5)),
+ None,
+ "the end is exclusive"
+ );
}
#[test]
@@ -421,9 +450,21 @@ mod tests {
let clip = timeline.clip(id).expect("clip exists");
assert_eq!(clip.video_fade_factor(seconds(2)), 0.0, "starts at zero");
- assert_eq!(clip.video_fade_factor(Rational::new(5, 2)), 0.5, "halfway up");
- assert_eq!(clip.video_fade_factor(seconds(3)), 1.0, "holds at one between fades");
- assert_eq!(clip.video_fade_factor(Rational::new(9, 2)), 0.5, "halfway down");
+ assert_eq!(
+ clip.video_fade_factor(Rational::new(5, 2)),
+ 0.5,
+ "halfway up"
+ );
+ assert_eq!(
+ clip.video_fade_factor(seconds(3)),
+ 1.0,
+ "holds at one between fades"
+ );
+ assert_eq!(
+ clip.video_fade_factor(Rational::new(9, 2)),
+ 0.5,
+ "halfway down"
+ );
}
#[test]
@@ -447,7 +488,10 @@ mod tests {
assert_eq!(timeline.duration(), seconds(5));
timeline
- .add_clip(track, Clip::new(MediaRef::new("b.mp4"), seconds(5), seconds(4)))
+ .add_clip(
+ track,
+ Clip::new(MediaRef::new("b.mp4"), seconds(5), seconds(4)),
+ )
.expect("track exists");
assert_eq!(timeline.duration(), seconds(9));
assert_eq!(timeline.frame_count(), 270); // 9s at 30fps
@@ -477,9 +521,18 @@ mod tests {
let (mut timeline, track, clip) = fixture();
assert!(timeline.remove_clip(clip).is_some());
- assert!(timeline.track(track).expect("track exists").clips().is_empty());
+ assert!(
+ timeline
+ .track(track)
+ .expect("track exists")
+ .clips()
+ .is_empty()
+ );
assert_eq!(timeline.clip_on_track_at(track, seconds(3)), None);
- assert!(timeline.remove_clip(clip).is_none(), "removing twice is harmless");
+ assert!(
+ timeline.remove_clip(clip).is_none(),
+ "removing twice is harmless"
+ );
}
#[test]
@@ -487,7 +540,10 @@ mod tests {
let (mut timeline, track, _) = fixture();
timeline.remove_track(track);
- let orphan = timeline.add_clip(track, Clip::new(MediaRef::new("c.mp4"), seconds(0), seconds(1)));
+ let orphan = timeline.add_clip(
+ track,
+ Clip::new(MediaRef::new("c.mp4"), seconds(0), seconds(1)),
+ );
assert_eq!(orphan, None);
assert_eq!(timeline.clip_count(), 0);
}
diff --git a/engine/crates/concat-export/src/chains.rs b/engine/crates/concat-export/src/chains.rs
index 2fa4cf59..06fe3aef 100644
--- a/engine/crates/concat-export/src/chains.rs
+++ b/engine/crates/concat-export/src/chains.rs
@@ -211,85 +211,181 @@ fn fx_shake(p: &Params, _index: usize) -> String {
/// Every video effect, in the catalogue's order. Ids are forever: they are
/// written into project files.
static EFFECTS: &[Entry] = &[
- Entry { id: "black-white", params: &[], chain: fx_black_white },
- Entry { id: "sepia", params: &[], chain: fx_sepia },
- Entry { id: "invert", params: &[], chain: fx_invert },
+ Entry {
+ id: "black-white",
+ params: &[],
+ chain: fx_black_white,
+ },
+ Entry {
+ id: "sepia",
+ params: &[],
+ chain: fx_sepia,
+ },
+ Entry {
+ id: "invert",
+ params: &[],
+ chain: fx_invert,
+ },
Entry {
id: "sharpen",
- params: &[Param { key: "amount", min: 0.2, max: 3.0, default: 1.0 }],
+ params: &[Param {
+ key: "amount",
+ min: 0.2,
+ max: 3.0,
+ default: 1.0,
+ }],
chain: fx_sharpen,
},
Entry {
id: "gaussian-blur",
- params: &[Param { key: "radius", min: 1.0, max: 50.0, default: 10.0 }],
+ params: &[Param {
+ key: "radius",
+ min: 1.0,
+ max: 50.0,
+ default: 10.0,
+ }],
chain: fx_gaussian_blur,
},
Entry {
id: "box-blur",
- params: &[Param { key: "radius", min: 1.0, max: 30.0, default: 6.0 }],
+ params: &[Param {
+ key: "radius",
+ min: 1.0,
+ max: 30.0,
+ default: 6.0,
+ }],
chain: fx_box_blur,
},
Entry {
id: "motion-blur",
- params: &[Param { key: "length", min: 2.0, max: 60.0, default: 18.0 }],
+ params: &[Param {
+ key: "length",
+ min: 2.0,
+ max: 60.0,
+ default: 18.0,
+ }],
chain: fx_motion_blur,
},
Entry {
id: "warm",
- params: &[Param { key: "temperature", min: 3000.0, max: 6000.0, default: 4600.0 }],
+ params: &[Param {
+ key: "temperature",
+ min: 3000.0,
+ max: 6000.0,
+ default: 4600.0,
+ }],
chain: fx_temperature,
},
Entry {
id: "cool",
- params: &[Param { key: "temperature", min: 7000.0, max: 11000.0, default: 8500.0 }],
+ params: &[Param {
+ key: "temperature",
+ min: 7000.0,
+ max: 11000.0,
+ default: 8500.0,
+ }],
chain: fx_temperature,
},
Entry {
id: "vibrance",
- params: &[Param { key: "intensity", min: 0.1, max: 2.0, default: 0.7 }],
+ params: &[Param {
+ key: "intensity",
+ min: 0.1,
+ max: 2.0,
+ default: 0.7,
+ }],
chain: fx_vibrance,
},
Entry {
id: "contrast-pop",
- params: &[Param { key: "contrast", min: 1.0, max: 2.0, default: 1.25 }],
+ params: &[Param {
+ key: "contrast",
+ min: 1.0,
+ max: 2.0,
+ default: 1.25,
+ }],
chain: fx_contrast_pop,
},
Entry {
id: "vignette",
- params: &[Param { key: "strength", min: 10.0, max: 100.0, default: 50.0 }],
+ params: &[Param {
+ key: "strength",
+ min: 10.0,
+ max: 100.0,
+ default: 50.0,
+ }],
chain: fx_vignette,
},
Entry {
id: "film-grain",
- params: &[Param { key: "amount", min: 2.0, max: 40.0, default: 12.0 }],
+ params: &[Param {
+ key: "amount",
+ min: 2.0,
+ max: 40.0,
+ default: 12.0,
+ }],
chain: fx_film_grain,
},
Entry {
id: "glow",
- params: &[Param { key: "amount", min: 10.0, max: 100.0, default: 45.0 }],
+ params: &[Param {
+ key: "amount",
+ min: 10.0,
+ max: 100.0,
+ default: 45.0,
+ }],
chain: fx_glow,
},
Entry {
id: "posterize",
- params: &[Param { key: "levels", min: 2.0, max: 8.0, default: 4.0 }],
+ params: &[Param {
+ key: "levels",
+ min: 2.0,
+ max: 8.0,
+ default: 4.0,
+ }],
chain: fx_posterize,
},
Entry {
id: "pixelate",
- params: &[Param { key: "size", min: 2.0, max: 64.0, default: 16.0 }],
+ params: &[Param {
+ key: "size",
+ min: 2.0,
+ max: 64.0,
+ default: 16.0,
+ }],
chain: fx_pixelate,
},
- Entry { id: "mirror", params: &[], chain: fx_mirror },
+ Entry {
+ id: "mirror",
+ params: &[],
+ chain: fx_mirror,
+ },
Entry {
id: "fisheye",
- params: &[Param { key: "strength", min: 5.0, max: 100.0, default: 50.0 }],
+ params: &[Param {
+ key: "strength",
+ min: 5.0,
+ max: 100.0,
+ default: 50.0,
+ }],
chain: fx_fisheye,
},
Entry {
id: "shake",
params: &[
- Param { key: "amount", min: 2.0, max: 40.0, default: 12.0 },
- Param { key: "speed", min: 2.0, max: 30.0, default: 13.0 },
+ Param {
+ key: "amount",
+ min: 2.0,
+ max: 40.0,
+ default: 12.0,
+ },
+ Param {
+ key: "speed",
+ min: 2.0,
+ max: 30.0,
+ default: 13.0,
+ },
],
chain: fx_shake,
},
@@ -321,7 +417,11 @@ fn af_sweet(p: &Params, _index: usize) -> String {
// The coefficients are the TS entry's own (see its comment on why they
// are steeper than the reference chain it was ported from).
- let shift = if pitch > 0.0 { pitch } else { 1.2 + 3.8 * strength };
+ let shift = if pitch > 0.0 {
+ pitch
+ } else {
+ 1.2 + 3.8 * strength
+ };
let presence = fixed(0.8 + 5.2 * strength, 3);
let air = fixed(1.0 + 7.0 * strength, 3);
let deess = fixed(0.18 + 0.32 * strength, 3);
@@ -356,7 +456,10 @@ fn af_deep(p: &Params, _index: usize) -> String {
fn af_chipmunk(p: &Params, _index: usize) -> String {
let mut parts = pitch_shift(p.get("pitch"));
- parts.extend(["highpass=f=120".to_owned(), "alimiter=limit=0.94".to_owned()]);
+ parts.extend([
+ "highpass=f=120".to_owned(),
+ "alimiter=limit=0.94".to_owned(),
+ ]);
parts.join(",")
}
@@ -421,57 +524,117 @@ static FILTERS: &[Entry] = &[
Entry {
id: "sweet",
params: &[
- Param { key: "amount", min: 0.0, max: 100.0, default: 65.0 },
+ Param {
+ key: "amount",
+ min: 0.0,
+ max: 100.0,
+ default: 65.0,
+ },
// 0 means "follow amount", which is what the original did when
// no explicit pitch was passed.
- Param { key: "pitch", min: 0.0, max: 8.0, default: 0.0 },
+ Param {
+ key: "pitch",
+ min: 0.0,
+ max: 8.0,
+ default: 0.0,
+ },
],
chain: af_sweet,
},
Entry {
id: "deep",
params: &[
- Param { key: "pitch", min: -8.0, max: -1.0, default: -3.0 },
- Param { key: "body", min: 0.0, max: 8.0, default: 3.0 },
+ Param {
+ key: "pitch",
+ min: -8.0,
+ max: -1.0,
+ default: -3.0,
+ },
+ Param {
+ key: "body",
+ min: 0.0,
+ max: 8.0,
+ default: 3.0,
+ },
],
chain: af_deep,
},
Entry {
id: "chipmunk",
- params: &[Param { key: "pitch", min: 3.0, max: 12.0, default: 7.0 }],
+ params: &[Param {
+ key: "pitch",
+ min: 3.0,
+ max: 12.0,
+ default: 7.0,
+ }],
chain: af_chipmunk,
},
Entry {
id: "robot",
- params: &[Param { key: "depth", min: 1.0, max: 10.0, default: 5.0 }],
+ params: &[Param {
+ key: "depth",
+ min: 1.0,
+ max: 10.0,
+ default: 5.0,
+ }],
chain: af_robot,
},
Entry {
id: "bass",
- params: &[Param { key: "gain", min: 0.0, max: 12.0, default: 5.0 }],
+ params: &[Param {
+ key: "gain",
+ min: 0.0,
+ max: 12.0,
+ default: 5.0,
+ }],
chain: af_bass,
},
Entry {
id: "treble",
- params: &[Param { key: "gain", min: 0.0, max: 12.0, default: 4.0 }],
+ params: &[Param {
+ key: "gain",
+ min: 0.0,
+ max: 12.0,
+ default: 4.0,
+ }],
chain: af_treble,
},
Entry {
id: "telephone",
- params: &[Param { key: "drive", min: 0.0, max: 10.0, default: 3.0 }],
+ params: &[Param {
+ key: "drive",
+ min: 0.0,
+ max: 10.0,
+ default: 3.0,
+ }],
chain: af_telephone,
},
Entry {
id: "echo",
params: &[
- Param { key: "delay", min: 0.05, max: 1.0, default: 0.25 },
- Param { key: "decay", min: 0.1, max: 0.9, default: 0.4 },
+ Param {
+ key: "delay",
+ min: 0.05,
+ max: 1.0,
+ default: 0.25,
+ },
+ Param {
+ key: "decay",
+ min: 0.1,
+ max: 0.9,
+ default: 0.4,
+ },
],
chain: af_echo,
},
Entry {
id: "room",
- params: &[Param { key: "size", min: 0.0, max: 100.0, default: 40.0 }],
+ params: &[Param {
+ key: "size",
+ min: 0.0,
+ max: 100.0,
+ default: 40.0,
+ }],
chain: af_room,
},
];
@@ -485,7 +648,12 @@ fn resolve(entry: &Entry, set: &BTreeMap<String, f64>) -> Params {
entry
.params
.iter()
- .map(|param| (param.key, set.get(param.key).copied().unwrap_or(param.default)))
+ .map(|param| {
+ (
+ param.key,
+ set.get(param.key).copied().unwrap_or(param.default),
+ )
+ })
.collect(),
)
}
@@ -532,7 +700,10 @@ mod tests {
fn applied(id: &str, params: &[(&str, f64)]) -> AppliedFilter {
AppliedFilter {
id: id.to_owned(),
- params: params.iter().map(|(key, value)| ((*key).to_owned(), *value)).collect(),
+ params: params
+ .iter()
+ .map(|(key, value)| ((*key).to_owned(), *value))
+ .collect(),
enabled: true,
}
}
@@ -547,14 +718,20 @@ mod tests {
/// The definition's sliders, all pushed to one bound.
fn at_bound(catalogue: &[Entry], id: &str, minimum: bool) -> AppliedFilter {
- let entry = catalogue.iter().find(|entry| entry.id == id).expect("known id");
+ let entry = catalogue
+ .iter()
+ .find(|entry| entry.id == id)
+ .expect("known id");
AppliedFilter {
id: id.to_owned(),
params: entry
.params
.iter()
.map(|param| {
- (param.key.to_owned(), if minimum { param.min } else { param.max })
+ (
+ param.key.to_owned(),
+ if minimum { param.min } else { param.max },
+ )
})
.collect(),
enabled: true,
@@ -567,7 +744,10 @@ mod tests {
let mut covered: Vec<&str> = table.iter().map(|(id, _)| *id).collect();
covered.sort_unstable();
ids.sort_unstable();
- assert_eq!(covered, ids, "the {which} fixture table must cover the catalogue");
+ assert_eq!(
+ covered, ids,
+ "the {which} fixture table must cover the catalogue"
+ );
}
fn parameterised(catalogue: &[Entry]) -> Vec<&'static str> {
@@ -583,7 +763,10 @@ mod tests {
const EFFECT_DEFAULTS: &[(&str, &str)] = &[
("black-white", "hue=s=0"),
- ("sepia", "colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131"),
+ (
+ "sepia",
+ "colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131",
+ ),
("invert", "negate"),
("sharpen", "unsharp=5:5:1.00:5:5:0"),
("gaussian-blur", "gblur=sigma=10.0"),
@@ -600,9 +783,15 @@ mod tests {
"split[glowa0][glowb0];[glowb0]gblur=sigma=18[glowg0];\
[glowa0][glowg0]blend=all_mode=screen:all_opacity=0.45",
),
- ("posterize", "lutrgb=r=trunc(val/64)*64:g=trunc(val/64)*64:b=trunc(val/64)*64"),
+ (
+ "posterize",
+ "lutrgb=r=trunc(val/64)*64:g=trunc(val/64)*64:b=trunc(val/64)*64",
+ ),
("pixelate", "pixelize=width=16:height=16"),
- ("mirror", "crop=iw/2:ih:0:0,split[mirl0][mirr0];[mirr0]hflip[mirf0];[mirl0][mirf0]hstack"),
+ (
+ "mirror",
+ "crop=iw/2:ih:0:0,split[mirl0][mirr0];[mirr0]hflip[mirf0];[mirl0][mirf0]hstack",
+ ),
("fisheye", "lenscorrection=k1=-0.275:k2=-0.100:i=bilinear"),
("shake", "crop=iw-24:ih-24:12+12*sin(t*13):12+12*cos(t*17)"),
];
@@ -623,7 +812,10 @@ mod tests {
"split[glowa0][glowb0];[glowb0]gblur=sigma=18[glowg0];\
[glowa0][glowg0]blend=all_mode=screen:all_opacity=0.10",
),
- ("posterize", "lutrgb=r=trunc(val/128)*128:g=trunc(val/128)*128:b=trunc(val/128)*128"),
+ (
+ "posterize",
+ "lutrgb=r=trunc(val/128)*128:g=trunc(val/128)*128:b=trunc(val/128)*128",
+ ),
("pixelate", "pixelize=width=2:height=2"),
("fisheye", "lenscorrection=k1=-0.028:k2=-0.010:i=bilinear"),
("shake", "crop=iw-4:ih-4:2+2*sin(t*2):2+2*cos(t*3)"),
@@ -645,7 +837,10 @@ mod tests {
"split[glowa0][glowb0];[glowb0]gblur=sigma=18[glowg0];\
[glowa0][glowg0]blend=all_mode=screen:all_opacity=1.00",
),
- ("posterize", "lutrgb=r=trunc(val/32)*32:g=trunc(val/32)*32:b=trunc(val/32)*32"),
+ (
+ "posterize",
+ "lutrgb=r=trunc(val/32)*32:g=trunc(val/32)*32:b=trunc(val/32)*32",
+ ),
("pixelate", "pixelize=width=64:height=64"),
("fisheye", "lenscorrection=k1=-0.550:k2=-0.200:i=bilinear"),
("shake", "crop=iw-80:ih-80:40+40*sin(t*30):40+40*cos(t*39)"),
@@ -684,7 +879,10 @@ mod tests {
"highpass=f=400,lowpass=f=3400,equalizer=f=1600:t=q:w=1.4:g=4.50,alimiter=limit=0.92",
),
("echo", "aecho=0.8:0.85:250:0.40"),
- ("room", "aecho=0.8:0.88:24|43|70|101:0.320|0.240|0.170|0.110"),
+ (
+ "room",
+ "aecho=0.8:0.88:24|43|70|101:0.320|0.240|0.170|0.110",
+ ),
];
const FILTER_MINIMA: &[(&str, &str)] = &[
@@ -760,7 +958,10 @@ mod tests {
"highpass=f=400,lowpass=f=3400,equalizer=f=1600:t=q:w=1.4:g=8.00,alimiter=limit=0.92",
),
("echo", "aecho=0.8:0.85:1000:0.90"),
- ("room", "aecho=0.8:0.88:46|82|134|194:0.320|0.240|0.170|0.110"),
+ (
+ "room",
+ "aecho=0.8:0.88:46|82|134|194:0.320|0.240|0.170|0.110",
+ ),
];
#[test]
@@ -928,6 +1129,9 @@ mod tests {
#[test]
fn a_set_parameter_overrides_the_default() {
// The filters.test.ts chainKey fixture's chain, minus the key parts.
- assert_eq!(one_filter("echo", &[("delay", 0.5)]), "aecho=0.8:0.85:500:0.40");
+ assert_eq!(
+ one_filter("echo", &[("delay", 0.5)]),
+ "aecho=0.8:0.85:500:0.40"
+ );
}
}
diff --git a/engine/crates/concat-export/src/flatten.rs b/engine/crates/concat-export/src/flatten.rs
index 1d66f854..5d2c1001 100644
--- a/engine/crates/concat-export/src/flatten.rs
+++ b/engine/crates/concat-export/src/flatten.rs
@@ -76,10 +76,13 @@ pub fn flatten_timeline(project: &Project, timeline_id: Option<&str>) -> Vec<Exp
// flattener pre-filtered with its own 1/60 gate - identical
// at 30fps, and where the two differ, resolving is the
// kinder read of what the user placed.
- transition: clip.transition_in.as_ref().map(|transition| TransitionSpec {
- kind: transition.id.clone(),
- duration: transition.duration,
- }),
+ transition: clip
+ .transition_in
+ .as_ref()
+ .map(|transition| TransitionSpec {
+ kind: transition.id.clone(),
+ duration: transition.duration,
+ }),
video_fade_in: 0.0,
media_width: media.width,
media_height: media.height,
@@ -104,8 +107,8 @@ fn pick_timeline<'a>(project: &'a Project, timeline_id: Option<&str>) -> Option<
mod tests {
use std::collections::BTreeMap;
- use concat_project::model::AppliedFilter;
use concat_project::commands::{ClipPatch, NewMedia, TrackFlag};
+ use concat_project::model::AppliedFilter;
use concat_project::{Command, Editor};
use super::*;
@@ -135,7 +138,11 @@ mod tests {
.expect("mints an id");
let track_id = editor.project().timelines[0].tracks[0].id.clone();
let clip_id = editor
- .apply(Command::AddClip { media_id: media_id.clone(), track_id, start: 1.0 })
+ .apply(Command::AddClip {
+ media_id: media_id.clone(),
+ track_id,
+ start: 1.0,
+ })
.expect("adds clip")
.created_id
.expect("mints an id");
@@ -231,7 +238,9 @@ mod tests {
#[test]
fn a_clip_whose_media_vanished_is_skipped_not_fatal() {
let (mut editor, media_id, _) = project_with_clip();
- editor.apply(Command::RemoveMedia { media_id }).expect("removes");
+ editor
+ .apply(Command::RemoveMedia { media_id })
+ .expect("removes");
// RemoveMedia also removes the clips; force the orphan case instead
// by asking for a timeline that does not exist.
assert!(flatten_timeline(editor.project(), None).is_empty());
diff --git a/engine/crates/concat-export/src/lib.rs b/engine/crates/concat-export/src/lib.rs
index 51c439d7..8e4ad454 100644
--- a/engine/crates/concat-export/src/lib.rs
+++ b/engine/crates/concat-export/src/lib.rs
@@ -29,15 +29,13 @@ use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, Ordering};
-use serde::Deserialize;
use concat_core::frame::Frame;
use concat_core::time::{FrameRate, Rational};
use concat_core::timeline::{Clip, ClipId, MediaRef, Timeline, Track, TrackKind, Transform};
use concat_media::audio::{self, AudioClip};
-use concat_media::{
- DecodeOptions, EncodeOptions, FfmpegDecoder, FfmpegEncoder, FrameSink, FrameSource,
-};
+use concat_media::{DecodeOptions, Decoder, EncodeOptions, Encoder, FrameSink, FrameSource};
use concat_render::{Compositor, CpuCompositor, Layer, Placement, plan_frame};
+use serde::Deserialize;
/// What a flattened clip is. Typed, so a kind check the compiler has not
/// seen cannot exist - the wire still says "video"/"audio"/"image".
@@ -240,7 +238,9 @@ fn resolve_transitions(clips: &mut [ExportClip], rate: FrameRate, bake_fades: bo
}
let mut cuts: Vec<Cut> = Vec::new();
for (incoming, clip) in clips.iter().enumerate() {
- let Some(transition) = &clip.transition else { continue };
+ let Some(transition) = &clip.transition else {
+ continue;
+ };
if clip.hidden || !clip.kind.is_visual() {
continue;
}
@@ -301,7 +301,11 @@ fn resolve_transitions(clips: &mut [ExportClip], rate: FrameRate, bake_fades: bo
// at decode. Frame-based, because the decoder emits exactly
// one frame per output frame - so the fade lands on the same
// frames the timeline arithmetic says it covers.
- let colour = if cut.kind == "fade-white" { ":color=white" } else { "" };
+ let colour = if cut.kind == "fade-white" {
+ ":color=white"
+ } else {
+ ""
+ };
let half = cut.duration / 2.0;
{
let a = &mut clips[cut.outgoing];
@@ -402,13 +406,22 @@ pub fn render(request: &ExportRequest, mut reporter: Reporter<'_>) -> Result<Str
// Render into siblings of the output so the move at the end stays on one
// filesystem, then clean up whatever we made.
- let stem = output.file_stem().map_or_else(|| "concat".into(), |s| s.to_string_lossy());
+ let stem = output
+ .file_stem()
+ .map_or_else(|| "concat".into(), |s| s.to_string_lossy());
let directory = output.parent().unwrap_or(Path::new("."));
let silent = directory.join(format!(".{stem}.concat-video.mp4"));
let mixed = directory.join(format!(".{stem}.concat-audio.m4a"));
let result = (|| -> Result<(), String> {
- render_picture(request, rate, total_frames, &visible, &silent, &mut reporter)?;
+ render_picture(
+ request,
+ rate,
+ total_frames,
+ &visible,
+ &silent,
+ &mut reporter,
+ )?;
if sound.is_empty() {
std::fs::rename(&silent, &output)
@@ -498,10 +511,14 @@ fn render_picture(
destination: &Path,
reporter: &mut Reporter<'_>,
) -> Result<(), String> {
- let BuiltTimeline { timeline, stills, decode_sizes, filter_chains } =
- build_timeline(request, rate, visible);
-
- let mut encoder = FfmpegEncoder::create(
+ let BuiltTimeline {
+ timeline,
+ stills,
+ decode_sizes,
+ filter_chains,
+ } = build_timeline(request, rate, visible);
+
+ let mut encoder = Encoder::create(
destination,
request.width,
request.height,
@@ -521,7 +538,7 @@ fn render_picture(
// opened at `output rate / clip speed`, so pulling exactly one frame per
// output frame keeps each of them in step with the plan's source times
// without any seeking - including retimed clips.
- let mut decoders: HashMap<ClipId, FfmpegDecoder> = HashMap::new();
+ let mut decoders: HashMap<ClipId, Decoder> = HashMap::new();
for index in 0..total_frames {
reporter.cancelled()?;
@@ -572,8 +589,7 @@ fn render_picture(
}
entry.insert(
- FfmpegDecoder::open(&layer.media, &options)
- .map_err(|error| error.to_string())?,
+ Decoder::open(&layer.media, &options).map_err(|error| error.to_string())?,
)
}
};
@@ -594,7 +610,9 @@ fn render_picture(
.collect();
let composed = compositor.composite(request.width, request.height, &layers);
- encoder.write_frame(&composed).map_err(|error| error.to_string())?;
+ encoder
+ .write_frame(&composed)
+ .map_err(|error| error.to_string())?;
// Retire decoders whose clip has finished, so a long timeline does not
// hold an ffmpeg process open for every clip it has ever passed.
@@ -622,7 +640,11 @@ struct BuiltTimeline {
}
/// Converts the flattened clip list into an engine timeline.
-fn build_timeline(request: &ExportRequest, rate: FrameRate, visible: &[&ExportClip]) -> BuiltTimeline {
+fn build_timeline(
+ request: &ExportRequest,
+ rate: FrameRate,
+ visible: &[&ExportClip],
+) -> BuiltTimeline {
let mut timeline = Timeline::new(request.width, request.height, rate);
let mut stills = std::collections::HashSet::new();
let mut decode_sizes: HashMap<ClipId, (u32, u32)> = HashMap::new();
@@ -648,8 +670,8 @@ fn build_timeline(request: &ExportRequest, rate: FrameRate, visible: &[&ExportCl
// The same clamp the audio path applies, so a 2x clip means the same
// thing to picture and sound. A still has no meaningful rate.
if clip.kind != ClipKind::Image {
- engine_clip.speed = Rational::approximate(audio::clamp_speed(clip.speed))
- .unwrap_or(Rational::ONE);
+ engine_clip.speed =
+ Rational::approximate(audio::clamp_speed(clip.speed)).unwrap_or(Rational::ONE);
}
engine_clip.transform = Transform {
scale: clip.scale,
@@ -675,7 +697,12 @@ fn build_timeline(request: &ExportRequest, rate: FrameRate, visible: &[&ExportCl
}
}
- BuiltTimeline { timeline, stills, decode_sizes, filter_chains }
+ BuiltTimeline {
+ timeline,
+ stills,
+ decode_sizes,
+ filter_chains,
+ }
}
/// The source's contain-fitted size inside the output frame, or `None` when
@@ -730,8 +757,12 @@ pub fn preview_frame(
request: &PreviewFrameRequest,
) -> Result<Vec<u8>, String> {
let rate = FrameRate::new(Rational::new(request.rate_num, request.rate_den));
- let BuiltTimeline { timeline, stills, decode_sizes, filter_chains } =
- preview_timeline(request, rate);
+ let BuiltTimeline {
+ timeline,
+ stills,
+ decode_sizes,
+ filter_chains,
+ } = preview_timeline(request, rate);
let plan = plan_frame(&timeline, quantise(request.time, rate));
let mut sources: Vec<(std::sync::Arc<Frame>, f32, Transform)> =
@@ -773,7 +804,13 @@ pub fn preview_frame(
let layers: Vec<Layer<'_>> = sources
.iter()
.map(|(frame, opacity, transform)| {
- place_layer(frame.as_ref(), *opacity, transform, request.width, request.height)
+ place_layer(
+ frame.as_ref(),
+ *opacity,
+ transform,
+ request.width,
+ request.height,
+ )
})
.collect();
@@ -824,8 +861,12 @@ pub fn preview_prefetch(
frames: u32,
) {
let rate = FrameRate::new(Rational::new(request.rate_num, request.rate_den));
- let BuiltTimeline { timeline, stills, decode_sizes, filter_chains } =
- preview_timeline(request, rate);
+ let BuiltTimeline {
+ timeline,
+ stills,
+ decode_sizes,
+ filter_chains,
+ } = preview_timeline(request, rate);
let fps = rate.fps().as_f64();
for ahead in 0..frames {
@@ -888,7 +929,10 @@ mod tests {
}
fn spec(kind: &str, duration: f64) -> Option<TransitionSpec> {
- Some(TransitionSpec { kind: kind.to_owned(), duration })
+ Some(TransitionSpec {
+ kind: kind.to_owned(),
+ duration,
+ })
}
/// End to end against a real FFmpeg: the paused monitor's frame must show
@@ -896,16 +940,12 @@ mod tests {
#[test]
fn preview_frame_shows_the_footage_not_black() {
use concat_core::frame::Frame;
- use concat_media::{EncodeOptions, FfmpegEncoder, FrameSink};
+ use concat_media::{EncodeOptions, Encoder, FrameSink};
let path = std::env::temp_dir().join("concat-preview-test.mp4");
- let Ok(mut encoder) = FfmpegEncoder::create(
- &path,
- 64,
- 64,
- FrameRate::THIRTY,
- &EncodeOptions::default(),
- ) else {
+ let Ok(mut encoder) =
+ Encoder::create(&path, 64, 64, FrameRate::THIRTY, &EncodeOptions::default())
+ else {
return; // no ffmpeg here
};
for _ in 0..30 {
@@ -986,14 +1026,20 @@ mod tests {
#[test]
fn a_cross_fade_overlaps_the_incoming_clip_on_its_own_lane() {
- let mut clips = vec![clip("video", 0, 0.0, 4.0, 0.0), clip("video", 0, 4.0, 4.0, 2.0)];
+ let mut clips = vec![
+ clip("video", 0, 0.0, 4.0, 0.0),
+ clip("video", 0, 4.0, 4.0, 2.0),
+ ];
clips[1].transition = spec("cross-fade", 1.0);
resolve_transitions(&mut clips, FrameRate::THIRTY, true);
let b = &clips[1];
assert_eq!(b.start, 3.0, "extends backwards over the cut");
assert_eq!(b.duration, 5.0);
- assert_eq!(b.source_start, 1.0, "consumes the handle before the in-point");
+ assert_eq!(
+ b.source_start, 1.0,
+ "consumes the handle before the in-point"
+ );
assert_eq!(b.video_fade_in, 1.0);
assert_eq!(b.fade_in, 1.0, "sound rides the picture");
assert_eq!(clips[0].track, 0, "outgoing stays on its doubled lane");
@@ -1002,7 +1048,10 @@ mod tests {
#[test]
fn a_cross_fade_clamps_to_the_available_handle() {
- let mut clips = vec![clip("video", 0, 0.0, 4.0, 0.0), clip("video", 0, 4.0, 4.0, 0.25)];
+ let mut clips = vec![
+ clip("video", 0, 0.0, 4.0, 0.0),
+ clip("video", 0, 4.0, 4.0, 0.25),
+ ];
clips[1].transition = spec("cross-fade", 2.0);
resolve_transitions(&mut clips, FrameRate::THIRTY, true);
@@ -1015,28 +1064,46 @@ mod tests {
#[test]
fn a_still_needs_no_handle_to_dissolve() {
- let mut clips = vec![clip("video", 0, 0.0, 4.0, 0.0), clip("image", 0, 4.0, 4.0, 0.0)];
+ let mut clips = vec![
+ clip("video", 0, 0.0, 4.0, 0.0),
+ clip("image", 0, 4.0, 4.0, 0.0),
+ ];
clips[1].transition = spec("cross-fade", 1.0);
resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert_eq!(clips[1].video_fade_in, 1.0);
- assert_eq!(clips[1].source_start, 0.0, "a still has no source clock to rewind");
+ assert_eq!(
+ clips[1].source_start, 0.0,
+ "a still has no source clock to rewind"
+ );
}
#[test]
fn a_fade_to_black_splits_across_the_cut_as_fade_filters() {
- let mut clips = vec![clip("video", 0, 0.0, 4.0, 0.0), clip("video", 0, 4.0, 4.0, 0.0)];
+ let mut clips = vec![
+ clip("video", 0, 0.0, 4.0, 0.0),
+ clip("video", 0, 4.0, 4.0, 0.0),
+ ];
clips[1].transition = spec("fade-black", 1.0);
resolve_transitions(&mut clips, FrameRate::THIRTY, true);
// Half a second each side at 30fps is 15 frames.
- assert_eq!(clips[0].video_filter_chain, "fade=t=out:start_frame=105:nb_frames=15");
- assert_eq!(clips[1].video_filter_chain, "fade=t=in:start_frame=0:nb_frames=15");
+ assert_eq!(
+ clips[0].video_filter_chain,
+ "fade=t=out:start_frame=105:nb_frames=15"
+ );
+ assert_eq!(
+ clips[1].video_filter_chain,
+ "fade=t=in:start_frame=0:nb_frames=15"
+ );
assert_eq!(clips[1].start, 4.0, "nothing moves for an edge fade");
}
#[test]
fn a_fade_to_white_names_its_colour() {
- let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("video", 0, 2.0, 2.0, 0.0)];
+ let mut clips = vec![
+ clip("video", 0, 0.0, 2.0, 0.0),
+ clip("video", 0, 2.0, 2.0, 0.0),
+ ];
clips[1].transition = spec("fade-white", 0.5);
resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert!(clips[0].video_filter_chain.ends_with(":color=white"));
@@ -1045,7 +1112,10 @@ mod tests {
#[test]
fn transition_fades_append_after_the_clips_own_effects() {
- let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("video", 0, 2.0, 2.0, 0.0)];
+ let mut clips = vec![
+ clip("video", 0, 0.0, 2.0, 0.0),
+ clip("video", 0, 2.0, 2.0, 0.0),
+ ];
clips[1].video_filter_chain = "hue=s=0".to_owned();
clips[1].transition = spec("fade-black", 0.5);
resolve_transitions(&mut clips, FrameRate::THIRTY, true);
@@ -1058,7 +1128,10 @@ mod tests {
#[test]
fn a_transition_with_no_adjacent_clip_is_orphaned_not_fatal() {
- let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("video", 0, 5.0, 2.0, 0.0)];
+ let mut clips = vec![
+ clip("video", 0, 0.0, 2.0, 0.0),
+ clip("video", 0, 5.0, 2.0, 0.0),
+ ];
clips[1].transition = spec("cross-fade", 1.0);
resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert_eq!(clips[1].start, 5.0);
@@ -1067,7 +1140,10 @@ mod tests {
#[test]
fn track_indices_are_doubled_for_everyone() {
- let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("audio", 3, 0.0, 2.0, 0.0)];
+ let mut clips = vec![
+ clip("video", 0, 0.0, 2.0, 0.0),
+ clip("audio", 3, 0.0, 2.0, 0.0),
+ ];
resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert_eq!(clips[0].track, 0);
assert_eq!(clips[1].track, 6);
@@ -1075,7 +1151,10 @@ mod tests {
#[test]
fn an_unknown_transition_kind_renders_as_a_plain_cut() {
- let mut clips = vec![clip("video", 0, 0.0, 2.0, 0.0), clip("video", 0, 2.0, 2.0, 0.0)];
+ let mut clips = vec![
+ clip("video", 0, 0.0, 2.0, 0.0),
+ clip("video", 0, 2.0, 2.0, 0.0),
+ ];
clips[1].transition = spec("wipe-left", 1.0);
resolve_transitions(&mut clips, FrameRate::THIRTY, true);
assert_eq!(clips[1].start, 2.0);
diff --git a/engine/crates/concat-host/Cargo.toml b/engine/crates/concat-host/Cargo.toml
new file mode 100644
index 00000000..e9efea20
--- /dev/null
+++ b/engine/crates/concat-host/Cargo.toml
@@ -0,0 +1,28 @@
+[package]
+name = "concat-host"
+version.workspace = true
+edition.workspace = true
+rust-version.workspace = true
+license.workspace = true
+description = "What the editor window needs that is not the edit itself: sessions, project folders, previews, playback, templates."
+
+[features]
+# The wgpu compositor for previews and exports, off by default for the same
+# reason as everywhere else in the engine; the window turns it on.
+gpu = ["concat-export/gpu"]
+
+[dependencies]
+concat-core.workspace = true
+concat-media.workspace = true
+concat-project.workspace = true
+concat-export.workspace = true
+serde = { version = "1", features = ["derive"] }
+serde_json.workspace = true
+
+# Engine-owned audio playback: cpal talks to the output device, memmap2 keeps
+# decoded PCM disk-backed instead of resident.
+cpal = "0.16"
+memmap2 = "0.9"
+
+[lints]
+workspace = true
diff --git a/engine/crates/concat-host/src/dirs.rs b/engine/crates/concat-host/src/dirs.rs
new file mode 100644
index 00000000..9f497f65
--- /dev/null
+++ b/engine/crates/concat-host/src/dirs.rs
@@ -0,0 +1,92 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Where the app keeps what belongs to this machine rather than to a
+//! project: the recents list, remembered settings, downloaded models.
+//!
+//! Two directories, named by the app identifier and following each
+//! platform's convention for configuration and data. On macOS and Windows
+//! they are the same folder; on Linux they follow the XDG split.
+
+use std::path::{Path, PathBuf};
+
+/// The app identifier, which is also the folder name everywhere.
+pub const IDENTIFIER: &str = "app.concat.editor";
+
+/// The app's own directories on this machine.
+#[derive(Clone, Debug)]
+pub struct AppDirs {
+ /// Small state: recents, remembered settings, the template library.
+ pub config: PathBuf,
+ /// Large state: downloaded models.
+ pub data: PathBuf,
+}
+
+impl AppDirs {
+ /// The platform's directories for this app, created lazily by whoever
+ /// writes into them.
+ pub fn locate() -> Result<AppDirs, String> {
+ if cfg!(target_os = "macos") {
+ let dir = home()?
+ .join("Library")
+ .join("Application Support")
+ .join(IDENTIFIER);
+ Ok(AppDirs {
+ config: dir.clone(),
+ data: dir,
+ })
+ } else if cfg!(windows) {
+ let dir = std::env::var_os("APPDATA")
+ .map(PathBuf::from)
+ .ok_or_else(|| "APPDATA is not set".to_owned())?
+ .join(IDENTIFIER);
+ Ok(AppDirs {
+ config: dir.clone(),
+ data: dir,
+ })
+ } else {
+ let home = home()?;
+ let config_base = std::env::var_os("XDG_CONFIG_HOME")
+ .map(PathBuf::from)
+ .filter(|path| path.is_absolute())
+ .unwrap_or_else(|| home.join(".config"));
+ let data_base = std::env::var_os("XDG_DATA_HOME")
+ .map(PathBuf::from)
+ .filter(|path| path.is_absolute())
+ .unwrap_or_else(|| home.join(".local").join("share"));
+ Ok(AppDirs {
+ config: config_base.join(IDENTIFIER),
+ data: data_base.join(IDENTIFIER),
+ })
+ }
+ }
+
+ /// Both directories under one root. For tests, and for anyone running
+ /// a portable build out of a folder.
+ pub fn under(root: &Path) -> AppDirs {
+ AppDirs {
+ config: root.to_path_buf(),
+ data: root.to_path_buf(),
+ }
+ }
+}
+
+fn home() -> Result<PathBuf, String> {
+ std::env::var_os("HOME")
+ .or_else(|| std::env::var_os("USERPROFILE"))
+ .map(PathBuf::from)
+ .ok_or_else(|| "could not locate the home directory".to_owned())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn the_directories_are_named_by_the_identifier() {
+ let dirs = AppDirs::locate().expect("locates");
+ assert!(dirs.config.ends_with(IDENTIFIER));
+ assert!(dirs.data.ends_with(IDENTIFIER));
+ assert!(dirs.config.is_absolute());
+ }
+}
diff --git a/engine/crates/concat-host/src/export.rs b/engine/crates/concat-host/src/export.rs
new file mode 100644
index 00000000..dbed3a7e
--- /dev/null
+++ b/engine/crates/concat-host/src/export.rs
@@ -0,0 +1,111 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Rendering the session to a file.
+//!
+//! The rendering is `concat-export`'s; what is here is the seam the window
+//! uses: the session flattens itself, the window adds the destination and
+//! the quality, and the one export that can run at a time reports through a
+//! callback and stops through a flag.
+
+use std::sync::Arc;
+use std::sync::atomic::AtomicBool;
+
+pub use concat_export::{ExportClip, ExportRequest, Reporter, render};
+
+use crate::jobs::{Job, SingleFlight};
+use crate::session::Session;
+
+/// What the window decides about an export: where it goes, how good it is,
+/// and any rasterised titles rejoining as image clips.
+#[derive(Clone, Debug, Default)]
+pub struct ExportSpec {
+ /// The file to write.
+ pub output: String,
+ /// Constant rate factor; lower is better quality and a bigger file.
+ pub crf: u8,
+ /// The x264 speed/size preset name, e.g. "medium".
+ pub preset: String,
+}
+
+/// One progress report: which frame of how many, in which stage.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct Progress {
+ /// Frames done.
+ pub frame: i64,
+ /// Frames in total.
+ pub total: i64,
+ /// What the exporter is doing: "video", "audio", "mux".
+ pub stage: &'static str,
+}
+
+/// The session flattened into a request the engine can render.
+pub fn request(session: &Session, spec: &ExportSpec, titles: Vec<ExportClip>) -> ExportRequest {
+ let settings = session.settings();
+ let mut clips = session.flattened_clips();
+ clips.extend(titles);
+ ExportRequest {
+ output: spec.output.clone(),
+ width: settings.width,
+ height: settings.height,
+ rate_num: settings.rate_num,
+ rate_den: settings.rate_den,
+ crf: spec.crf,
+ preset: spec.preset.clone(),
+ clips,
+ }
+}
+
+/// Renders `request` and returns the path written, reporting through
+/// `progress` and stopping at the next frame once `cancel` is set. Blocks for
+/// the whole render: run it on its own thread.
+pub fn run(
+ request: &ExportRequest,
+ cancel: &AtomicBool,
+ mut progress: impl FnMut(Progress),
+) -> Result<String, String> {
+ let mut report = |frame: i64, total: i64, stage: &'static str| {
+ progress(Progress {
+ frame,
+ total,
+ stage,
+ });
+ };
+ render(
+ request,
+ Reporter {
+ progress: &mut report,
+ cancel,
+ },
+ )
+}
+
+/// The slot for the one export that can run at a time - enforced, not
+/// documented: a second export while one runs is refused instead of racing
+/// the first for its temp files and cancel flag.
+#[derive(Clone, Default)]
+pub struct Exporter {
+ slot: Arc<SingleFlight>,
+}
+
+impl Exporter {
+ /// An idle exporter.
+ pub fn new() -> Self {
+ Self::default()
+ }
+
+ /// Claims the slot for a new export, whose cancel flag the render polls.
+ pub fn begin(&self) -> Result<Job, String> {
+ self.slot.begin("export")
+ }
+
+ /// Asks the running export to stop at the next frame. Idle is a no-op.
+ pub fn cancel(&self) {
+ self.slot.cancel();
+ }
+
+ /// Whether an export is running.
+ pub fn is_busy(&self) -> bool {
+ self.slot.is_busy()
+ }
+}
diff --git a/engine/crates/concat-host/src/jobs.rs b/engine/crates/concat-host/src/jobs.rs
new file mode 100644
index 00000000..3aedd01a
--- /dev/null
+++ b/engine/crates/concat-host/src/jobs.rs
@@ -0,0 +1,151 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! One-at-a-time job slots for long-running work.
+//!
+//! Export, transcription and model downloads are all "one at a time", and
+//! that invariant used to live in doc comments while the commands happily
+//! accepted a second run: two exports would race each other's temp files, and
+//! starting a second job *cleared* the shared cancel flag out from under the
+//! first. [`SingleFlight`] makes the invariant code: `begin` refuses a second
+//! concurrent job, and every run gets its own cancel flag, so a cancel can
+//! only ever stop the job that is actually running and a new job can never be
+//! un-cancelled by a stale start or stopped by a stale cancel.
+
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::sync::{Arc, Mutex};
+
+/// The slot: at most one running job, identified by its own cancel flag.
+pub struct SingleFlight {
+ slot: Mutex<Option<Arc<AtomicBool>>>,
+}
+
+impl Default for SingleFlight {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SingleFlight {
+ /// An empty slot.
+ pub fn new() -> Self {
+ Self {
+ slot: Mutex::new(None),
+ }
+ }
+
+ /// Claims the slot for a new job, or reports the one already running.
+ ///
+ /// `what` is the user-facing job name ("export", "transcription", ...).
+ pub fn begin(self: &Arc<Self>, what: &str) -> Result<Job, String> {
+ let mut slot = self
+ .slot
+ .lock()
+ .map_err(|_| format!("{what} state poisoned"))?;
+ if slot.is_some() {
+ return Err(format!(
+ "a {what} is already running - wait for it or cancel it first"
+ ));
+ }
+ let cancel = Arc::new(AtomicBool::new(false));
+ *slot = Some(Arc::clone(&cancel));
+ Ok(Job {
+ owner: Arc::clone(self),
+ cancel,
+ })
+ }
+
+ /// Whether a job currently holds the slot.
+ pub fn is_busy(&self) -> bool {
+ self.slot.lock().map(|slot| slot.is_some()).unwrap_or(false)
+ }
+
+ /// Cancels the running job. Idle is a harmless no-op.
+ pub fn cancel(&self) {
+ if let Ok(slot) = self.slot.lock()
+ && let Some(cancel) = slot.as_ref()
+ {
+ cancel.store(true, Ordering::Relaxed);
+ }
+ }
+}
+
+/// A claim on the slot, released on drop - however the job ended, panic
+/// included, the slot frees and the next `begin` succeeds.
+pub struct Job {
+ owner: Arc<SingleFlight>,
+ cancel: Arc<AtomicBool>,
+}
+
+impl Job {
+ /// This run's stop flag, for the worker to poll.
+ pub fn cancel_flag(&self) -> &AtomicBool {
+ &self.cancel
+ }
+
+ /// An owned handle on the same flag, for callbacks that outlive borrows
+ /// of the job (an FFI progress callback must be `'static`).
+ pub fn cancel_handle(&self) -> Arc<AtomicBool> {
+ Arc::clone(&self.cancel)
+ }
+
+ /// Whether this run has been asked to stop.
+ pub fn cancelled(&self) -> bool {
+ self.cancel.load(Ordering::Relaxed)
+ }
+}
+
+impl Drop for Job {
+ fn drop(&mut self) {
+ if let Ok(mut slot) = self.owner.slot.lock() {
+ // Release only our own claim. Guards are one-per-begin so the
+ // check is belt and braces, but it makes a stale drop harmless.
+ if slot
+ .as_ref()
+ .is_some_and(|flag| Arc::ptr_eq(flag, &self.cancel))
+ {
+ *slot = None;
+ }
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn refuses_a_second_concurrent_job() {
+ let flight = Arc::new(SingleFlight::new());
+ let job = flight.begin("export").expect("first begin");
+ assert!(flight.is_busy());
+ match flight.begin("export") {
+ Err(error) => assert!(error.contains("already running")),
+ Ok(_) => panic!("second begin must be refused"),
+ }
+ drop(job);
+ assert!(!flight.is_busy());
+ assert!(flight.begin("export").is_ok(), "slot frees on drop");
+ }
+
+ #[test]
+ fn cancel_reaches_only_the_running_job() {
+ let flight = Arc::new(SingleFlight::new());
+ let first = flight.begin("export").expect("begin");
+ flight.cancel();
+ assert!(first.cancelled());
+
+ drop(first);
+ // A new job starts with a fresh, uncancelled flag: the earlier cancel
+ // cannot leak forward, and starting fresh cannot un-cancel anyone.
+ let second = flight.begin("export").expect("begin again");
+ assert!(!second.cancelled());
+ }
+
+ #[test]
+ fn cancel_when_idle_is_a_no_op() {
+ let flight = Arc::new(SingleFlight::new());
+ flight.cancel();
+ assert!(flight.begin("export").is_ok());
+ }
+}
diff --git a/engine/crates/concat-host/src/lib.rs b/engine/crates/concat-host/src/lib.rs
new file mode 100644
index 00000000..07b7d25f
--- /dev/null
+++ b/engine/crates/concat-host/src/lib.rs
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! What the editor window needs that is not the edit itself.
+//!
+//! The edit lives in `concat-project`, pixels in `concat-render` and
+//! `concat-export`, files in `concat-media`. This crate is the layer the
+//! window talks to: it opens a project folder as a [`session::Session`],
+//! keeps the recents list, caches waveforms and filmstrips beside the
+//! project, composites the paused monitor's true frame, plays the audible
+//! clips, packs templates, and holds the one-at-a-time slots for long jobs.
+//!
+//! It used to be the Tauri host of the React editor, where every one of
+//! these was a `#[tauri::command]` converting wire formats. There is no wire
+//! any more - the Slint window calls these functions in-process - so what is
+//! left is the plumbing itself, with the same doctrine: no editing decision
+//! is made here. If a function starts deciding what an edit means, it
+//! belongs in `concat-project`.
+//!
+//! Nothing here knows about a window. Long work reports through callbacks
+//! and cancels through flags, and the caller decides which thread it runs on.
+
+pub mod dirs;
+pub mod export;
+pub mod jobs;
+pub mod media;
+pub mod playback;
+pub mod preview;
+pub mod projects;
+pub mod session;
+pub mod templates;
+
+pub use dirs::AppDirs;
+pub use jobs::{Job, SingleFlight};
+pub use projects::ProjectInfo;
+pub use session::{EditorView, Session, SettingsView};
diff --git a/engine/crates/concat-host/src/media.rs b/engine/crates/concat-host/src/media.rs
new file mode 100644
index 00000000..ce992f45
--- /dev/null
+++ b/engine/crates/concat-host/src/media.rs
@@ -0,0 +1,540 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Media as the window sees it: what a file is, its waveform, its
+//! filmstrip, a project's poster - and the caches beside the project that
+//! keep each of those from being computed twice.
+
+use std::path::{Path, PathBuf};
+
+use concat_core::frame::Frame;
+use concat_core::time::Rational;
+use concat_media::{DecodeOptions, Decoder, FrameSource, SeekableSource};
+use serde::Serialize;
+
+use crate::projects;
+
+/// A video stream, as the UI sees it.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct VideoStreamInfo {
+ /// Stream index within the file.
+ pub index: u32,
+ /// Codec short name.
+ pub codec: String,
+ /// Displayed width in pixels.
+ pub width: u32,
+ /// Displayed height in pixels.
+ pub height: u32,
+ /// Decimal fps, for display only.
+ pub frame_rate: f64,
+ /// The exact fraction the engine actually works in, e.g. "30000/1001".
+ pub frame_rate_fraction: String,
+}
+
+/// An audio stream, as the UI sees it.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct AudioStreamInfo {
+ /// Stream index within the file.
+ pub index: u32,
+ /// Codec short name.
+ pub codec: String,
+ /// Samples per second.
+ pub sample_rate: u32,
+ /// Channel count.
+ pub channels: u32,
+}
+
+/// What [`probe`] hands back.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct MediaSummary {
+ /// The file, as given.
+ pub path: String,
+ /// Container duration in seconds, when the container states one.
+ pub duration: Option<f64>,
+ /// What the file is.
+ pub kind: concat_project::model::MediaKind,
+ /// First video stream, if any.
+ pub video: Option<VideoStreamInfo>,
+ /// First audio stream, if any.
+ pub audio: Option<AudioStreamInfo>,
+}
+
+impl MediaSummary {
+ /// The bin entry this file becomes, named after its basename.
+ pub fn to_new_media(&self) -> concat_project::commands::NewMedia {
+ concat_project::commands::NewMedia {
+ path: self.path.clone(),
+ name: Path::new(&self.path)
+ .file_name()
+ .map(|name| name.to_string_lossy().into_owned())
+ .unwrap_or_else(|| self.path.clone()),
+ duration: self.duration,
+ kind: self.kind,
+ width: self.video.as_ref().map(|video| video.width),
+ height: self.video.as_ref().map(|video| video.height),
+ frame_rate: self.video.as_ref().map(|video| video.frame_rate),
+ frame_rate_fraction: self
+ .video
+ .as_ref()
+ .map(|video| video.frame_rate_fraction.clone()),
+ video_codec: self.video.as_ref().map(|video| video.codec.clone()),
+ audio_codec: self.audio.as_ref().map(|audio| audio.codec.clone()),
+ has_audio: self.audio.is_some(),
+ }
+ }
+}
+
+/// Extensions Concat is willing to treat as stills.
+const IMAGE_EXTENSIONS: &[&str] = &[
+ "png", "jpg", "jpeg", "webp", "bmp", "tif", "tiff", "avif", "heic", "heif", "gif",
+];
+
+/// Decides whether a file is footage, sound or a still.
+///
+/// Extension first, because that is what the user means by "a png", and the
+/// demuxer is genuinely ambiguous here: a PNG presents as a one-frame video
+/// stream, usually with a frame rate of 25/1 invented by the demuxer.
+///
+/// The duration check is what separates a still from an animation. An animated
+/// GIF or WebP reports a duration; a single image does not. It is a heuristic,
+/// and a deliberately conservative one - misreading an animation as a still
+/// shows its first frame rather than failing.
+fn classify(info: &concat_media::MediaInfo) -> concat_project::model::MediaKind {
+ use concat_project::model::MediaKind;
+ if info.video.is_none() {
+ return MediaKind::Audio;
+ }
+
+ let extension = info
+ .path
+ .extension()
+ .and_then(|value| value.to_str())
+ .unwrap_or_default()
+ .to_ascii_lowercase();
+
+ if IMAGE_EXTENSIONS.contains(&extension.as_str()) && info.duration.is_none() {
+ MediaKind::Image
+ } else {
+ MediaKind::Video
+ }
+}
+
+impl From<concat_media::MediaInfo> for MediaSummary {
+ fn from(info: concat_media::MediaInfo) -> Self {
+ Self {
+ kind: classify(&info),
+ path: info.path.to_string_lossy().into_owned(),
+ // Exact rational time stops at this boundary: the document is
+ // f64 seconds, and the UI only ever displays these numbers.
+ duration: info.duration.map(|duration| duration.as_f64()),
+ video: info.video.map(|video| VideoStreamInfo {
+ index: video.index,
+ codec: video.codec,
+ width: video.width,
+ height: video.height,
+ frame_rate: video.frame_rate.fps().as_f64(),
+ frame_rate_fraction: format!(
+ "{}/{}",
+ video.frame_rate.fps().numerator(),
+ video.frame_rate.fps().denominator()
+ ),
+ }),
+ audio: info.audio.map(|audio| AudioStreamInfo {
+ index: audio.index,
+ codec: audio.codec,
+ sample_rate: audio.sample_rate,
+ channels: audio.channels,
+ }),
+ }
+ }
+}
+
+/// Reports what is inside a media file. Opening a file on a slow or network
+/// volume takes real time, so call this off the UI thread.
+pub fn probe(path: &str) -> Result<MediaSummary, String> {
+ concat_media::probe(path)
+ .map(MediaSummary::from)
+ .map_err(describe)
+}
+
+/// The most [`read_bytes`] will hand back at once.
+///
+/// Its callers decode still images and register fonts - assets that are
+/// megabytes, not gigabytes. The cap is what keeps the function from quietly
+/// growing into a whole-disk read primitive.
+pub const MEDIA_READ_CAP: u64 = 64 * 1024 * 1024;
+
+/// Reads a whole file, refusing anything past [`MEDIA_READ_CAP`].
+pub fn read_bytes(path: &str) -> Result<Vec<u8>, String> {
+ let size = std::fs::metadata(path)
+ .map_err(|error| format!("could not read {path}: {error}"))?
+ .len();
+ if size > MEDIA_READ_CAP {
+ return Err(format!(
+ "refusing to read {path}: {size} bytes is over the {MEDIA_READ_CAP} byte limit"
+ ));
+ }
+ std::fs::read(path).map_err(|error| format!("could not read {path}: {error}"))
+}
+
+/// Resolution of the cached waveform.
+///
+/// 200 buckets per second is roughly two buckets per pixel at the default
+/// timeline zoom, which is enough that the drawn shape does not visibly
+/// change as you zoom in a step or two, without storing the whole decoded
+/// file.
+pub const PEAKS_BUCKETS_PER_SECOND: u32 = 200;
+
+/// Waveform peaks for one media file: engine-decoded, project-cached.
+///
+/// The engine streams the decode into min/max buckets, so neither the file
+/// nor its samples are ever resident. The result is cached in the project's
+/// `cache/` folder under a key derived from the path, and served from there
+/// on every later call; `project: None` (an unsaved session) just skips the
+/// cache.
+pub fn peaks(path: &str, project: Option<&str>) -> Result<concat_media::peaks::Peaks, String> {
+ use concat_media::peaks::Peaks;
+
+ let cached = project.and_then(|project| artwork_file(project, &peaks_key(path)).ok());
+ if let Some(file) = &cached
+ && let Ok(bytes) = std::fs::read(file)
+ && let Some(peaks) = Peaks::decode(&bytes)
+ {
+ // A corrupt entry falls through to regeneration rather than being
+ // served.
+ return Ok(peaks);
+ }
+
+ let peaks = concat_media::peaks::extract(Path::new(path), PEAKS_BUCKETS_PER_SECOND)
+ .map_err(describe)?;
+
+ // Best-effort, like every artwork write: a failed cache write only
+ // means decoding again next launch.
+ if let Some(file) = &cached {
+ if let Some(parent) = file.parent() {
+ let _ = std::fs::create_dir_all(parent);
+ }
+ let _ = std::fs::write(file, peaks.encode());
+ }
+ Ok(peaks)
+}
+
+/// The cache filename for one file's peaks.
+///
+/// FNV-1a 64 over the path, like the audio cache's `decode_key` and for the
+/// same reason: these keys name files that outlive the process, and
+/// `DefaultHasher` is free to change between Rust releases. The bucket rate
+/// rides in the name so a resolution change regenerates instead of serving
+/// yesterday's shape.
+fn peaks_key(path: &str) -> String {
+ format!(
+ "{:016x}-b{PEAKS_BUCKETS_PER_SECOND}.peaks",
+ fnv1a(path.as_bytes())
+ )
+}
+
+/// FNV-1a 64, for cache keys that must survive toolchain upgrades.
+pub fn fnv1a(bytes: &[u8]) -> u64 {
+ let mut hash: u64 = 0xcbf2_9ce4_8422_2325;
+ for &byte in bytes {
+ hash ^= u64::from(byte);
+ hash = hash.wrapping_mul(0x0000_0100_0000_01b3);
+ }
+ hash
+}
+
+/// Where one artwork file lives inside a project's cache.
+///
+/// The cache sits in the project folder so it travels with the project and
+/// vanishes with it. The key is confined to a single flat filename - anything
+/// that could walk out of the folder is refused rather than sanitised,
+/// because the only caller is our own window and a strange key is a bug.
+pub fn artwork_file(project: &str, key: &str) -> Result<PathBuf, String> {
+ if key.is_empty()
+ || !key
+ .chars()
+ .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-'))
+ || key.starts_with('.')
+ {
+ return Err(format!("refusing artwork key {key:?}"));
+ }
+ let root = Path::new(project);
+ // A real project, not merely a directory: the manifest is what makes a
+ // folder ours to write a `cache/` into.
+ if !projects::is_project(root) {
+ return Err(format!("{project} is not a project folder"));
+ }
+ Ok(root.join("cache").join(key))
+}
+
+/// Returns one cached artwork file, or an error the caller treats as a miss.
+pub fn read_artwork(project: &str, key: &str) -> Result<Vec<u8>, String> {
+ let file = artwork_file(project, key)?;
+ std::fs::read(&file).map_err(|error| format!("no cached artwork {key}: {error}"))
+}
+
+/// Stores one artwork file in the project's cache. Best-effort in spirit: a
+/// failed write only means regenerating next launch.
+pub fn write_artwork(project: &str, key: &str, bytes: &[u8]) -> Result<(), String> {
+ let file = artwork_file(project, key)?;
+ if let Some(parent) = file.parent() {
+ std::fs::create_dir_all(parent)
+ .map_err(|error| format!("could not create {}: {error}", parent.display()))?;
+ }
+ std::fs::write(&file, bytes).map_err(|error| format!("could not write {key}: {error}"))
+}
+
+/// Renders a strip of evenly spaced frames from a video as one picture.
+///
+/// One image rather than N, because the timeline draws the frames as slices
+/// of a single texture - that is one texture upload instead of twenty-four.
+/// Each frame is sought directly, so the cost is `count` seeks, not a decode
+/// of the whole file. If the container reports no duration there is nothing
+/// to space frames across, so this refuses rather than guessing.
+pub fn filmstrip(path: &str, count: u32, height: u32) -> Result<Frame, String> {
+ let count = count.clamp(1, 60);
+ let height = height.clamp(16, 240);
+
+ let info = concat_media::probe(path).map_err(describe)?;
+ let video = info.require_video().map_err(describe)?;
+ let duration = info
+ .duration
+ .map(|duration| duration.as_f64())
+ .filter(|seconds| *seconds > 0.0)
+ .ok_or_else(|| format!("{path} reports no duration"))?;
+ // Aspect-correct and even, which is what the scaler is happiest with.
+ let width = ((f64::from(height) * f64::from(video.width) / f64::from(video.height)).round()
+ as u32)
+ .max(2)
+ & !1;
+
+ let mut decoder = Decoder::open(path, &DecodeOptions::default().scaled_to(width, height))
+ .map_err(describe)?;
+ let mut strip = Frame::black(width * count, height);
+ let mut last: Option<Frame> = None;
+ for index in 0..count {
+ // Sample the middle of each slice, so the first frame is not always
+ // the file's own first (often black) frame.
+ let at = duration * (f64::from(index) + 0.5) / f64::from(count);
+ let time = Rational::approximate(at).unwrap_or(Rational::ZERO);
+ let frame = match decoder.seek(time).and_then(|()| decoder.next_frame()) {
+ Ok(Some(frame)) => Some(frame),
+ // Past the last frame, or a stretch that will not decode: repeat
+ // the last picture rather than leaving a hole.
+ _ => None,
+ };
+ let frame = frame.or_else(|| last.take());
+ if let Some(frame) = &frame {
+ strip.blit(frame, index * width, 0);
+ }
+ last = frame;
+ }
+ Ok(strip)
+}
+
+/// A small poster frame for one project, as a JPEG, for the launch screen's
+/// recents.
+///
+/// Grabbed from the earliest visible clip of the project's active timeline
+/// and cached as `cache/preview.jpg` in the project folder; the cache is
+/// fresh as long as it is newer than the manifest, so an edited project gets
+/// a new poster on its next appearance and an untouched one costs a stat.
+pub fn poster_frame(project: &str) -> Result<Vec<u8>, String> {
+ let root = Path::new(project);
+ let manifest = projects::manifest_path(root);
+ let cached = root.join("cache").join("preview.jpg");
+
+ let fresh = match (std::fs::metadata(&cached), std::fs::metadata(&manifest)) {
+ (Ok(cache), Ok(source)) => match (cache.modified(), source.modified()) {
+ (Ok(cache), Ok(source)) => cache >= source,
+ _ => false,
+ },
+ _ => false,
+ };
+ if fresh && let Ok(bytes) = std::fs::read(&cached) {
+ return Ok(bytes);
+ }
+
+ let (media_path, source_start, is_still) = poster_source(project)?;
+ let frame = still_at(&media_path, if is_still { 0.0 } else { source_start }, 480)?;
+ let bytes = concat_media::jpeg(&frame, 4).map_err(describe)?;
+
+ // Best effort: a failed cache write only means regenerating next launch.
+ if let Some(parent) = cached.parent() {
+ let _ = std::fs::create_dir_all(parent);
+ let _ = std::fs::write(&cached, &bytes);
+ }
+ Ok(bytes)
+}
+
+/// Which frame of which file is a project's poster: the earliest clip with
+/// a picture on the active timeline - exactly the frame the user last saw
+/// open.
+fn poster_source(project: &str) -> Result<(String, f64, bool), String> {
+ let manifest = projects::manifest_path(Path::new(project));
+ let text = std::fs::read_to_string(&manifest)
+ .map_err(|error| format!("could not read {}: {error}", manifest.display()))?;
+ let document: serde_json::Value =
+ serde_json::from_str(&text).map_err(|error| format!("not a project: {error}"))?;
+
+ // Typed access through the engine's own reader, not hand-parsed JSON -
+ // a schema change breaks this at compile time now, not silently at the
+ // next launch screen.
+ let Some(project) = concat_project::from_document(&document) else {
+ return Err("the project has no timeline to preview".to_owned());
+ };
+ let timeline = project
+ .timelines
+ .iter()
+ .find(|timeline| timeline.id == project.active_timeline_id)
+ .or_else(|| project.timelines.first());
+ let Some(timeline) = timeline else {
+ return Err("the project has no timeline to preview".to_owned());
+ };
+
+ use concat_project::model::ClipKind;
+ let mut poster: Option<(f64, String, f64, bool)> = None;
+ for clip in &timeline.clips {
+ if clip.kind != ClipKind::Video && clip.kind != ClipKind::Image {
+ continue;
+ }
+ if poster
+ .as_ref()
+ .is_some_and(|(best, ..)| *best <= clip.start)
+ {
+ continue;
+ }
+ let Some(media) = project.media.iter().find(|item| item.id == clip.media_id) else {
+ continue;
+ };
+ poster = Some((
+ clip.start,
+ media.path.clone(),
+ clip.source_start,
+ clip.kind == ClipKind::Image,
+ ));
+ }
+ poster
+ .map(|(_, path, source_start, still)| (path, source_start, still))
+ .ok_or_else(|| "nothing on the timeline to preview".to_owned())
+}
+
+/// One frame of `path` at `seconds`, scaled to `width` across with the
+/// height following the picture's shape.
+pub fn still_at(path: &str, seconds: f64, width: u32) -> Result<Frame, String> {
+ let info = concat_media::probe(path).map_err(describe)?;
+ let video = info.require_video().map_err(describe)?;
+ let height = ((f64::from(width) * f64::from(video.height) / f64::from(video.width)).round()
+ as u32)
+ .max(2)
+ & !1;
+ let mut options = DecodeOptions::default().scaled_to(width, height);
+ if seconds > 0.0 {
+ options = options.starting_at(Rational::approximate(seconds).unwrap_or(Rational::ZERO));
+ }
+ let mut decoder = Decoder::open(path, &options).map_err(describe)?;
+ match decoder.next_frame().map_err(describe)? {
+ Some(frame) => Ok(frame),
+ // Past the end: the file's first frame beats nothing.
+ None => {
+ decoder.seek(Rational::ZERO).map_err(describe)?;
+ decoder
+ .next_frame()
+ .map_err(describe)?
+ .ok_or_else(|| format!("no frame to show for {path}"))
+ }
+ }
+}
+
+/// Flattens an error and its causes into one line.
+///
+/// `Display` on a `thiserror` enum prints only the outermost message, and the
+/// useful half - what FFmpeg or the OS actually said - is in the source chain.
+pub fn describe(error: concat_media::Error) -> String {
+ use std::error::Error;
+
+ let mut message = error.to_string();
+ let mut cause = error.source();
+ while let Some(current) = cause {
+ message.push_str(&format!(": {current}"));
+ cause = current.source();
+ }
+ message
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn artwork_keys_cannot_leave_the_cache() {
+ let scratch =
+ std::env::temp_dir().join(format!("concat-artwork-test-{}", std::process::id()));
+ let _ = std::fs::remove_dir_all(&scratch);
+ std::fs::create_dir_all(&scratch).expect("scratch dir");
+ let project = scratch.to_string_lossy().into_owned();
+
+ // Not a project yet: refused however good the key.
+ assert!(artwork_file(&project, "ok.jpg").is_err());
+ std::fs::write(scratch.join("concat.json"), b"{}").expect("writes");
+ assert!(artwork_file(&project, "ok.jpg").is_ok());
+ for bad in ["", "../x", ".hidden", "a/b", "a\\b"] {
+ assert!(
+ artwork_file(&project, bad).is_err(),
+ "{bad:?} must be refused"
+ );
+ }
+
+ write_artwork(&project, "poster.jpg", b"jpeg").expect("writes");
+ assert_eq!(
+ read_artwork(&project, "poster.jpg").expect("reads"),
+ b"jpeg"
+ );
+ let _ = std::fs::remove_dir_all(&scratch);
+ }
+
+ #[test]
+ fn cache_keys_are_stable() {
+ // Pinned: a changed hash would orphan every project's caches.
+ assert_eq!(fnv1a(b""), 0xcbf2_9ce4_8422_2325);
+ assert_eq!(
+ peaks_key("/a.mp4"),
+ format!("{:016x}-b200.peaks", fnv1a(b"/a.mp4"))
+ );
+ }
+
+ #[test]
+ fn a_filmstrip_of_a_synthetic_video_has_the_asked_for_shape() {
+ use concat_core::time::FrameRate;
+ use concat_media::{EncodeOptions, Encoder, FrameSink};
+
+ let path =
+ std::env::temp_dir().join(format!("concat-filmstrip-test-{}.mp4", std::process::id()));
+ let mut encoder =
+ Encoder::create(&path, 64, 32, FrameRate::THIRTY, &EncodeOptions::default())
+ .expect("encodes");
+ for index in 0..60u32 {
+ let mut frame = Frame::black(64, 32);
+ frame.fill([(index * 4).min(255) as u8, 60, 60, 255]);
+ encoder.write_frame(&frame).expect("writes");
+ }
+ encoder.finish().expect("finishes");
+
+ let strip = filmstrip(&path.to_string_lossy(), 4, 32).expect("strips");
+ assert_eq!((strip.width(), strip.height()), (4 * 64, 32));
+ // Later slices come from later in the file: the red ramps up.
+ let first = strip.pixel(32, 16).expect("in bounds")[0];
+ let last = strip.pixel(3 * 64 + 32, 16).expect("in bounds")[0];
+ assert!(
+ last > first,
+ "strip is not in time order: {first} then {last}"
+ );
+
+ let poster = still_at(&path.to_string_lossy(), 1.0, 32).expect("still");
+ assert_eq!(poster.width(), 32);
+ let _ = std::fs::remove_file(&path);
+ }
+}
diff --git a/engine/crates/concat-host/src/playback.rs b/engine/crates/concat-host/src/playback.rs
new file mode 100644
index 00000000..3bced280
--- /dev/null
+++ b/engine/crates/concat-host/src/playback.rs
@@ -0,0 +1,1086 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Engine-owned audio playback.
+//!
+//! The engine decodes, mixes and clocks all audible clips itself, and the
+//! window is a controller that follows.
+//!
+//! The shape of it:
+//!
+//! - Every audible clip's span is decoded **once** straight to raw PCM -
+//! the clip's filter chain and speed baked in with the same filters the
+//! exporter uses, so the preview and the export cannot disagree. The
+//! PCM lands as a file in the project's cache and is memory-mapped, so
+//! memory stays bounded however long the material is, and a reopened
+//! project pays nothing.
+//!
+//! - One cpal output stream mixes the mapped clips sample by sample, with
+//! volume and fades applied at mix time. Gain-only edits therefore never
+//! re-decode, and what a fade sounds like has exactly one definition: this
+//! file's `gain_at`.
+//!
+//! - The playback clock is the audio device's own sample counter. The window
+//! reads [`Playback::position`] and interpolates; there is no second clock
+//! to drift against.
+//!
+//! - The stream is *supervised*: pulled headphones, a changed default
+//! device, or a device that was missing at launch all rebuild the stream
+//! rather than leaving the session silent until restart. The supervisor
+//! re-seeds the new stream from the shared clock and the last clip set,
+//! so a rebuild sounds like a hiccup, not a reset.
+//!
+//! The mix callback owns its state and receives changes as messages; the
+//! only lock it touches is an uncontended try-lock on the message channel
+//! (contended solely during a rebuild, when the callback is being replaced
+//! anyway). Decodes run on a small worker pool, newest request first - a
+//! scrubbed trim queues many spans and the one under the playhead matters
+//! most - and the clip set is re-sent to the callback as each one lands.
+//!
+//! Both caches are bounded. In memory, mappings not referenced by the
+//! current clip set are dropped beyond a cap, oldest use first. On disk,
+//! `cache/audio` keeps a byte budget: least-recently-modified files beyond
+//! it are removed, never one the current clip set is using.
+
+use std::collections::{HashMap, HashSet, VecDeque};
+use std::path::{Path, PathBuf};
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::sync::{Arc, Condvar, Mutex, mpsc};
+
+use serde::Deserialize;
+
+/// Every clip is decoded to this rate; the callback resamples to the device.
+const PCM_RATE: u32 = 48_000;
+
+/// Decoded spans kept mapped beyond the current clip set. Enough that
+/// undo/redo across a few edits never re-decodes; small enough that a long
+/// session cannot accumulate hundreds of mappings.
+const MEMORY_CACHE_CAP: usize = 64;
+
+/// Byte budget for `cache/audio` on disk. Stereo 16-bit 48k is ~11 MB a
+/// minute, so this keeps hours of decoded material before anything is
+/// evicted.
+const DISK_CACHE_BUDGET: u64 = 2 * 1024 * 1024 * 1024;
+
+/// Concurrent FFmpeg decode processes. Two keeps a dual-span edit prompt
+/// without letting a scrubbed trim fan out a dozen full-file decodes.
+const DECODE_WORKERS: usize = 2;
+
+/// One audible clip, as the window describes it.
+#[derive(Clone, Debug, Deserialize)]
+#[serde(rename_all = "camelCase")]
+pub struct ClipSpec {
+ /// The media file.
+ pub path: String,
+ /// Timeline seconds.
+ pub start: f64,
+ /// Timeline seconds the clip occupies.
+ pub duration: f64,
+ /// Seconds into the source file.
+ pub source_start: f64,
+ /// Linear gain.
+ pub volume: f32,
+ /// Fade-in length in seconds.
+ pub fade_in: f64,
+ /// Fade-out length in seconds.
+ pub fade_out: f64,
+ /// Playback speed; 1.0 is the source's own.
+ pub speed: f64,
+ /// Whether a speed change keeps the pitch.
+ pub preserve_pitch: bool,
+ /// FFmpeg audio filter chain, or empty.
+ #[serde(default)]
+ pub chain: String,
+}
+
+/// Where playback tells the window what it cannot ask: the clock, and the
+/// failures a user would otherwise experience as unexplained silence.
+pub trait PlaybackEvents: Send + Sync + 'static {
+ /// Timeline position in seconds, at ~30Hz while playing.
+ fn position(&self, seconds: f64);
+ /// A failure the user should see.
+ fn error(&self, message: String);
+}
+
+/// Decoded audio: 16-bit little-endian stereo at [`PCM_RATE`] inside a WAV
+/// container, memory-mapped from the cache file so the OS decides what stays
+/// resident.
+///
+/// WAV rather than raw PCM so the cache files are ordinary audio files -
+/// openable in anything, and self-describing if the format ever changes. The
+/// header costs a one-time chunk walk here; the samples inside are the
+/// identical bytes.
+struct Pcm {
+ map: memmap2::Mmap,
+ /// Byte offset of the first sample - just past the `data` chunk header.
+ start: usize,
+ frames: u64,
+}
+
+impl Pcm {
+ fn open(path: &Path) -> Result<Pcm, String> {
+ let file = std::fs::File::open(path)
+ .map_err(|error| format!("could not open {}: {error}", path.display()))?;
+ // SAFETY: the cache file is written by our own decode, renamed into
+ // place only when complete, and never modified afterwards; the sweep
+ // deletes rather than truncates, and a deleted file's mapping stays
+ // valid. Reads are bounds-checked against the mapped length.
+ let map = unsafe { memmap2::Mmap::map(&file) }
+ .map_err(|error| format!("could not map {}: {error}", path.display()))?;
+ let (start, length) = wav_data_range(&map)
+ .ok_or_else(|| format!("{} is not the wav this cache writes", path.display()))?;
+ let frames = (length / 4) as u64;
+ Ok(Pcm { map, start, frames })
+ }
+
+ /// One sample as a float in [-1, 1]. Out of range reads are silence.
+ fn sample(&self, frame: u64, channel: usize) -> f32 {
+ if frame >= self.frames {
+ return 0.0;
+ }
+ let at = self.start + (frame as usize * 2 + channel) * 2;
+ let raw = i16::from_le_bytes([self.map[at], self.map[at + 1]]);
+ f32::from(raw) / 32768.0
+ }
+}
+
+/// Finds the samples inside a WAV file: byte offset of the `data` chunk's
+/// payload and its usable length.
+///
+/// A plain chunk walk, not a format library: the file was written by our own
+/// [`WavWriter`], so the only variability is which bookkeeping chunks precede
+/// `data`. A `u32::MAX` size - what a writer that could not seek back would
+/// leave - means the payload is simply the rest of the file.
+fn wav_data_range(bytes: &[u8]) -> Option<(usize, usize)> {
+ if bytes.len() < 12 || &bytes[0..4] != b"RIFF" || &bytes[8..12] != b"WAVE" {
+ return None;
+ }
+
+ let mut at = 12;
+ while at + 8 <= bytes.len() {
+ let id = &bytes[at..at + 4];
+ let size = u32::from_le_bytes([bytes[at + 4], bytes[at + 5], bytes[at + 6], bytes[at + 7]]);
+ let payload = at + 8;
+
+ if id == b"data" {
+ let available = bytes.len() - payload;
+ let length = if size == u32::MAX {
+ available
+ } else {
+ (size as usize).min(available)
+ };
+ return Some((payload, length));
+ }
+
+ // A placeholder size on any other chunk means the walk cannot
+ // continue - refuse rather than misread samples as headers.
+ if size == u32::MAX {
+ return None;
+ }
+ // Chunks are word-aligned; an odd size is followed by a pad byte.
+ at = payload + size as usize + (size as usize & 1);
+ }
+ None
+}
+
+/// A clip the mix callback can play: spec fields it needs, plus the audio.
+/// Clone is cheap - the samples are behind an `Arc` - and the supervisor
+/// uses it to seed a rebuilt stream with the set the old one was playing.
+#[derive(Clone)]
+struct ActiveClip {
+ start: f64,
+ duration: f64,
+ volume: f32,
+ fade_in: f64,
+ fade_out: f64,
+ pcm: Arc<Pcm>,
+}
+
+/// The one definition of a clip's gain envelope.
+fn gain_at(clip: &ActiveClip, local: f64) -> f32 {
+ if local < 0.0 || local > clip.duration {
+ return 0.0;
+ }
+ let mut gain = f64::from(clip.volume);
+ if clip.fade_in > 0.0 {
+ gain *= (local / clip.fade_in).min(1.0);
+ }
+ if clip.fade_out > 0.0 {
+ gain *= ((clip.duration - local) / clip.fade_out).min(1.0);
+ }
+ gain.max(0.0) as f32
+}
+
+enum Msg {
+ SetClips(Vec<ActiveClip>),
+ Play(f64),
+ Pause,
+ Seek(f64),
+}
+
+struct Shared {
+ /// Timeline position in microseconds, written by the mix callback.
+ position_micros: AtomicU64,
+ playing: AtomicBool,
+}
+
+/// One decode waiting for a worker.
+struct DecodeJob {
+ spec: ClipSpec,
+ project: PathBuf,
+ key: String,
+}
+
+/// The decode queue: newest job first, because a scrub over a trim handle
+/// enqueues a span per pause and the latest is the one under the playhead.
+struct DecodeQueue {
+ jobs: Mutex<VecDeque<DecodeJob>>,
+ available: Condvar,
+}
+
+/// A mapped span plus when it was last part of the clip set, for eviction.
+struct CacheEntry {
+ pcm: Arc<Pcm>,
+ last_used: u64,
+}
+
+/// The audio engine: decode workers, the supervised output stream, and the
+/// clock. One per app, for the app's lifetime.
+pub struct Playback {
+ tx: mpsc::Sender<Msg>,
+ shared: Arc<Shared>,
+ /// For surfacing failures; a mute app must never be a mystery.
+ events: Arc<dyn PlaybackEvents>,
+ /// Decoded clips by decode key. Bounded: see [`MEMORY_CACHE_CAP`].
+ cache: Mutex<HashMap<String, CacheEntry>>,
+ /// Monotonic use counter feeding `CacheEntry::last_used`.
+ tick: AtomicU64,
+ decoding: Mutex<HashSet<String>>,
+ /// What the timeline currently wants audible.
+ specs: Mutex<Vec<ClipSpec>>,
+ /// The project whose cache the disk GC sweeps; set by `set_clips`.
+ project: Mutex<Option<PathBuf>>,
+ /// True while a disk sweep runs, so sweeps never pile up.
+ sweeping: AtomicBool,
+ queue: Arc<DecodeQueue>,
+ /// The set most recently sent to the callback, for re-seeding a rebuilt
+ /// stream. The supervisor reads it; `resync` writes it.
+ last_active: Arc<Mutex<Vec<ActiveClip>>>,
+}
+
+/// A playback failure the user would otherwise experience as unexplained
+/// silence. Logged for the console, reported for the window's toast - the
+/// difference between "audio is broken" and a bug report that names a file.
+fn report(events: &dyn PlaybackEvents, message: String) {
+ eprintln!("concat: {message}");
+ events.error(message);
+}
+
+impl Playback {
+ /// Starts the audio threads: the supervised output stream, the clock
+ /// publisher and the decode workers.
+ pub fn start(events: Arc<dyn PlaybackEvents>) -> Arc<Playback> {
+ let (tx, rx) = mpsc::channel::<Msg>();
+ let shared = Arc::new(Shared {
+ position_micros: AtomicU64::new(0),
+ playing: AtomicBool::new(false),
+ });
+ let last_active: Arc<Mutex<Vec<ActiveClip>>> = Arc::new(Mutex::new(Vec::new()));
+
+ {
+ let shared = Arc::clone(&shared);
+ let events = Arc::clone(&events);
+ let last_active = Arc::clone(&last_active);
+ std::thread::Builder::new()
+ .name("audio-output".into())
+ .spawn(move || supervise_stream(rx, shared, events, last_active))
+ .expect("could not spawn the audio thread");
+ }
+
+ // Position events at ~30Hz while playing. The window interpolates
+ // between them, so this cadence bounds correction error, not
+ // smoothness.
+ {
+ let shared = Arc::clone(&shared);
+ let events = Arc::clone(&events);
+ std::thread::Builder::new()
+ .name("transport-events".into())
+ .spawn(move || {
+ loop {
+ std::thread::sleep(std::time::Duration::from_millis(33));
+ if shared.playing.load(Ordering::Relaxed) {
+ let position =
+ shared.position_micros.load(Ordering::Relaxed) as f64 / 1_000_000.0;
+ events.position(position);
+ }
+ }
+ })
+ .expect("could not spawn the transport event thread");
+ }
+
+ let playback = Arc::new(Playback {
+ tx,
+ shared,
+ events,
+ cache: Mutex::new(HashMap::new()),
+ tick: AtomicU64::new(0),
+ decoding: Mutex::new(HashSet::new()),
+ specs: Mutex::new(Vec::new()),
+ project: Mutex::new(None),
+ sweeping: AtomicBool::new(false),
+ queue: Arc::new(DecodeQueue {
+ jobs: Mutex::new(VecDeque::new()),
+ available: Condvar::new(),
+ }),
+ last_active,
+ });
+
+ for index in 0..DECODE_WORKERS {
+ let this = Arc::clone(&playback);
+ std::thread::Builder::new()
+ .name(format!("audio-decode-{index}"))
+ .spawn(move || decode_worker(&this))
+ .expect("could not spawn a decode worker");
+ }
+
+ playback
+ }
+
+ /// Starts playing from `position` seconds.
+ pub fn play(&self, position: f64) {
+ self.shared.playing.store(true, Ordering::Relaxed);
+ self.shared
+ .position_micros
+ .store((position.max(0.0) * 1_000_000.0) as u64, Ordering::Relaxed);
+ let _ = self.tx.send(Msg::Play(position.max(0.0)));
+ }
+
+ /// Stops playing; the position holds.
+ pub fn pause(&self) {
+ self.shared.playing.store(false, Ordering::Relaxed);
+ let _ = self.tx.send(Msg::Pause);
+ }
+
+ /// Moves the playhead, playing or paused.
+ pub fn seek(&self, position: f64) {
+ let clamped = position.max(0.0);
+ // The callback owns the clock while playing, but a paused callback
+ // returns before storing - so a paused seek must land the shared
+ // position itself, or the atomic keeps serving the pre-seek time.
+ self.shared
+ .position_micros
+ .store((clamped * 1_000_000.0) as u64, Ordering::Relaxed);
+ let _ = self.tx.send(Msg::Seek(clamped));
+ }
+
+ /// The clock: timeline seconds, from the device's own sample counter.
+ pub fn position(&self) -> f64 {
+ self.shared.position_micros.load(Ordering::Relaxed) as f64 / 1_000_000.0
+ }
+
+ /// Whether the transport is rolling.
+ pub fn is_playing(&self) -> bool {
+ self.shared.playing.load(Ordering::Relaxed)
+ }
+
+ /// Replaces the audible clip set.
+ ///
+ /// Anything already decoded plays immediately; anything new joins the
+ /// decode queue and joins the mix when it lands. `project` is the
+ /// project folder, whose cache holds the PCM files.
+ pub fn set_clips(self: &Arc<Self>, project: PathBuf, specs: Vec<ClipSpec>) {
+ *self.specs.lock().unwrap() = specs.clone();
+ *self.project.lock().unwrap() = Some(project.clone());
+
+ for spec in specs {
+ let key = decode_key(&spec);
+ if self.cache.lock().unwrap().contains_key(&key) {
+ continue;
+ }
+ if !self.decoding.lock().unwrap().insert(key.clone()) {
+ continue;
+ }
+ let mut jobs = self.queue.jobs.lock().unwrap();
+ jobs.push_back(DecodeJob {
+ spec,
+ project: project.clone(),
+ key,
+ });
+ self.queue.available.notify_one();
+ }
+
+ self.resync();
+ self.evict_memory();
+ self.sweep_disk(project);
+ }
+
+ /// Sends the mix callback everything currently wanted *and* decoded, and
+ /// remembers the set so a rebuilt stream can be seeded with it.
+ fn resync(&self) {
+ let specs = self.specs.lock().unwrap();
+ let now = self.tick.fetch_add(1, Ordering::Relaxed) + 1;
+ let mut cache = self.cache.lock().unwrap();
+ let active: Vec<ActiveClip> = specs
+ .iter()
+ .filter_map(|spec| {
+ let entry = cache.get_mut(&decode_key(spec))?;
+ entry.last_used = now;
+ Some(ActiveClip {
+ start: spec.start,
+ duration: spec.duration,
+ volume: spec.volume,
+ fade_in: spec.fade_in,
+ fade_out: spec.fade_out,
+ pcm: Arc::clone(&entry.pcm),
+ })
+ })
+ .collect();
+ drop(cache);
+ *self.last_active.lock().unwrap() = active.clone();
+ let _ = self.tx.send(Msg::SetClips(active));
+ }
+
+ /// Drops mapped spans beyond [`MEMORY_CACHE_CAP`], oldest use first,
+ /// never one the current clip set references. The mapping is address
+ /// space and page cache, not copies - but a long session accumulates a
+ /// mapping per edit of every trimmed span, and those add up.
+ fn evict_memory(&self) {
+ let live: HashSet<String> = self.specs.lock().unwrap().iter().map(decode_key).collect();
+ let mut cache = self.cache.lock().unwrap();
+ while cache.len() > MEMORY_CACHE_CAP {
+ let doomed = cache
+ .iter()
+ .filter(|(key, _)| !live.contains(*key))
+ .min_by_key(|(_, entry)| entry.last_used)
+ .map(|(key, _)| key.clone());
+ match doomed {
+ Some(key) => cache.remove(&key),
+ // Everything over the cap is live: a genuinely enormous clip
+ // set. Keeping it is correct; evicting live spans would just
+ // re-decode them on the next resync.
+ None => break,
+ };
+ }
+ }
+
+ /// Sweeps `cache/audio` down to [`DISK_CACHE_BUDGET`] on a worker
+ /// thread: least-recently-modified first, never a file the current clip
+ /// set is using. Old `.decoding` leftovers from crashed runs go too.
+ fn sweep_disk(self: &Arc<Self>, project: PathBuf) {
+ if self.sweeping.swap(true, Ordering::Relaxed) {
+ return;
+ }
+ let this = Arc::clone(self);
+ std::thread::Builder::new()
+ .name("audio-cache-sweep".into())
+ .spawn(move || {
+ let live: HashSet<String> =
+ this.specs.lock().unwrap().iter().map(decode_key).collect();
+ let directory = project.join("cache").join("audio");
+ if let Some(doomed) = sweep_plan(&directory, &live, DISK_CACHE_BUDGET) {
+ for path in doomed {
+ let _ = std::fs::remove_file(path);
+ }
+ }
+ this.sweeping.store(false, Ordering::Relaxed);
+ })
+ .expect("could not spawn the cache sweep thread");
+ }
+}
+
+/// One decode worker: takes the newest queued job, skips jobs whose span the
+/// timeline no longer wants, decodes, lands the result, resyncs.
+fn decode_worker(playback: &Arc<Playback>) {
+ loop {
+ let job = {
+ let mut jobs = playback.queue.jobs.lock().unwrap();
+ loop {
+ // Newest first: pop from the back where set_clips pushes.
+ match jobs.pop_back() {
+ Some(job) => break job,
+ None => jobs = playback.queue.available.wait(jobs).unwrap(),
+ }
+ }
+ };
+
+ // A span queued by an edit that has since been edited again is dead
+ // weight; skip it before paying for FFmpeg.
+ let wanted = playback
+ .specs
+ .lock()
+ .unwrap()
+ .iter()
+ .any(|spec| decode_key(spec) == job.key);
+ if !wanted {
+ playback.decoding.lock().unwrap().remove(&job.key);
+ continue;
+ }
+
+ match decode(&job.spec, &job.project, &job.key) {
+ Ok(pcm) => {
+ let now = playback.tick.fetch_add(1, Ordering::Relaxed) + 1;
+ playback.cache.lock().unwrap().insert(
+ job.key.clone(),
+ CacheEntry {
+ pcm: Arc::new(pcm),
+ last_used: now,
+ },
+ );
+ }
+ Err(error) => report(
+ playback.events.as_ref(),
+ format!("audio decode failed for {}: {error}", job.spec.path),
+ ),
+ }
+ playback.decoding.lock().unwrap().remove(&job.key);
+ playback.resync();
+ }
+}
+
+/// Which files a sweep of `directory` should delete to fit `budget` bytes:
+/// least-recently-modified first, never a `.wav` whose key is in `live`,
+/// plus any `.decoding` leftover older than a day. Returns `None` when the
+/// directory cannot be read (not created yet - nothing to sweep).
+fn sweep_plan(directory: &Path, live: &HashSet<String>, budget: u64) -> Option<Vec<PathBuf>> {
+ let entries = std::fs::read_dir(directory).ok()?;
+ let now = std::time::SystemTime::now();
+
+ let mut doomed: Vec<PathBuf> = Vec::new();
+ let mut candidates: Vec<(std::time::SystemTime, u64, PathBuf)> = Vec::new();
+ let mut total: u64 = 0;
+
+ for entry in entries.flatten() {
+ let path = entry.path();
+ let name = entry.file_name().to_string_lossy().into_owned();
+ let Ok(meta) = entry.metadata() else { continue };
+
+ // A `.decoding` file is a crashed or torn run once it is old enough
+ // that no live decode can still be writing it.
+ if name.ends_with(".decoding") {
+ let stale = meta
+ .modified()
+ .ok()
+ .and_then(|modified| now.duration_since(modified).ok())
+ .is_some_and(|age| age.as_secs() > 24 * 3600);
+ if stale {
+ doomed.push(path);
+ }
+ continue;
+ }
+
+ let Some(key) = name.strip_suffix(".wav") else {
+ continue;
+ };
+ total += meta.len();
+ if live.contains(key) {
+ continue;
+ }
+ let modified = meta.modified().unwrap_or(now);
+ candidates.push((modified, meta.len(), path));
+ }
+
+ if total > budget {
+ candidates.sort_by_key(|(modified, ..)| *modified);
+ for (_, size, path) in candidates {
+ if total <= budget {
+ break;
+ }
+ total = total.saturating_sub(size);
+ doomed.push(path);
+ }
+ }
+ Some(doomed)
+}
+
+/// Identity of one decoded span: everything that changes the samples.
+/// Volume, fades and timeline position are deliberately absent - they apply
+/// at mix time.
+///
+/// FNV-1a rather than the standard library's hasher, because these keys name
+/// files that outlive the process: `DefaultHasher` is free to change between
+/// Rust releases, and a changed hash would silently orphan every project's
+/// audio cache on a toolchain upgrade.
+fn decode_key(spec: &ClipSpec) -> String {
+ let mut hash: u64 = 0xcbf2_9ce4_8422_2325;
+ let mut eat = |bytes: &[u8]| {
+ for &byte in bytes {
+ hash ^= u64::from(byte);
+ hash = hash.wrapping_mul(0x0000_0100_0000_01b3);
+ }
+ };
+ eat(spec.path.as_bytes());
+ eat(&spec.source_start.to_bits().to_le_bytes());
+ eat(&spec.duration.to_bits().to_le_bytes());
+ eat(&spec.speed.to_bits().to_le_bytes());
+ eat(&[u8::from(spec.preserve_pitch)]);
+ eat(spec.chain.as_bytes());
+ format!("{hash:016x}")
+}
+
+/// Decodes one clip span to the cache and maps it.
+///
+/// The decode mirrors the exporter's treatment of filters and speed, so
+/// what the preview mixes is what the export renders. A file already in the
+/// cache is reused as is - that is the point of the cache.
+fn decode(spec: &ClipSpec, project: &Path, key: &str) -> Result<Pcm, String> {
+ use concat_media::{AudioDecoder, AudioOptions, SampleFormat};
+
+ concat_media::audio::validate_chain(&spec.chain).map_err(|error| error.to_string())?;
+
+ let directory = project.join("cache").join("audio");
+ let destination = directory.join(format!("{key}.wav"));
+ if destination.is_file() {
+ return Pcm::open(&destination);
+ }
+ std::fs::create_dir_all(&directory)
+ .map_err(|error| format!("could not create {}: {error}", directory.display()))?;
+
+ // Speed first, then the clip's own filters - the same order the export
+ // graph applies, because these are one definition, not two. The engine
+ // owns what speed *means*; this function only decodes.
+ let mut filters: Vec<String> =
+ concat_media::audio::speed_filters(spec.speed, spec.preserve_pitch);
+ if !spec.chain.is_empty() {
+ filters.push(spec.chain.clone());
+ }
+
+ // A sped-up clip reads further into the file than it occupies on the
+ // timeline: the source window is `duration * speed`, exactly the window
+ // the exporter trims.
+ let window = spec.duration * concat_media::audio::clamp_speed(spec.speed);
+
+ let mut decoder = AudioDecoder::open(
+ &spec.path,
+ &AudioOptions {
+ start: Some(spec.source_start),
+ duration: Some(window),
+ filters,
+ rate: PCM_RATE,
+ channels: 2,
+ format: SampleFormat::I16,
+ },
+ )
+ .map_err(|error| error.to_string())?;
+
+ let temporary = directory.join(format!("{key}.wav.decoding"));
+ let result = (|| -> Result<(), String> {
+ let mut file = std::io::BufWriter::new(
+ std::fs::File::create(&temporary)
+ .map_err(|error| format!("could not create {}: {error}", temporary.display()))?,
+ );
+ let mut writer = WavWriter::start(&mut file)?;
+ while let Some(frame) = decoder.next_frame().map_err(|error| error.to_string())? {
+ writer.write(&mut file, &decoder.bytes_of(&frame))?;
+ }
+ writer.finish(&mut file)
+ })();
+ if let Err(error) = result {
+ let _ = std::fs::remove_file(&temporary);
+ return Err(error);
+ }
+
+ std::fs::rename(&temporary, &destination)
+ .map_err(|error| format!("could not commit {}: {error}", destination.display()))?;
+ Pcm::open(&destination)
+}
+
+/// Writes the RIFF/WAVE framing around 16-bit stereo samples at
+/// [`PCM_RATE`]: a header with placeholder sizes first, patched once the
+/// sample count is known.
+struct WavWriter {
+ data_bytes: u64,
+}
+
+impl WavWriter {
+ fn start<W: std::io::Write + std::io::Seek>(out: &mut W) -> Result<Self, String> {
+ let io = |error: std::io::Error| format!("could not write the audio cache: {error}");
+ let channels: u16 = 2;
+ let bits: u16 = 16;
+ let block = channels * bits / 8;
+ out.write_all(b"RIFF").map_err(io)?;
+ out.write_all(&0u32.to_le_bytes()).map_err(io)?;
+ out.write_all(b"WAVE").map_err(io)?;
+ out.write_all(b"fmt ").map_err(io)?;
+ out.write_all(&16u32.to_le_bytes()).map_err(io)?;
+ out.write_all(&1u16.to_le_bytes()).map_err(io)?; // PCM
+ out.write_all(&channels.to_le_bytes()).map_err(io)?;
+ out.write_all(&PCM_RATE.to_le_bytes()).map_err(io)?;
+ out.write_all(&(PCM_RATE * u32::from(block)).to_le_bytes())
+ .map_err(io)?;
+ out.write_all(&block.to_le_bytes()).map_err(io)?;
+ out.write_all(&bits.to_le_bytes()).map_err(io)?;
+ out.write_all(b"data").map_err(io)?;
+ out.write_all(&0u32.to_le_bytes()).map_err(io)?;
+ Ok(Self { data_bytes: 0 })
+ }
+
+ fn write<W: std::io::Write>(&mut self, out: &mut W, bytes: &[u8]) -> Result<(), String> {
+ out.write_all(bytes)
+ .map_err(|error| format!("could not write the audio cache: {error}"))?;
+ self.data_bytes += bytes.len() as u64;
+ Ok(())
+ }
+
+ fn finish<W: std::io::Write + std::io::Seek>(self, out: &mut W) -> Result<(), String> {
+ use std::io::SeekFrom;
+ let io = |error: std::io::Error| format!("could not finish the audio cache: {error}");
+ let data = u32::try_from(self.data_bytes).unwrap_or(u32::MAX);
+ out.seek(SeekFrom::Start(4)).map_err(io)?;
+ out.write_all(&data.saturating_add(36).to_le_bytes())
+ .map_err(io)?;
+ out.seek(SeekFrom::Start(40)).map_err(io)?;
+ out.write_all(&data.to_le_bytes()).map_err(io)?;
+ out.flush().map_err(io)
+ }
+}
+
+/// Owns the output stream for the life of the app, rebuilding it whenever
+/// it dies or the default device changes.
+///
+/// Setup failure is a retry with backoff, not a permanent return: a machine
+/// that boots Concat before its audio device is ready, or loses its only
+/// device mid-session, gets sound back when the device does. A rebuilt
+/// stream is seeded from the shared clock and the last clip set, so playback
+/// carries straight on.
+fn supervise_stream(
+ rx: mpsc::Receiver<Msg>,
+ shared: Arc<Shared>,
+ events: Arc<dyn PlaybackEvents>,
+ last_active: Arc<Mutex<Vec<ActiveClip>>>,
+) {
+ use cpal::traits::{DeviceTrait, HostTrait};
+
+ // The callback drains this through a try-lock: uncontended in steady
+ // state (the supervisor only takes it between streams), and a missed
+ // drain on a contended tick is caught on the next callback.
+ let rx = Arc::new(Mutex::new(rx));
+ let mut backoff = std::time::Duration::from_secs(1);
+ let mut reported_missing = false;
+
+ loop {
+ let host = cpal::default_host();
+ let built = host
+ .default_output_device()
+ .ok_or_else(|| "no audio output device".to_owned())
+ .and_then(|device| {
+ let name = device.name().unwrap_or_default();
+ build_stream(&device, &rx, &shared, &events, &last_active).map(|s| (s, name))
+ });
+
+ match built {
+ Ok((stream, device_name)) => {
+ backoff = std::time::Duration::from_secs(1);
+ reported_missing = false;
+
+ // Watch for the two reasons to rebuild: the stream reported
+ // an error, or the default device is no longer the one the
+ // stream was built on (headphones unplugged, output switched
+ // in system settings).
+ let failed = stream.failed;
+ let _stream = stream.stream;
+ loop {
+ std::thread::sleep(std::time::Duration::from_millis(1000));
+ if failed.load(Ordering::Relaxed) {
+ report(
+ events.as_ref(),
+ "audio stream failed; rebuilding".to_owned(),
+ );
+ break;
+ }
+ let current = cpal::default_host()
+ .default_output_device()
+ .and_then(|d| d.name().ok());
+ if let Some(current) = current
+ && current != device_name
+ {
+ // Informational only - switching outputs is a
+ // normal act, not an error toast.
+ eprintln!("concat: audio device changed to {current}; rebuilding");
+ break;
+ }
+ }
+ // `_stream` drops here, releasing the device before rebuild.
+ }
+ Err(error) => {
+ // Once, not every retry: a machine with no sound card would
+ // otherwise toast every backoff tick forever.
+ if !reported_missing {
+ report(
+ events.as_ref(),
+ format!("audio output unavailable: {error}; will keep trying"),
+ );
+ reported_missing = true;
+ }
+ std::thread::sleep(backoff);
+ backoff = (backoff * 2).min(std::time::Duration::from_secs(30));
+ }
+ }
+ }
+}
+
+/// A built, playing stream plus its error flag.
+struct BuiltStream {
+ stream: cpal::Stream,
+ failed: Arc<AtomicBool>,
+}
+
+/// Builds and starts one output stream on `device`, seeded with the current
+/// clip set and clock so a rebuild resumes where the last stream stopped.
+fn build_stream(
+ device: &cpal::Device,
+ rx: &Arc<Mutex<mpsc::Receiver<Msg>>>,
+ shared: &Arc<Shared>,
+ events: &Arc<dyn PlaybackEvents>,
+ last_active: &Arc<Mutex<Vec<ActiveClip>>>,
+) -> Result<BuiltStream, String> {
+ use cpal::traits::DeviceTrait;
+
+ let config = device
+ .default_output_config()
+ .map_err(|error| format!("no default output config: {error}"))?;
+ let sample_format = config.sample_format();
+ let config: cpal::StreamConfig = config.into();
+
+ match sample_format {
+ cpal::SampleFormat::F32 => {
+ stream_for::<f32>(device, &config, rx, shared, events, last_active)
+ }
+ cpal::SampleFormat::I16 => {
+ stream_for::<i16>(device, &config, rx, shared, events, last_active)
+ }
+ cpal::SampleFormat::U16 => {
+ stream_for::<u16>(device, &config, rx, shared, events, last_active)
+ }
+ other => Err(format!("unsupported sample format {other:?}")),
+ }
+}
+
+fn stream_for<T>(
+ device: &cpal::Device,
+ config: &cpal::StreamConfig,
+ rx: &Arc<Mutex<mpsc::Receiver<Msg>>>,
+ shared: &Arc<Shared>,
+ events: &Arc<dyn PlaybackEvents>,
+ last_active: &Arc<Mutex<Vec<ActiveClip>>>,
+) -> Result<BuiltStream, String>
+where
+ T: cpal::SizedSample + cpal::FromSample<f32>,
+{
+ use cpal::traits::{DeviceTrait, StreamTrait};
+
+ let channels = config.channels as usize;
+ let step = 1.0 / f64::from(config.sample_rate.0);
+
+ // Seed from the world as it is, not from zero: a stream rebuilt mid-
+ // playback resumes the same clips at the shared clock's position.
+ let mut clips: Vec<ActiveClip> = last_active.lock().unwrap().clone();
+ let mut playing = shared.playing.load(Ordering::Relaxed);
+ let mut position = shared.position_micros.load(Ordering::Relaxed) as f64 / 1_000_000.0;
+
+ let failed = Arc::new(AtomicBool::new(false));
+ let rx = Arc::clone(rx);
+ let shared = Arc::clone(shared);
+
+ let stream = device
+ .build_output_stream(
+ config,
+ move |data: &mut [T], _| {
+ let mut sought = false;
+ // try_lock, not lock: the audio callback must never block.
+ // Contention only exists while the supervisor swaps streams,
+ // and then this callback is on its way out anyway.
+ if let Ok(receiver) = rx.try_lock() {
+ while let Ok(message) = receiver.try_recv() {
+ match message {
+ Msg::SetClips(next) => clips = next,
+ Msg::Play(at) => {
+ position = at;
+ playing = true;
+ }
+ Msg::Pause => playing = false,
+ Msg::Seek(at) => {
+ position = at;
+ sought = true;
+ }
+ }
+ }
+ }
+
+ if !playing {
+ // A paused seek must reach the shared clock - the playing
+ // store below never runs. Only on an actual seek, so this
+ // silent branch cannot overwrite a fresher play() store
+ // with its own stale idea of the position.
+ if sought {
+ shared
+ .position_micros
+ .store((position * 1_000_000.0) as u64, Ordering::Relaxed);
+ }
+ data.fill(T::from_sample(0.0));
+ return;
+ }
+
+ for frame in data.chunks_mut(channels) {
+ let mut left = 0.0f32;
+ let mut right = 0.0f32;
+
+ for clip in &clips {
+ let local = position - clip.start;
+ if local < 0.0 || local >= clip.duration {
+ continue;
+ }
+ let gain = gain_at(clip, local);
+ if gain <= 0.0 {
+ continue;
+ }
+
+ // Linear interpolation across the 48k source grid -
+ // exact when the device runs at 48k, inaudibly close
+ // when it does not.
+ let read = local * f64::from(PCM_RATE);
+ let index = read.floor() as u64;
+ let fraction = (read - read.floor()) as f32;
+ let pcm = &clip.pcm;
+ left += gain
+ * (pcm.sample(index, 0) * (1.0 - fraction)
+ + pcm.sample(index + 1, 0) * fraction);
+ right += gain
+ * (pcm.sample(index, 1) * (1.0 - fraction)
+ + pcm.sample(index + 1, 1) * fraction);
+ }
+
+ // Hard limit. A mix of boosted clips can exceed full
+ // scale; wrapping would be far worse than flattening.
+ frame[0] = T::from_sample(left.clamp(-1.0, 1.0));
+ if channels > 1 {
+ frame[1] = T::from_sample(right.clamp(-1.0, 1.0));
+ }
+ for extra in frame.iter_mut().skip(2) {
+ *extra = T::from_sample(0.0);
+ }
+
+ position += step;
+ }
+
+ shared
+ .position_micros
+ .store((position * 1_000_000.0) as u64, Ordering::Relaxed);
+ },
+ {
+ let events = Arc::clone(events);
+ let failed = Arc::clone(&failed);
+ move |error| {
+ // The supervisor rebuilds on this flag; the report says
+ // why the sound hiccuped.
+ report(events.as_ref(), format!("audio stream error: {error}"));
+ failed.store(true, Ordering::Relaxed);
+ }
+ },
+ None,
+ )
+ .map_err(|error| format!("could not build the output stream: {error}"))?;
+
+ stream
+ .play()
+ .map_err(|error| format!("could not start the output stream: {error}"))?;
+
+ Ok(BuiltStream { stream, failed })
+}
+
+#[cfg(test)]
+mod tests {
+ use super::{sweep_plan, wav_data_range};
+
+ /// A minimal RIFF/WAVE: fmt chunk, then a data chunk with `samples`.
+ fn wav(data_size: u32, samples: &[u8]) -> Vec<u8> {
+ let mut bytes = Vec::new();
+ bytes.extend_from_slice(b"RIFF");
+ bytes.extend_from_slice(&u32::MAX.to_le_bytes()); // pipe placeholder
+ bytes.extend_from_slice(b"WAVE");
+ bytes.extend_from_slice(b"fmt ");
+ bytes.extend_from_slice(&16u32.to_le_bytes());
+ bytes.extend_from_slice(&[0u8; 16]);
+ bytes.extend_from_slice(b"data");
+ bytes.extend_from_slice(&data_size.to_le_bytes());
+ bytes.extend_from_slice(samples);
+ bytes
+ }
+
+ #[test]
+ fn finds_the_data_chunk_past_the_headers() {
+ let bytes = wav(8, &[1, 2, 3, 4, 5, 6, 7, 8]);
+ let (start, length) = wav_data_range(&bytes).expect("valid wav");
+ assert_eq!(length, 8);
+ assert_eq!(&bytes[start..start + length], &[1, 2, 3, 4, 5, 6, 7, 8]);
+ }
+
+ #[test]
+ fn a_placeholder_data_size_means_the_rest_of_the_file() {
+ // What the wav muxer writes to a pipe: it cannot seek back to patch.
+ let bytes = wav(u32::MAX, &[9, 9, 9, 9]);
+ let (start, length) = wav_data_range(&bytes).expect("valid wav");
+ assert_eq!(length, 4);
+ assert_eq!(start, bytes.len() - 4);
+ }
+
+ #[test]
+ fn a_stated_size_is_clamped_to_the_file() {
+ // A truncated decode must read short, not out of bounds.
+ let bytes = wav(1000, &[1, 2]);
+ assert_eq!(wav_data_range(&bytes).map(|(_, length)| length), Some(2));
+ }
+
+ #[test]
+ fn refuses_files_that_are_not_wav() {
+ assert_eq!(wav_data_range(b""), None);
+ assert_eq!(wav_data_range(b"RIFFxxxxJUNK"), None);
+ assert_eq!(wav_data_range(&[0u8; 64]), None);
+ }
+
+ #[test]
+ fn the_sweep_removes_the_oldest_dead_files_first_and_never_live_ones() {
+ use std::collections::HashSet;
+
+ let directory = std::env::temp_dir().join(format!(
+ "concat-sweep-test-{}-{}",
+ std::process::id(),
+ std::time::SystemTime::now()
+ .duration_since(std::time::UNIX_EPOCH)
+ .map(|elapsed| elapsed.as_nanos())
+ .unwrap_or(0),
+ ));
+ std::fs::create_dir_all(&directory).expect("scratch dir");
+
+ // Three 4-byte caches, written oldest-first so mtimes order them.
+ for name in ["old", "live", "new"] {
+ std::fs::write(directory.join(format!("{name}.wav")), [0u8; 4]).expect("writes");
+ std::thread::sleep(std::time::Duration::from_millis(20));
+ }
+
+ let live: HashSet<String> = ["live".to_owned()].into();
+
+ // Budget for two files: only the oldest dead file goes; the live one
+ // survives despite being older than "new".
+ let doomed = sweep_plan(&directory, &live, 8).expect("plans");
+ let names: Vec<_> = doomed
+ .iter()
+ .filter_map(|path| {
+ path.file_name()
+ .map(|name| name.to_string_lossy().into_owned())
+ })
+ .collect();
+ assert_eq!(names, vec!["old.wav"]);
+
+ // Inside budget: nothing goes.
+ assert_eq!(
+ sweep_plan(&directory, &live, 1024).expect("plans"),
+ Vec::<std::path::PathBuf>::new()
+ );
+
+ // A directory that does not exist yet is nothing to sweep, not a
+ // panic - the first decode has simply not happened.
+ assert!(sweep_plan(&directory.join("missing"), &live, 8).is_none());
+
+ let _ = std::fs::remove_dir_all(&directory);
+ }
+}
diff --git a/engine/crates/concat-host/src/preview.rs b/engine/crates/concat-host/src/preview.rs
new file mode 100644
index 00000000..bd0fb007
--- /dev/null
+++ b/engine/crates/concat-host/src/preview.rs
@@ -0,0 +1,103 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! The paused monitor's true frame.
+//!
+//! One reader pool for the app's lifetime: its whole value is what stays
+//! warm between scrubs. Frames are composited through the pool's mutex, one
+//! at a time, in order, on whatever thread the caller chose - the window
+//! debounces and drops stale results, so a slow decode never wedges anything
+//! but itself.
+
+use std::sync::{Arc, Mutex};
+
+use concat_export::{ExportClip, PreviewFrameRequest};
+use concat_project::DocumentSettings;
+
+/// A frame request: the instant and the size, with the clips coming from
+/// the session that owns them.
+#[derive(Clone, Copy, Debug)]
+pub struct FrameSpec {
+ /// The timeline instant to composite, in seconds.
+ pub time: f64,
+ /// Preview frame width in pixels.
+ pub width: u32,
+ /// Preview frame height in pixels.
+ pub height: u32,
+}
+
+/// The reader pool behind the monitor, shareable across threads.
+#[derive(Clone)]
+pub struct Monitor {
+ pool: Arc<Mutex<concat_media::ReaderPool>>,
+}
+
+impl Default for Monitor {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl Monitor {
+ /// A monitor with the engine's default pool budget.
+ pub fn new() -> Self {
+ Self {
+ pool: Arc::new(Mutex::new(concat_media::ReaderPool::with_defaults())),
+ }
+ }
+
+ fn request(
+ clips: Vec<ExportClip>,
+ settings: &DocumentSettings,
+ spec: FrameSpec,
+ ) -> PreviewFrameRequest {
+ PreviewFrameRequest {
+ time: spec.time,
+ width: spec.width,
+ height: spec.height,
+ rate_num: settings.rate_num,
+ rate_den: settings.rate_den,
+ clips,
+ }
+ }
+
+ /// The engine-composited frame at one instant, as raw RGBA bytes:
+ /// exactly `width * height * 4` of them.
+ pub fn frame(
+ &self,
+ clips: Vec<ExportClip>,
+ settings: &DocumentSettings,
+ spec: FrameSpec,
+ ) -> Result<Vec<u8>, String> {
+ let request = Self::request(clips, settings, spec);
+ let mut pool = self
+ .pool
+ .lock()
+ .map_err(|_| "reader pool poisoned".to_owned())?;
+ concat_export::preview_frame(&mut pool, &request)
+ }
+
+ /// Decode-ahead for the playback stream: warms the pool for the next
+ /// `frames` instants after `spec.time`, so the following [`Monitor::frame`]
+ /// pulls are cache hits instead of decode waits. Clamped, so a confused
+ /// caller cannot park the pool's mutex on a long decode march.
+ pub fn prefetch(
+ &self,
+ clips: Vec<ExportClip>,
+ settings: &DocumentSettings,
+ spec: FrameSpec,
+ frames: u32,
+ ) {
+ let request = Self::request(clips, settings, spec);
+ if let Ok(mut pool) = self.pool.lock() {
+ concat_export::preview_prefetch(&mut pool, &request, frames.min(8));
+ }
+ }
+
+ /// Forgets every cached frame and reader, for when the project closes.
+ pub fn clear(&self) {
+ if let Ok(mut pool) = self.pool.lock() {
+ pool.clear();
+ }
+ }
+}
diff --git a/engine/crates/concat-host/src/projects.rs b/engine/crates/concat-host/src/projects.rs
new file mode 100644
index 00000000..dc4bcdbf
--- /dev/null
+++ b/engine/crates/concat-host/src/projects.rs
@@ -0,0 +1,386 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Project folders on disk, and the list of recently opened ones.
+//!
+//! A project is a directory containing `concat.json`. That file is written the
+//! moment the project is created, so a project is a real thing from the start
+//! rather than a promise the app keeps only if you remember to save.
+//!
+//! The recents list lives in the app's config directory, not in any project.
+//! It describes this machine's history, and copying a project folder to
+//! another machine should not drag one person's recent files along with it.
+
+use std::path::{Path, PathBuf};
+use std::time::{SystemTime, UNIX_EPOCH};
+
+use serde::{Deserialize, Serialize};
+
+const MANIFEST: &str = "concat.json";
+/// The manifest's earlier names, newest first. Projects created before a
+/// rename keep theirs forever: the manifest is read and written under the
+/// name it has, and only brand-new projects get the current one. A rename
+/// must never orphan an edit.
+const LEGACY_MANIFESTS: &[&str] = &["wolfcut.json"];
+
+/// The manifest file this project actually uses: the current name, a legacy
+/// name if that is what exists, the current name for a fresh folder.
+pub fn manifest_path(root: &Path) -> PathBuf {
+ let current = root.join(MANIFEST);
+ if current.is_file() {
+ return current;
+ }
+ for legacy in LEGACY_MANIFESTS {
+ let candidate = root.join(legacy);
+ if candidate.is_file() {
+ return candidate;
+ }
+ }
+ current
+}
+
+/// Whether `root` holds a project: any manifest, current or legacy.
+pub fn is_project(root: &Path) -> bool {
+ manifest_path(root).is_file()
+}
+
+const RECENTS: &str = "recents.json";
+/// Long enough to be useful, short enough that the list stays scannable.
+const MAX_RECENTS: usize = 12;
+
+/// Everything needed to reopen a project.
+#[derive(Serialize, Deserialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct ProjectInfo {
+ /// The project folder.
+ pub path: String,
+ /// The name shown in the title bar and the recents list.
+ pub name: String,
+ /// Output width in pixels.
+ pub width: u32,
+ /// Output height in pixels.
+ pub height: u32,
+ /// Frame rate as an exact fraction; the decimal is never stored.
+ pub rate_num: i64,
+ /// Denominator of the frame rate.
+ pub rate_den: i64,
+ /// Milliseconds since the epoch.
+ pub opened_at: u64,
+}
+
+#[derive(Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+struct Manifest {
+ /// The version of the app that wrote the file, under whichever name the
+ /// app had at the time. Informational: nothing reads it back.
+ #[serde(alias = "wolfcut")]
+ concat: String,
+ name: String,
+ video: Video,
+ /// Everything else in the file is ignored when reading settings for the
+ /// recents list. `flatten` keeps it from being dropped on a rewrite.
+ #[serde(flatten)]
+ #[allow(dead_code)]
+ rest: serde_json::Map<String, serde_json::Value>,
+}
+
+#[derive(Serialize, Deserialize)]
+#[serde(rename_all = "camelCase")]
+struct Video {
+ width: u32,
+ height: u32,
+ rate_num: i64,
+ rate_den: i64,
+}
+
+fn now_millis() -> u64 {
+ SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .map(|elapsed| elapsed.as_millis() as u64)
+ .unwrap_or(0)
+}
+
+/// Creates the project folder and writes its manifest.
+///
+/// Refuses to touch a folder that already holds a manifest. Overwriting
+/// someone's edit because they reused a name is not a recoverable mistake.
+pub fn create(
+ location: &str,
+ name: &str,
+ width: u32,
+ height: u32,
+ rate_num: i64,
+ rate_den: i64,
+) -> Result<ProjectInfo, String> {
+ let root = Path::new(location).join(folder_name(name));
+ let manifest = root.join(MANIFEST);
+
+ // Any name counts as an existing project - a legacy folder must be
+ // just as safe from being clobbered as a new one.
+ if is_project(&root) {
+ return Err(format!(
+ "a Concat project already exists at {}",
+ root.display()
+ ));
+ }
+
+ std::fs::create_dir_all(&root)
+ .map_err(|error| format!("could not create {}: {error}", root.display()))?;
+
+ // Settings only: a fresh project has no edit yet. The full document -
+ // timelines included - is written by the session's save from the first
+ // change.
+ let document = Manifest {
+ concat: env!("CARGO_PKG_VERSION").to_owned(),
+ name: name.to_owned(),
+ video: Video {
+ width,
+ height,
+ rate_num,
+ rate_den,
+ },
+ rest: serde_json::Map::new(),
+ };
+
+ let encoded = serde_json::to_vec_pretty(&document)
+ .map_err(|error| format!("could not encode the manifest: {error}"))?;
+ std::fs::write(&manifest, encoded)
+ .map_err(|error| format!("could not write {}: {error}", manifest.display()))?;
+
+ Ok(ProjectInfo {
+ path: root.to_string_lossy().into_owned(),
+ name: name.to_owned(),
+ width,
+ height,
+ rate_num,
+ rate_den,
+ opened_at: now_millis(),
+ })
+}
+
+/// Writes the whole project document to the project's manifest.
+///
+/// The document is passed through as opaque JSON rather than being mirrored
+/// into Rust types: the engine (`concat-project`) owns the canonical model and
+/// produced this document, and the host has no decisions to make about its
+/// contents beyond writing it safely.
+///
+/// Written to a temporary file and renamed into place, because a save
+/// interrupted halfway is worse than no save at all - a truncated manifest
+/// loses the project, while a failed rename leaves the previous one intact.
+pub fn save(path: &str, document: &serde_json::Value) -> Result<(), String> {
+ let root = PathBuf::from(path);
+ let manifest = manifest_path(&root);
+ let temporary = root.join(format!("{MANIFEST}.saving"));
+
+ std::fs::create_dir_all(&root)
+ .map_err(|error| format!("could not create {}: {error}", root.display()))?;
+
+ let encoded = serde_json::to_vec_pretty(document)
+ .map_err(|error| format!("could not encode the project: {error}"))?;
+
+ std::fs::write(&temporary, encoded)
+ .map_err(|error| format!("could not write {}: {error}", temporary.display()))?;
+
+ std::fs::rename(&temporary, &manifest)
+ .map_err(|error| format!("could not replace {}: {error}", manifest.display()))
+}
+
+/// True when a document carries no edit state at all - the settings-only
+/// manifest `create` writes, in a project closed before its first save.
+/// Such a document is a legitimately empty project. A document that claims
+/// edit state the build then cannot load is a different thing: corrupt.
+pub fn is_settings_only(document: &serde_json::Value) -> bool {
+ document.is_object()
+ && ["timelines", "tracks", "clips", "media"]
+ .iter()
+ .all(|key| document.get(key).is_none())
+}
+
+/// Reads the whole project document back.
+pub fn read_document(path: &str) -> Result<serde_json::Value, String> {
+ let manifest = manifest_path(Path::new(path));
+ let bytes = std::fs::read(&manifest)
+ .map_err(|error| format!("could not read {}: {error}", manifest.display()))?;
+
+ serde_json::from_slice(&bytes)
+ .map_err(|error| format!("{} is not a Concat project: {error}", manifest.display()))
+}
+
+/// Reads an existing project's settings.
+pub fn open(path: &str) -> Result<ProjectInfo, String> {
+ let root = PathBuf::from(path);
+ let manifest = manifest_path(&root);
+
+ let bytes = std::fs::read(&manifest)
+ .map_err(|error| format!("could not read {}: {error}", manifest.display()))?;
+ let document: Manifest = serde_json::from_slice(&bytes)
+ .map_err(|error| format!("{} is not a Concat project: {error}", manifest.display()))?;
+
+ if document.video.rate_den == 0 {
+ return Err(format!("{} has an invalid frame rate", manifest.display()));
+ }
+
+ Ok(ProjectInfo {
+ path: root.to_string_lossy().into_owned(),
+ name: document.name,
+ width: document.video.width,
+ height: document.video.height,
+ rate_num: document.video.rate_num,
+ rate_den: document.video.rate_den,
+ opened_at: now_millis(),
+ })
+}
+
+/// Moves a project to the front of the recents list.
+pub fn remember(config: &Path, project: &ProjectInfo) -> Result<(), String> {
+ let mut entries = read_recents(config);
+ entries.retain(|entry| !same_path(&entry.path, &project.path));
+ entries.insert(0, project.clone());
+ entries.truncate(MAX_RECENTS);
+ write_recents(config, &entries)
+}
+
+/// The recents list, most recent first, with vanished folders dropped.
+///
+/// Filtering on read rather than pruning on write means a project on a drive
+/// that happens to be unplugged comes back when the drive does.
+pub fn list(config: &Path) -> Vec<ProjectInfo> {
+ read_recents(config)
+ .into_iter()
+ .filter(|entry| is_project(Path::new(&entry.path)))
+ .collect()
+}
+
+/// Removes one project from the list. The folder itself is left alone.
+pub fn forget(config: &Path, path: &str) -> Result<(), String> {
+ let mut entries = read_recents(config);
+ entries.retain(|entry| !same_path(&entry.path, path));
+ write_recents(config, &entries)
+}
+
+fn read_recents(config: &Path) -> Vec<ProjectInfo> {
+ // A missing or corrupt list is not an error worth showing anyone; it just
+ // means there is no history yet.
+ std::fs::read(config.join(RECENTS))
+ .ok()
+ .and_then(|bytes| serde_json::from_slice(&bytes).ok())
+ .unwrap_or_default()
+}
+
+fn write_recents(config: &Path, entries: &[ProjectInfo]) -> Result<(), String> {
+ std::fs::create_dir_all(config)
+ .map_err(|error| format!("could not create {}: {error}", config.display()))?;
+
+ let encoded = serde_json::to_vec_pretty(entries)
+ .map_err(|error| format!("could not encode the recents list: {error}"))?;
+ std::fs::write(config.join(RECENTS), encoded)
+ .map_err(|error| format!("could not write the recents list: {error}"))
+}
+
+/// Compares paths case-insensitively, because Windows does.
+fn same_path(left: &str, right: &str) -> bool {
+ left.eq_ignore_ascii_case(right)
+}
+
+/// Turns a project name into something a filesystem will accept. Also used
+/// for template bundle folders and the files inside them.
+pub fn folder_name(name: &str) -> String {
+ let cleaned: String = name
+ .trim()
+ .chars()
+ .map(|character| {
+ if character.is_control() || r#"<>:"/\|?*"#.contains(character) {
+ '-'
+ } else {
+ character
+ }
+ })
+ .collect();
+
+ // Windows refuses names ending in a dot or a space.
+ let trimmed = cleaned.trim_matches(['.', ' ']);
+ if trimmed.is_empty() {
+ "Untitled project".to_owned()
+ } else {
+ trimmed.to_owned()
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn sanitises_names_windows_would_reject() {
+ assert_eq!(folder_name("My: Project?"), "My- Project-");
+ assert_eq!(folder_name(" trailing. "), "trailing");
+ assert_eq!(folder_name("..."), "Untitled project");
+ assert_eq!(folder_name(""), "Untitled project");
+ }
+
+ #[test]
+ fn a_legacy_manifest_keeps_its_name() {
+ for legacy in LEGACY_MANIFESTS {
+ let scratch = std::env::temp_dir().join(format!("concat-legacy-{legacy}-test"));
+ let _ = std::fs::remove_dir_all(&scratch);
+ std::fs::create_dir_all(&scratch).expect("scratch dir");
+ // A fresh folder resolves to the current name...
+ assert!(manifest_path(&scratch).ends_with(MANIFEST));
+ assert!(!is_project(&scratch));
+ // ...a pre-rename project keeps the manifest it has, forever...
+ std::fs::write(scratch.join(legacy), b"{}").expect("writes");
+ assert!(manifest_path(&scratch).ends_with(legacy));
+ assert!(is_project(&scratch));
+ // ...and the current name wins only where it actually exists.
+ std::fs::write(scratch.join(MANIFEST), b"{}").expect("writes");
+ assert!(manifest_path(&scratch).ends_with(MANIFEST));
+ let _ = std::fs::remove_dir_all(&scratch);
+ }
+ }
+
+ #[test]
+ fn a_manifest_under_any_of_its_names_opens() {
+ let scratch = std::env::temp_dir().join("concat-manifest-key-test");
+ let _ = std::fs::remove_dir_all(&scratch);
+ std::fs::create_dir_all(&scratch).expect("scratch dir");
+ for key in ["concat", "wolfcut"] {
+ let document = serde_json::json!({
+ key: "0.2.0",
+ "name": "Old",
+ "video": { "width": 1920, "height": 1080, "rateNum": 30, "rateDen": 1 },
+ });
+ std::fs::write(
+ scratch.join(MANIFEST),
+ serde_json::to_vec(&document).unwrap(),
+ )
+ .expect("writes");
+ let info = open(&scratch.to_string_lossy()).expect("opens");
+ assert_eq!((info.width, info.height, info.rate_num), (1920, 1080, 30));
+ }
+ let _ = std::fs::remove_dir_all(&scratch);
+ }
+
+ #[test]
+ fn a_fresh_manifest_is_settings_only_and_a_saved_edit_is_not() {
+ // Exactly what `create` writes: settings, no edit state.
+ let fresh = serde_json::json!({
+ "concat": "0.2.0",
+ "name": "Untitled project",
+ "video": { "width": 1080, "height": 1920, "rateNum": 30, "rateDen": 1 },
+ });
+ assert!(is_settings_only(&fresh));
+ // Any edit state present means the document must load or fail loudly.
+ assert!(!is_settings_only(&serde_json::json!({ "tracks": [] })));
+ assert!(!is_settings_only(
+ &serde_json::json!({ "timelines": "garbage" })
+ ));
+ assert!(!is_settings_only(&serde_json::json!("nonsense")));
+ }
+
+ #[test]
+ fn paths_compare_case_insensitively() {
+ assert!(same_path("D:\\Work\\Film", "d:\\work\\film"));
+ assert!(!same_path("D:\\Work\\Film", "D:\\Work\\Other"));
+ }
+}
diff --git a/engine/crates/concat-host/src/session.rs b/engine/crates/concat-host/src/session.rs
new file mode 100644
index 00000000..e7de2c54
--- /dev/null
+++ b/engine/crates/concat-host/src/session.rs
@@ -0,0 +1,298 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! The engine-owned editing session.
+//!
+//! Open a project folder and the engine holds the edit: every mutation is a
+//! `concat_project` [`Command`], applied with undo recorded, and the new
+//! state is what the window draws. The window never keeps a model of its
+//! own; it renders the [`Project`] this session hands back.
+//!
+//! Saving reuses `projects::save`'s temp-file-and-rename, so the document on
+//! disk is written by exactly one code path.
+
+use concat_project::{Command, DocumentSettings, Editor, Project};
+use serde::Serialize;
+
+use crate::projects;
+
+/// One open project: its folder, its settings and its undo history.
+pub struct Session {
+ /// The project folder, for saving.
+ path: String,
+ settings: DocumentSettings,
+ editor: Editor,
+}
+
+/// What every mutating call returns: the authoritative state plus history
+/// availability, so undo/redo affordances are never guessing.
+#[derive(Serialize, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct EditorView {
+ /// The whole project, as the engine holds it.
+ pub project: Project,
+ /// Whether there is something to undo.
+ pub can_undo: bool,
+ /// Whether there is something to redo.
+ pub can_redo: bool,
+ /// The settings as the session holds them - the document's own output
+ /// size wins over the manifest's on open.
+ pub settings: SettingsView,
+ /// The id a creating command minted, if any.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub created_id: Option<String>,
+}
+
+/// The session's settings, as the window shows them.
+#[derive(Serialize, Clone)]
+#[serde(rename_all = "camelCase")]
+pub struct SettingsView {
+ /// Project name.
+ pub name: String,
+ /// Output width in pixels.
+ pub width: u32,
+ /// Output height in pixels.
+ pub height: u32,
+ /// Frame rate numerator.
+ pub rate_num: i64,
+ /// Frame rate denominator.
+ pub rate_den: i64,
+}
+
+impl Session {
+ /// Opens a project folder as the editing session.
+ ///
+ /// A folder whose document is missing or unreadable opens as an empty
+ /// project rather than failing, but a *corrupt* document is an error,
+ /// because silently replacing an edit with emptiness is how projects get
+ /// lost. `settings` come from the manifest; the document's own output
+ /// size wins over them, because that is where an edited size was saved.
+ pub fn open(path: &str, mut settings: DocumentSettings) -> Result<Session, String> {
+ let editor = match projects::read_document(path) {
+ Ok(document) => {
+ if let Some(video) = document.get("video")
+ && let (Some(width), Some(height)) = (
+ video.get("width").and_then(|value| value.as_u64()),
+ video.get("height").and_then(|value| value.as_u64()),
+ )
+ && width > 0
+ && height > 0
+ {
+ settings.width = width as u32;
+ settings.height = height as u32;
+ }
+ match Editor::from_document(&document) {
+ Some(editor) => editor,
+ // The settings-only manifest `create` writes: a project
+ // closed before its first edit reopens empty, it is not
+ // corrupt.
+ None if projects::is_settings_only(&document) => Editor::new(),
+ None => {
+ return Err(format!("{path} holds a document this build cannot read"));
+ }
+ }
+ }
+ // No document yet - a project created moments ago.
+ Err(_) => Editor::new(),
+ };
+ Ok(Session {
+ path: path.to_owned(),
+ settings,
+ editor,
+ })
+ }
+
+ /// Opens the project a [`projects::ProjectInfo`] describes.
+ pub fn open_info(info: &projects::ProjectInfo) -> Result<Session, String> {
+ Session::open(
+ &info.path,
+ DocumentSettings {
+ name: info.name.clone(),
+ width: info.width,
+ height: info.height,
+ rate_num: info.rate_num,
+ rate_den: info.rate_den,
+ },
+ )
+ }
+
+ /// The project folder.
+ pub fn path(&self) -> &str {
+ &self.path
+ }
+
+ /// The session's settings.
+ pub fn settings(&self) -> &DocumentSettings {
+ &self.settings
+ }
+
+ /// The edit as it stands.
+ pub fn project(&self) -> &Project {
+ self.editor.project()
+ }
+
+ /// Whether there is something to undo.
+ pub fn can_undo(&self) -> bool {
+ self.editor.can_undo()
+ }
+
+ /// Whether there is something to redo.
+ pub fn can_redo(&self) -> bool {
+ self.editor.can_redo()
+ }
+
+ /// The current state without changing anything.
+ pub fn view(&self) -> EditorView {
+ self.view_with(None)
+ }
+
+ fn view_with(&self, created_id: Option<String>) -> EditorView {
+ EditorView {
+ project: self.editor.project().clone(),
+ can_undo: self.editor.can_undo(),
+ can_redo: self.editor.can_redo(),
+ settings: SettingsView {
+ name: self.settings.name.clone(),
+ width: self.settings.width,
+ height: self.settings.height,
+ rate_num: self.settings.rate_num,
+ rate_den: self.settings.rate_den,
+ },
+ created_id,
+ }
+ }
+
+ /// Applies one edit command and returns the new state.
+ pub fn apply(&mut self, command: Command) -> Result<EditorView, String> {
+ let outcome = self
+ .editor
+ .apply(command)
+ .map_err(|error| error.to_string())?;
+ Ok(self.view_with(outcome.created_id))
+ }
+
+ /// Steps the history back one edit.
+ pub fn undo(&mut self) -> EditorView {
+ self.editor.undo();
+ self.view()
+ }
+
+ /// Steps the history forward one edit.
+ pub fn redo(&mut self) -> EditorView {
+ self.editor.redo();
+ self.view()
+ }
+
+ /// Takes new settings and hands back what a save must write: the folder
+ /// and the document. The output size can have been edited in the
+ /// preview footer and the name in the project details dialog, so both
+ /// ride along here. The disk write is the caller's, so it can happen off
+ /// the thread that owns the session; [`Session::save`] does both.
+ pub fn prepare_save(
+ &mut self,
+ name: Option<&str>,
+ width: Option<u32>,
+ height: Option<u32>,
+ ) -> (String, serde_json::Value) {
+ if let Some(name) = name {
+ let trimmed = name.trim();
+ if !trimmed.is_empty() {
+ self.settings.name = trimmed.to_owned();
+ }
+ }
+ // A zero dimension is never a real output size, only a caller bug -
+ // saving it would poison the document until the next open.
+ if let Some(width) = width.filter(|width| *width > 0) {
+ self.settings.width = width;
+ }
+ if let Some(height) = height.filter(|height| *height > 0) {
+ self.settings.height = height;
+ }
+ (self.path.clone(), self.editor.to_document(&self.settings))
+ }
+
+ /// Writes the session's document to its project folder.
+ pub fn save(
+ &mut self,
+ name: Option<&str>,
+ width: Option<u32>,
+ height: Option<u32>,
+ ) -> Result<(), String> {
+ let (path, document) = self.prepare_save(name, width, height);
+ projects::save(&path, &document)
+ }
+
+ /// The document as it would be saved.
+ pub fn document(&self) -> serde_json::Value {
+ self.editor.to_document(&self.settings)
+ }
+
+ /// The active timeline flattened for rendering. This is what export and
+ /// preview consume: the engine flattens its own session, so the pixels
+ /// rendered are the model's, never a copy of it.
+ pub fn flattened_clips(&self) -> Vec<concat_export::ExportClip> {
+ concat_export::flatten::flatten_timeline(self.editor.project(), None)
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn settings() -> DocumentSettings {
+ DocumentSettings {
+ name: "Test".to_owned(),
+ width: 1920,
+ height: 1080,
+ rate_num: 30,
+ rate_den: 1,
+ }
+ }
+
+ #[test]
+ fn a_fresh_project_opens_empty_and_round_trips_a_save() {
+ let scratch =
+ std::env::temp_dir().join(format!("concat-session-test-{}", std::process::id()));
+ let _ = std::fs::remove_dir_all(&scratch);
+ std::fs::create_dir_all(&scratch).expect("scratch dir");
+ let info = projects::create(&scratch.to_string_lossy(), "Fresh", 1920, 1080, 30, 1)
+ .expect("creates");
+
+ let mut session = Session::open_info(&info).expect("opens the settings-only manifest");
+ assert!(!session.can_undo());
+ let view = session.apply(Command::AddTrack).expect("adds a track");
+ assert!(view.can_undo);
+ session
+ .save(Some("Renamed"), Some(1280), Some(720))
+ .expect("saves");
+
+ let reopened = Session::open(&info.path, settings()).expect("reopens");
+ assert_eq!(
+ reopened.settings().name,
+ "Test",
+ "the manifest's name is what open gets"
+ );
+ assert_eq!(
+ (reopened.settings().width, reopened.settings().height),
+ (1280, 720)
+ );
+ assert_eq!(
+ reopened.project().active().tracks.len(),
+ session.project().active().tracks.len()
+ );
+
+ let _ = std::fs::remove_dir_all(&scratch);
+ }
+
+ #[test]
+ fn a_corrupt_document_is_refused() {
+ let scratch =
+ std::env::temp_dir().join(format!("concat-corrupt-test-{}", std::process::id()));
+ let _ = std::fs::remove_dir_all(&scratch);
+ std::fs::create_dir_all(&scratch).expect("scratch dir");
+ std::fs::write(scratch.join("concat.json"), br#"{"timelines": "garbage"}"#)
+ .expect("writes");
+ assert!(Session::open(&scratch.to_string_lossy(), settings()).is_err());
+ let _ = std::fs::remove_dir_all(&scratch);
+ }
+}
diff --git a/engine/crates/concat-host/src/templates.rs b/engine/crates/concat-host/src/templates.rs
new file mode 100644
index 00000000..d4ff6a70
--- /dev/null
+++ b/engine/crates/concat-host/src/templates.rs
@@ -0,0 +1,607 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Template bundles on disk.
+//!
+//! A template is a folder in the app config directory's `templates/`:
+//!
+//! - `template.json` - an ordinary project document whose placeholder media
+//! (engine `placeholder: true`) have blank paths, and whose bundled media
+//! and fonts - the music, overlays and faces that are part of the design -
+//! have paths relative to the bundle (`assets/...`).
+//! - `assets/` - those bundled files, copied in at save time.
+//! - `poster.jpg` - a frame of the edit, for the gallery card.
+//!
+//! Saving packs the open project into a bundle; instantiating unpacks one
+//! into a fresh project folder and fills every slot with the user's media
+//! *through the engine* (`Command::FillSlot` in a batch), so what a fill
+//! means is defined in exactly one place - `concat-project` - and the editor
+//! never opens on a slot with a dead path.
+
+use std::path::{Path, PathBuf};
+
+use concat_project::{Command, DocumentSettings, Editor};
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+
+use crate::{media, projects};
+
+const MANIFEST: &str = "template.json";
+const ASSETS: &str = "assets";
+const POSTER: &str = "poster.jpg";
+
+/// One template, as the gallery sees it.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct TemplateInfo {
+ /// The bundle folder.
+ pub path: String,
+ /// The template's name.
+ pub name: String,
+ /// Output width in pixels.
+ pub width: u32,
+ /// Output height in pixels.
+ pub height: u32,
+ /// Frame rate numerator.
+ pub rate_num: i64,
+ /// Frame rate denominator.
+ pub rate_den: i64,
+ /// The slots to fill, in the order they first appear on the timeline.
+ pub slots: Vec<SlotInfo>,
+ /// Whether the bundle carries a poster.
+ pub has_poster: bool,
+}
+
+/// One placeholder the user's media will replace.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct SlotInfo {
+ /// The placeholder media's id in the template document.
+ pub media_id: String,
+ /// The placeholder's name, as the creator labelled it.
+ pub name: String,
+ /// What kind of media the slot wants.
+ pub kind: concat_project::model::MediaKind,
+ /// Timeline seconds this slot covers, summed over its clips.
+ pub seconds: f64,
+}
+
+/// The user's media for one slot, straight from a probe.
+#[derive(Deserialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct SlotFill {
+ /// Which slot.
+ pub media_id: String,
+ /// What fills it.
+ pub item: concat_project::commands::NewMedia,
+}
+
+/// Where the library lives under the config directory.
+pub fn templates_dir(config: &Path) -> PathBuf {
+ config.join("templates")
+}
+
+/// Packs the given document into a new template bundle.
+///
+/// `project_path` is where the document's media paths currently resolve; it
+/// also donates the poster frame. Refuses a name that already exists rather
+/// than overwriting someone's work.
+pub fn save(
+ config: &Path,
+ document: &Value,
+ settings: &DocumentSettings,
+ project_path: &str,
+ name: &str,
+) -> Result<TemplateInfo, String> {
+ let root = templates_dir(config).join(projects::folder_name(name));
+ if root.join(MANIFEST).exists() {
+ return Err(format!("a template named {name:?} already exists"));
+ }
+ let assets = root.join(ASSETS);
+ std::fs::create_dir_all(&assets)
+ .map_err(|error| format!("could not create {}: {error}", assets.display()))?;
+
+ // Typed through the engine's reader and writer, not hand-edited JSON:
+ // the manifest is written by the same code path every save uses, and a
+ // schema change breaks this at compile time.
+ let Some(mut project) = concat_project::from_document(document) else {
+ return Err("the open project is not a document this build can read".to_owned());
+ };
+
+ // Placeholder media loses its path - it was the creator's own footage,
+ // standing in. Everything else is part of the design and ships in the
+ // bundle, referenced relative to it.
+ for item in &mut project.media {
+ if item.placeholder {
+ item.path = String::new();
+ continue;
+ }
+ item.path = bundle_file(&assets, &item.id.clone(), &item.path.clone())?;
+ }
+ for (index, font) in project.fonts.iter_mut().enumerate() {
+ font.path = bundle_file(&assets, &format!("font{index}"), &font.path.clone())?;
+ }
+
+ let mut settings = settings.clone();
+ settings.name = name.to_owned();
+ let document = concat_project::to_document(&settings, &project);
+
+ let encoded = serde_json::to_vec_pretty(&document)
+ .map_err(|error| format!("could not encode the template: {error}"))?;
+ std::fs::write(root.join(MANIFEST), encoded)
+ .map_err(|error| format!("could not write the template: {error}"))?;
+
+ // Best effort: a template without a poster is still a template.
+ if let Ok(bytes) = media::poster_frame(project_path) {
+ let _ = std::fs::write(root.join(POSTER), bytes);
+ }
+
+ read_info(&root)
+}
+
+/// Copies one source file into `assets/` and returns its bundle-relative
+/// path. The bundle must be self-contained, so a missing source is an error,
+/// not a warning - a template that cannot find its own music is not one.
+fn bundle_file(assets: &Path, id: &str, source: &str) -> Result<String, String> {
+ if source.is_empty() {
+ return Err(
+ "a media item has no file behind it; fill or remove it before saving a template"
+ .to_owned(),
+ );
+ }
+ let base = Path::new(source)
+ .file_name()
+ .and_then(|name| name.to_str())
+ .map(projects::folder_name)
+ .unwrap_or_else(|| "file".to_owned());
+ let name = format!("{}-{base}", projects::folder_name(id));
+ let destination = assets.join(&name);
+ std::fs::copy(source, &destination)
+ .map_err(|error| format!("could not bundle {source}: {error}"))?;
+ Ok(format!("{ASSETS}/{name}"))
+}
+
+/// Every template in the library, in name order.
+///
+/// A folder that does not parse is skipped rather than sinking the whole
+/// gallery - the same grace the recents list extends to vanished projects.
+pub fn list(config: &Path) -> Vec<TemplateInfo> {
+ let Ok(entries) = std::fs::read_dir(templates_dir(config)) else {
+ return Vec::new();
+ };
+ let mut templates: Vec<TemplateInfo> = entries
+ .filter_map(|entry| entry.ok())
+ .filter_map(|entry| read_info(&entry.path()).ok())
+ .collect();
+ templates.sort_by(|left, right| left.name.cmp(&right.name));
+ templates
+}
+
+/// Reads one bundle's manifest into gallery form.
+fn read_info(root: &Path) -> Result<TemplateInfo, String> {
+ let manifest = root.join(MANIFEST);
+ let bytes = std::fs::read(&manifest)
+ .map_err(|error| format!("could not read {}: {error}", manifest.display()))?;
+ let document: Value = serde_json::from_slice(&bytes)
+ .map_err(|error| format!("{} is not a template: {error}", manifest.display()))?;
+
+ let video = document
+ .get("video")
+ .ok_or("the template names no output format")?;
+ let dimension = |field: &str| {
+ video
+ .get(field)
+ .and_then(Value::as_u64)
+ .and_then(|value| u32::try_from(value).ok())
+ .filter(|value| *value > 0)
+ .ok_or_else(|| format!("the template has no usable {field}"))
+ };
+ let rate = |field: &str| {
+ video
+ .get(field)
+ .and_then(Value::as_i64)
+ .filter(|value| *value > 0)
+ .ok_or_else(|| format!("the template has no usable {field}"))
+ };
+
+ Ok(TemplateInfo {
+ path: root.to_string_lossy().into_owned(),
+ name: document
+ .get("name")
+ .and_then(Value::as_str)
+ .unwrap_or("Template")
+ .to_owned(),
+ width: dimension("width")?,
+ height: dimension("height")?,
+ rate_num: rate("rateNum")?,
+ rate_den: rate("rateDen")?,
+ slots: slots_of(&document),
+ has_poster: root.join(POSTER).exists(),
+ })
+}
+
+/// The placeholder media of a document, ordered by when each first appears
+/// on the active timeline - the order a user fills them in.
+///
+/// Typed through the engine's own reader: a schema change breaks this at
+/// compile time, not as a gallery that silently shows no slots.
+fn slots_of(document: &Value) -> Vec<SlotInfo> {
+ let Some(project) = concat_project::from_document(document) else {
+ return Vec::new();
+ };
+ let clips = project
+ .timelines
+ .iter()
+ .find(|timeline| timeline.id == project.active_timeline_id)
+ .or_else(|| project.timelines.first())
+ .map(|timeline| timeline.clips.as_slice())
+ .unwrap_or(&[]);
+
+ let mut slots: Vec<(f64, SlotInfo)> = Vec::new();
+ for item in &project.media {
+ if !item.placeholder {
+ continue;
+ }
+
+ let mut first = f64::MAX;
+ let mut seconds = 0.0;
+ for clip in clips.iter().filter(|clip| clip.media_id == item.id) {
+ first = first.min(clip.start);
+ seconds += clip.duration;
+ }
+
+ slots.push((
+ first,
+ SlotInfo {
+ media_id: item.id.clone(),
+ name: item.name.clone(),
+ kind: item.kind,
+ seconds,
+ },
+ ));
+ }
+ slots.sort_by(|(left, _), (right, _)| left.total_cmp(right));
+ slots.into_iter().map(|(_, slot)| slot).collect()
+}
+
+/// Unpacks a template into a fresh project with every slot filled.
+///
+/// The fills go through the engine as one `Batch` of `FillSlot` commands, so
+/// the semantics live in `concat-project` and a half-fillable set of media
+/// leaves no half-made project behind.
+pub fn instantiate(
+ template: &str,
+ location: &str,
+ name: &str,
+ fills: Vec<SlotFill>,
+) -> Result<projects::ProjectInfo, String> {
+ let bundle = PathBuf::from(template);
+ let manifest = bundle.join(MANIFEST);
+ let bytes = std::fs::read(&manifest)
+ .map_err(|error| format!("could not read {}: {error}", manifest.display()))?;
+ let mut document: Value = serde_json::from_slice(&bytes)
+ .map_err(|error| format!("{} is not a template: {error}", manifest.display()))?;
+ let info = read_info(&bundle)?;
+
+ let unfilled: Vec<&SlotInfo> = info
+ .slots
+ .iter()
+ .filter(|slot| !fills.iter().any(|fill| fill.media_id == slot.media_id))
+ .collect();
+ if !unfilled.is_empty() {
+ return Err(format!("{} slot(s) still need a clip", unfilled.len()));
+ }
+
+ let project = projects::create(
+ location,
+ name,
+ info.width,
+ info.height,
+ info.rate_num,
+ info.rate_den,
+ )?;
+ let root = PathBuf::from(&project.path);
+
+ // The bundle's assets move into the project so it stands alone; the
+ // document's relative paths become project-local absolute ones.
+ copy_assets(&bundle.join(ASSETS), &root.join(ASSETS))?;
+ document["name"] = Value::String(project.name.clone());
+ for section in ["media", "fonts"] {
+ if let Some(items) = document.get_mut(section).and_then(Value::as_array_mut) {
+ for item in items {
+ let Some(path) = item.get("path").and_then(Value::as_str) else {
+ continue;
+ };
+ if let Some(relative) = path.strip_prefix(&format!("{ASSETS}/")) {
+ let absolute = root.join(ASSETS).join(relative);
+ item["path"] = Value::String(absolute.to_string_lossy().into_owned());
+ }
+ }
+ }
+ }
+
+ let mut editor = Editor::from_document(&document)
+ .ok_or("this template holds a document this build cannot read")?;
+ if !fills.is_empty() {
+ editor
+ .apply(Command::Batch {
+ commands: fills
+ .into_iter()
+ .map(|fill| Command::FillSlot {
+ media_id: fill.media_id,
+ item: fill.item,
+ })
+ .collect(),
+ })
+ .map_err(|error| format!("could not fill the template: {error}"))?;
+ }
+
+ let settings = DocumentSettings {
+ name: project.name.clone(),
+ width: info.width,
+ height: info.height,
+ rate_num: info.rate_num,
+ rate_den: info.rate_den,
+ };
+ projects::save(&project.path, &editor.to_document(&settings))?;
+
+ // After the save, so the cached poster reads as fresh.
+ if bundle.join(POSTER).is_file() {
+ let cache = root.join("cache");
+ let _ = std::fs::create_dir_all(&cache);
+ let _ = std::fs::copy(bundle.join(POSTER), cache.join("preview.jpg"));
+ }
+
+ Ok(project)
+}
+
+fn copy_assets(from: &Path, to: &Path) -> Result<(), String> {
+ let Ok(entries) = std::fs::read_dir(from) else {
+ return Ok(()); // a template with no bundled assets
+ };
+ std::fs::create_dir_all(to)
+ .map_err(|error| format!("could not create {}: {error}", to.display()))?;
+ for entry in entries.filter_map(|entry| entry.ok()) {
+ let source = entry.path();
+ if source.is_file() {
+ let destination = to.join(entry.file_name());
+ std::fs::copy(&source, &destination)
+ .map_err(|error| format!("could not copy {}: {error}", source.display()))?;
+ }
+ }
+ Ok(())
+}
+
+/// Removes one template bundle for good.
+///
+/// Only ever a folder inside the templates directory that actually holds a
+/// manifest - this function deletes recursively, and a stray path here would
+/// be a catastrophe, so it is checked rather than trusted.
+pub fn delete(config: &Path, path: &str) -> Result<(), String> {
+ let target = PathBuf::from(path);
+ let library = templates_dir(config);
+ let (Ok(target), Ok(library)) = (target.canonicalize(), library.canonicalize()) else {
+ return Err(format!("{path} is not a template"));
+ };
+ if !target.starts_with(&library) || target == library || !target.join(MANIFEST).is_file() {
+ return Err(format!("{path} is not a template"));
+ }
+ std::fs::remove_dir_all(&target)
+ .map_err(|error| format!("could not delete {}: {error}", target.display()))
+}
+
+/// The poster bytes for one bundle, or an error the caller treats as "no art".
+pub fn poster(path: &str) -> Result<Vec<u8>, String> {
+ std::fs::read(Path::new(path).join(POSTER))
+ .map_err(|error| format!("no poster for {path}: {error}"))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use concat_project::model::MediaKind;
+ use serde_json::json;
+
+ fn template_document() -> Value {
+ json!({
+ "concat": "0.1.0", "version": 1, "name": "Beat Intro",
+ "video": { "width": 1920, "height": 1080, "rateNum": 30, "rateDen": 1 },
+ "media": [
+ { "id": "m1", "path": "", "name": "A-roll", "duration": 4.0, "kind": "video",
+ "hasAudio": true, "placeholder": true },
+ { "id": "m2", "path": "", "name": "B-roll", "duration": 2.0, "kind": "video",
+ "hasAudio": false, "placeholder": true }
+ ],
+ "tracks": [{ "id": "T1", "name": "Track 1", "visible": true, "muted": false }],
+ "clips": [
+ { "id": "c2", "trackId": "T1", "mediaId": "m2", "name": "B-roll",
+ "kind": "video", "start": 4.0, "duration": 2.0, "sourceStart": 0.0,
+ "volume": 1.0, "fadeIn": 0.0, "fadeOut": 0.0, "scale": 1.0,
+ "offsetX": 0.0, "offsetY": 0.0, "rotation": 0.0, "opacity": 1.0,
+ "speed": 1.0, "preservePitch": true, "filters": [], "videoEffects": [] },
+ { "id": "c1", "trackId": "T1", "mediaId": "m1", "name": "A-roll",
+ "kind": "video", "start": 0.0, "duration": 4.0, "sourceStart": 1.0,
+ "volume": 1.0, "fadeIn": 0.0, "fadeOut": 0.0, "scale": 1.0,
+ "offsetX": 0.0, "offsetY": 0.0, "rotation": 0.0, "opacity": 1.0,
+ "speed": 1.0, "preservePitch": true, "filters": [], "videoEffects": [] }
+ ]
+ })
+ }
+
+ fn fill(media_id: &str, path: &str, duration: f64) -> SlotFill {
+ SlotFill {
+ media_id: media_id.to_owned(),
+ item: concat_project::commands::NewMedia {
+ path: path.to_owned(),
+ name: path.rsplit('/').next().unwrap_or(path).to_owned(),
+ duration: Some(duration),
+ kind: MediaKind::Video,
+ width: Some(1920),
+ height: Some(1080),
+ frame_rate: Some(30.0),
+ frame_rate_fraction: Some("30/1".to_owned()),
+ video_codec: Some("h264".to_owned()),
+ audio_codec: None,
+ has_audio: false,
+ },
+ }
+ }
+
+ #[test]
+ fn slots_come_back_in_timeline_order() {
+ // c2 is listed first but starts later; the fill order is by time.
+ let slots = slots_of(&template_document());
+ assert_eq!(slots.len(), 2);
+ assert_eq!(slots[0].media_id, "m1");
+ assert_eq!(slots[0].seconds, 4.0);
+ assert_eq!(slots[1].media_id, "m2");
+ }
+
+ #[test]
+ fn saving_packs_a_bundle_and_it_round_trips() {
+ let scratch = std::env::temp_dir().join("concat-template-save-test");
+ let _ = std::fs::remove_dir_all(&scratch);
+ let config = scratch.join("config");
+ let project = scratch.join("project");
+ std::fs::create_dir_all(&project).expect("scratch dirs");
+
+ // A real file for the bundled music; the placeholder needs none.
+ let music = project.join("music.mp3");
+ std::fs::write(&music, b"not really an mp3").expect("writes");
+
+ let mut document = template_document();
+ {
+ let media = document["media"].as_array_mut().expect("media");
+ // m1 stays a placeholder; m2 becomes ordinary bundled media.
+ media[1]["placeholder"] = Value::Bool(false);
+ media[1]["path"] = Value::String(music.to_string_lossy().into_owned());
+ media[1]["kind"] = Value::String("audio".to_owned());
+ }
+
+ let settings = DocumentSettings {
+ name: "Beat Intro".to_owned(),
+ width: 1920,
+ height: 1080,
+ rate_num: 30,
+ rate_den: 1,
+ };
+ let info = save(
+ &config,
+ &document,
+ &settings,
+ &project.to_string_lossy(),
+ "Beat Intro",
+ )
+ .expect("saves");
+ assert_eq!(info.name, "Beat Intro");
+ assert_eq!(info.slots.len(), 1, "only the placeholder is a slot");
+ assert_eq!(info.slots[0].media_id, "m1");
+
+ // The same name again is refused, not overwritten.
+ assert!(
+ save(
+ &config,
+ &document,
+ &settings,
+ &project.to_string_lossy(),
+ "Beat Intro"
+ )
+ .is_err()
+ );
+
+ let bundle = PathBuf::from(&info.path);
+ let packed: Value =
+ serde_json::from_slice(&std::fs::read(bundle.join(MANIFEST)).expect("manifest"))
+ .expect("parses");
+ let media = packed["media"].as_array().expect("media");
+ assert_eq!(media[0]["path"], "", "the placeholder's path is blanked");
+ let bundled = media[1]["path"].as_str().expect("path");
+ assert!(
+ bundled.starts_with("assets/"),
+ "bundled media went relative: {bundled}"
+ );
+ assert!(
+ bundle.join(bundled).is_file(),
+ "and the file really is in the bundle"
+ );
+
+ assert_eq!(list(&config).len(), 1, "the library sees it");
+
+ // And it instantiates: the asset lands in the project, absolute.
+ let location = scratch.join("projects");
+ std::fs::create_dir_all(&location).expect("scratch dir");
+ let made = instantiate(
+ &info.path,
+ &location.to_string_lossy(),
+ "From bundle",
+ vec![fill("m1", "/mine/clip.mp4", 6.0)],
+ )
+ .expect("instantiates");
+
+ let opened = projects::read_document(&made.path).expect("reads back");
+ let media = opened["media"].as_array().expect("media");
+ let sound = media[1]["path"].as_str().expect("path");
+ assert!(
+ Path::new(sound).is_absolute() && Path::new(sound).is_file(),
+ "the bundled asset was copied into the project and repointed: {sound}"
+ );
+
+ let _ = std::fs::remove_dir_all(&scratch);
+ }
+
+ #[test]
+ fn instantiating_fills_through_the_engine() {
+ let scratch = std::env::temp_dir().join("concat-template-test");
+ let _ = std::fs::remove_dir_all(&scratch);
+ let bundle = scratch.join("templates").join("Beat Intro");
+ std::fs::create_dir_all(&bundle).expect("scratch dir");
+ std::fs::write(
+ bundle.join(MANIFEST),
+ serde_json::to_vec(&template_document()).expect("encodes"),
+ )
+ .expect("writes");
+
+ let projects_dir = scratch.join("projects");
+ std::fs::create_dir_all(&projects_dir).expect("scratch dir");
+ let location = projects_dir.to_string_lossy().into_owned();
+
+ // Missing a slot is refused before any folder is written.
+ let refused = instantiate(
+ &bundle.to_string_lossy(),
+ &location,
+ "Missing",
+ vec![fill("m1", "/mine/a.mp4", 9.0)],
+ );
+ assert!(refused.unwrap_err().contains("slot"));
+
+ let project = instantiate(
+ &bundle.to_string_lossy(),
+ &location,
+ "My edit",
+ vec![
+ fill("m1", "/mine/a.mp4", 9.0),
+ fill("m2", "/mine/b.mp4", 9.0),
+ ],
+ )
+ .expect("instantiates");
+ assert_eq!((project.width, project.height), (1920, 1080));
+
+ let document = projects::read_document(&project.path).expect("reads back");
+ let media = document["media"].as_array().expect("media");
+ assert!(media.iter().all(|item| {
+ !item
+ .get("placeholder")
+ .and_then(Value::as_bool)
+ .unwrap_or(false)
+ }));
+ assert_eq!(media[0]["path"], "/mine/a.mp4");
+ // The slot kept the template's timing; the in-point reset (c1 had 1.0).
+ let clips = document["clips"].as_array().expect("clips");
+ let c1 = clips.iter().find(|clip| clip["id"] == "c1").expect("c1");
+ assert_eq!(c1["start"], 0.0);
+ assert_eq!(c1["duration"], 4.0);
+ assert_eq!(c1["sourceStart"], 0.0);
+
+ let _ = std::fs::remove_dir_all(&scratch);
+ }
+}
diff --git a/engine/crates/concat-media/Cargo.toml b/engine/crates/concat-media/Cargo.toml
index 6413d114..1ac203b3 100644
--- a/engine/crates/concat-media/Cargo.toml
+++ b/engine/crates/concat-media/Cargo.toml
@@ -4,23 +4,18 @@ version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
-description = "Getting pixels in and out of media files. The only crate that knows FFmpeg exists."
+description = "Getting pixels and samples in and out of media files. The only crate that knows FFmpeg exists."
[dependencies]
concat-core.workspace = true
-rusty_ffmpeg = { version = "0.17.0", default-features = false, features = ["ffmpeg8", "use_prebuilt_binding"], optional = true }
-serde_json.workspace = true
+# Linked FFmpeg: the safe wrapper over libavformat, libavcodec, libavfilter,
+# libswscale and libswresample. Built against the FFmpeg development
+# libraries on the machine - pkg-config where there is one, FFMPEG_DIR
+# pointing at an unpacked shared build otherwise. FFmpeg 7 or newer: the
+# display-rotation side data is read from where 7.0 moved it.
+ffmpeg-the-third = "6"
+libc = "0.2"
thiserror.workspace = true
[lints]
workspace = true
-
-[features]
-# Links FFmpeg directly instead of only spawning it.
-#
-# Off by default, and deliberately so: the subprocess backend needs nothing but
-# `ffmpeg` on PATH, while this one needs a development build with headers and
-# import libraries. Building the engine must not require that.
-#
-# See docs/decisions/0002-ffmpeg-over-a-pipe.md.
-ffi = ["dep:rusty_ffmpeg"]
diff --git a/engine/crates/concat-media/src/audio.rs b/engine/crates/concat-media/src/audio.rs
index b64f9896..cdd227ca 100644
--- a/engine/crates/concat-media/src/audio.rs
+++ b/engine/crates/concat-media/src/audio.rs
@@ -4,21 +4,25 @@
//! Planning and running the audio mix.
//!
//! The audio equivalent of `concat-render`'s frame plan: [`mix_graph`] turns a
-//! set of audible clips into one FFmpeg `filter_complex`, as a pure function
-//! that is unit-tested without running anything. [`mix_to_file`] and [`mux`]
-//! then hand the plan to FFmpeg.
+//! set of audible clips into one FFmpeg filtergraph, as a pure function that
+//! is unit-tested without running anything. [`mix_to_file`] hands the plan to
+//! libavfilter and [`mux`] joins the result with the picture.
//!
-//! This lives in the engine, not the app host, so that there is exactly one
+//! This lives in the engine, not the app, so that there is exactly one
//! definition of what speed, fades and gain mean for sound - the same reason
//! `Clip::source_time_at` is the one definition for picture.
use std::path::Path;
-use crate::error::{Error, Result};
-use crate::process::{base_command, run_to_completion};
+use ffmpeg_the_third as ffmpeg;
+use ffmpeg_the_third::codec::encoder;
+use ffmpeg_the_third::filter;
+use ffmpeg_the_third::util::channel_layout::ChannelLayout;
+use ffmpeg_the_third::util::format::Sample;
+use ffmpeg_the_third::util::frame::audio::Audio;
-/// Name used in error messages; the binary run comes from [`crate::ffmpeg`].
-const FFMPEG: &str = "ffmpeg";
+use crate::error::{Error, Result};
+use crate::ffi;
/// The rate everything is mixed at.
pub const MIX_RATE: u32 = 48_000;
@@ -60,9 +64,9 @@ pub struct AudioClip {
/// Refuses a filter chain that could break out of its slot in the graph.
///
-/// A chain is spliced into a `filter_complex`, where `;` starts a new chain
-/// and `[..]` rebinds streams - a chain containing either is no longer a
-/// filter applied to this clip, whatever else it might be.
+/// A chain is spliced into a filtergraph, where `;` starts a new chain and
+/// `[..]` rebinds streams - a chain containing either is no longer a filter
+/// applied to this clip, whatever else it might be.
pub fn validate_chain(chain: &str) -> Result<()> {
let forbidden = |c: char| matches!(c, ';' | '[' | ']' | '\n' | '\r');
match chain.chars().find(|&c| forbidden(c)) {
@@ -85,7 +89,10 @@ pub fn speed_filters(speed: f64, preserve_pitch: bool) -> Vec<String> {
return Vec::new();
}
if preserve_pitch {
- tempo_stages(speed).iter().map(|factor| format!("atempo={factor:.6}")).collect()
+ tempo_stages(speed)
+ .iter()
+ .map(|factor| format!("atempo={factor:.6}"))
+ .collect()
} else {
vec![
format!("aresample={MIX_RATE}"),
@@ -116,13 +123,13 @@ pub fn tempo_stages(speed: f64) -> Vec<f64> {
stages
}
-/// Builds the `filter_complex` that trims, retimes, shapes, delays and mixes
-/// every clip into one `[out]` stream of exactly `duration` seconds.
+/// Builds the filtergraph that trims, retimes, shapes, delays and mixes every
+/// clip into one `[out]` stream of exactly `duration` seconds.
///
-/// Input `N` in the command must be `clips[N].path`, and every input must
-/// actually contain an audio stream - FFmpeg refuses a graph that names
-/// `[N:a]` on a silent input rather than treating it as silence. The caller
-/// decides membership by probing; this function decides what the mix means.
+/// Input `N` is `clips[N].path`, bound to the label `[N:a]`, and every input
+/// must actually contain an audio stream - the graph names `[N:a]` on it
+/// rather than treating silence as a stream. The caller decides membership
+/// by probing; this function decides what the mix means.
pub fn mix_graph(clips: &[AudioClip], duration: f64) -> Result<String> {
let mut chains: Vec<String> = Vec::new();
@@ -165,14 +172,19 @@ pub fn mix_graph(clips: &[AudioClip], duration: f64) -> Result<String> {
}
if clip.fade_out > 0.0 {
let start = (clip.duration - clip.fade_out).max(0.0);
- stage.push_str(&format!(",afade=t=out:st={start:.4}:d={:.4}", clip.fade_out));
+ stage.push_str(&format!(
+ ",afade=t=out:st={start:.4}:d={:.4}",
+ clip.fade_out
+ ));
}
stage.push_str(&format!(",adelay={delay_ms}:all=1[a{index}]"));
chains.push(stage);
}
- let inputs: String = (0..clips.len()).map(|index| format!("[a{index}]")).collect();
+ let inputs: String = (0..clips.len())
+ .map(|index| format!("[a{index}]"))
+ .collect();
let mix = if clips.len() == 1 {
// amix with one input would still apply its normalisation curve.
format!("{inputs}anull[mixed]")
@@ -208,43 +220,411 @@ pub fn mix_graph(clips: &[AudioClip], duration: f64) -> Result<String> {
Ok(chains.join(";"))
}
+/// The sample format the AAC encoder takes, which the graph's last stage
+/// lands on so no conversion happens between them.
+const MIX_FORMAT: Sample = Sample::F32(ffmpeg::util::format::sample::Type::Planar);
+
+/// One open input of the mix: its demuxer and decoder, and where in the
+/// graph its frames go.
+struct MixInput {
+ path: std::path::PathBuf,
+ input: ffmpeg::format::context::Input,
+ stream: usize,
+ decoder: ffmpeg::codec::decoder::Audio,
+ label: String,
+ done: bool,
+}
+
+impl MixInput {
+ /// One decoded frame, or `None` at the end of the file.
+ fn next(&mut self) -> Result<Option<Audio>> {
+ if self.done {
+ return Ok(None);
+ }
+ loop {
+ let mut frame = Audio::empty();
+ match self.decoder.receive_frame(&mut frame) {
+ Ok(()) => return Ok(Some(frame)),
+ Err(ffmpeg::Error::Eof) => {
+ self.done = true;
+ return Ok(None);
+ }
+ Err(error) if ffi::is_again(&error) => {}
+ Err(error) => return Err(ffi::fail("decode", &self.path, error)),
+ }
+ loop {
+ let mut packet = ffmpeg::Packet::empty();
+ match packet.read(&mut self.input) {
+ Ok(()) => {
+ if packet.stream() != self.stream {
+ continue;
+ }
+ match self.decoder.send_packet(&packet) {
+ Ok(()) => break,
+ Err(error) if ffi::is_again(&error) => break,
+ Err(error) => return Err(ffi::fail("send packet", &self.path, error)),
+ }
+ }
+ Err(ffmpeg::Error::Eof) => {
+ let _ = self.decoder.send_eof();
+ break;
+ }
+ Err(error) => return Err(ffi::fail("read", &self.path, error)),
+ }
+ }
+ }
+ }
+}
+
/// Mixes `clips` into one AAC file at `destination`.
///
/// Every clip's file must contain an audio stream; see [`mix_graph`].
pub fn mix_to_file(clips: &[AudioClip], duration: f64, destination: &Path) -> Result<()> {
- let graph = mix_graph(clips, duration)?;
+ ffi::init();
+ let graph_spec = mix_graph(clips, duration)?;
+ let missing = |name: &str| Error::Missing {
+ what: "filter",
+ name: name.to_owned(),
+ };
+
+ // Open every input first, and read one frame from each: the graph's
+ // sources are described by what actually comes out of the decoders.
+ let mut inputs: Vec<MixInput> = Vec::with_capacity(clips.len());
+ let mut first_frames: Vec<Audio> = Vec::with_capacity(clips.len());
+ let mut graph = filter::Graph::new();
+ for (index, clip) in clips.iter().enumerate() {
+ let path = clip.path.as_path();
+ let mut input =
+ ffmpeg::format::input(path).map_err(|error| ffi::fail("open", path, error))?;
+ let stream = input
+ .streams()
+ .best(ffmpeg::media::Type::Audio)
+ .ok_or_else(|| Error::NoAudioStream {
+ path: path.to_path_buf(),
+ })?;
+ let stream_index = stream.index();
+ let time_base = stream.time_base();
+ let decoder = ffmpeg::codec::Context::from_parameters(stream.parameters())
+ .and_then(|context| context.decoder().audio())
+ .map_err(|error| ffi::fail("open decoder", path, error))?;
+ // Near the in-point; atrim in the graph does the exact cut.
+ if clip.source_start > 0.0 {
+ let target = (clip.source_start * f64::from(ffmpeg::sys::AV_TIME_BASE)) as i64;
+ let _ = input.seek(target, ..=target);
+ }
+ let label = format!("{index}:a");
+ let mut mix_input = MixInput {
+ path: path.to_path_buf(),
+ input,
+ stream: stream_index,
+ decoder,
+ label,
+ done: false,
+ };
+ let first = mix_input.next()?.ok_or_else(|| Error::NoAudioStream {
+ path: path.to_path_buf(),
+ })?;
+ let args = format!(
+ "time_base={}/{}:sample_rate={}:sample_fmt={}:channel_layout={}",
+ time_base.numerator(),
+ time_base.denominator(),
+ first.rate(),
+ first.format().name(),
+ first.ch_layout().description()
+ );
+ graph
+ .add(
+ &filter::find("abuffer").ok_or_else(|| missing("abuffer"))?,
+ &mix_input.label,
+ &args,
+ )
+ .map_err(|error| ffi::fail("buffer source", path, error))?;
+ first_frames.push(first);
+ inputs.push(mix_input);
+ }
+ graph
+ .add(
+ &filter::find("abuffersink").ok_or_else(|| missing("abuffersink"))?,
+ "sink",
+ "",
+ )
+ .map_err(|error| ffi::fail("buffer sink", destination, error))?;
+
+ // The plan's `[out]` lands on the encoder's own format before the sink.
+ let spec = format!(
+ "{graph_spec};[out]aformat=sample_fmts=fltp:sample_rates={MIX_RATE}:channel_layouts=stereo[sink]"
+ );
+ let mut parser = graph
+ .input("sink", 0)
+ .map_err(|error| ffi::fail("filter graph", destination, error))?;
+ for input in &inputs {
+ parser = parser
+ .output(&input.label, 0)
+ .map_err(|error| ffi::fail("filter graph", destination, error))?;
+ }
+ parser
+ .parse(&spec)
+ .map_err(|error| ffi::fail("filter graph", destination, error))?;
+ graph
+ .validate()
+ .map_err(|error| ffi::fail("filter graph", destination, error))?;
+
+ // The encoder, and the file it writes into.
+ let codec = encoder::find_by_name("aac").ok_or_else(|| Error::Missing {
+ what: "encoder",
+ name: "aac".to_owned(),
+ })?;
+ let mut output = ffmpeg::format::output(destination)
+ .map_err(|error| ffi::fail("create", destination, error))?;
+ let global_header = output
+ .format()
+ .flags()
+ .contains(ffmpeg::format::Flags::GLOBAL_HEADER);
+ let mut audio = ffmpeg::codec::Context::new_with_codec(codec)
+ .encoder()
+ .audio()
+ .map_err(|error| ffi::fail("audio encoder", destination, error))?;
+ audio.set_rate(MIX_RATE as i32);
+ audio.set_format(MIX_FORMAT);
+ audio.set_ch_layout(ChannelLayout::STEREO);
+ audio.set_bit_rate(192_000);
+ audio.set_time_base(ffmpeg::Rational::new(1, MIX_RATE as i32));
+ if global_header {
+ audio.set_flags(ffmpeg::codec::Flags::GLOBAL_HEADER);
+ }
+ let mut encoder = audio
+ .open()
+ .map_err(|error| ffi::fail("open encoder", destination, error))?;
+ let frame_size = encoder.frame_size().max(1);
+ {
+ let mut stream = output
+ .add_stream(codec)
+ .map_err(|error| ffi::fail("add stream", destination, error))?;
+ stream.copy_parameters_from_context(&encoder);
+ stream.set_time_base(ffmpeg::Rational::new(1, MIX_RATE as i32));
+ }
+ output
+ .write_header()
+ .map_err(|error| ffi::fail("write header", destination, error))?;
+ let stream_time_base = output
+ .stream(0)
+ .map(|stream| stream.time_base())
+ .unwrap_or(ffmpeg::Rational::new(1, MIX_RATE as i32));
+ {
+ let mut sink = graph.get("sink").expect("the graph has a sink");
+ sink.sink().set_frame_size(frame_size);
+ }
+
+ let encoder_time_base = ffmpeg::Rational::new(1, MIX_RATE as i32);
+ let mut written_samples: i64 = 0;
+ let drain = |encoder: &mut encoder::audio::Encoder,
+ output: &mut ffmpeg::format::context::Output|
+ -> Result<()> {
+ loop {
+ let mut packet = ffmpeg::Packet::empty();
+ match encoder.receive_packet(&mut packet) {
+ Ok(()) => {
+ packet.set_stream(0);
+ packet.rescale_ts(encoder_time_base, stream_time_base);
+ packet
+ .write_interleaved(output)
+ .map_err(|error| ffi::fail("write packet", destination, error))?;
+ }
+ Err(ffmpeg::Error::Eof) => return Ok(()),
+ Err(error) if ffi::is_again(&error) => return Ok(()),
+ Err(error) => return Err(ffi::fail("encode", destination, error)),
+ }
+ }
+ };
- let mut command = base_command(crate::binaries::ffmpeg());
- command.arg("-y");
- for clip in clips {
- command.arg("-i").arg(&clip.path);
+ // Feed the sources round-robin and drain the sink after every push, so
+ // no input runs ahead of the others by more than a frame: what a filter
+ // has not consumed yet sits in memory.
+ for (index, first) in first_frames.into_iter().enumerate() {
+ let mut context = graph.get(&inputs[index].label).expect("source exists");
+ context
+ .source()
+ .add(&first)
+ .map_err(|error| ffi::fail("filter", &inputs[index].path, error))?;
+ }
+ let mut live = inputs.len();
+ loop {
+ // Pull everything the sink has.
+ loop {
+ let mut mixed = Audio::empty();
+ let pulled = {
+ let mut context = graph.get("sink").expect("the graph has a sink");
+ context.sink().frame(&mut mixed)
+ };
+ match pulled {
+ Ok(()) => {
+ mixed.set_pts(Some(written_samples));
+ written_samples += mixed.samples() as i64;
+ encoder
+ .send_frame(&mixed)
+ .map_err(|error| ffi::fail("encode", destination, error))?;
+ drain(&mut encoder, &mut output)?;
+ }
+ Err(ffmpeg::Error::Eof) => {
+ live = 0;
+ break;
+ }
+ Err(error) if ffi::is_again(&error) => break,
+ Err(error) => return Err(ffi::fail("filter output", destination, error)),
+ }
+ }
+ if live == 0 {
+ break;
+ }
+ // Push one frame from every input that still has one.
+ for input in inputs.iter_mut() {
+ if input.done {
+ continue;
+ }
+ match input.next()? {
+ Some(frame) => {
+ let mut context = graph.get(&input.label).expect("source exists");
+ context
+ .source()
+ .add(&frame)
+ .map_err(|error| ffi::fail("filter", &input.path, error))?;
+ }
+ None => {
+ let mut context = graph.get(&input.label).expect("source exists");
+ let _ = context.source().flush();
+ live -= 1;
+ }
+ }
+ }
+ // `live` counting down to zero means every source was flushed; the
+ // sink then drains to EOF on the next pass, which is what ends the
+ // outer loop.
+ if live == 0 {
+ loop {
+ let mut mixed = Audio::empty();
+ let pulled = {
+ let mut context = graph.get("sink").expect("the graph has a sink");
+ context.sink().frame(&mut mixed)
+ };
+ match pulled {
+ Ok(()) => {
+ mixed.set_pts(Some(written_samples));
+ written_samples += mixed.samples() as i64;
+ encoder
+ .send_frame(&mixed)
+ .map_err(|error| ffi::fail("encode", destination, error))?;
+ drain(&mut encoder, &mut output)?;
+ }
+ Err(ffmpeg::Error::Eof) => break,
+ Err(error) if ffi::is_again(&error) => break,
+ Err(error) => return Err(ffi::fail("filter output", destination, error)),
+ }
+ }
+ break;
+ }
}
- command
- .args(["-filter_complex", &graph])
- .args(["-map", "[out]"])
- .args(["-c:a", "aac", "-b:a", "192k"])
- .arg(destination);
- run_to_completion(&mut command, FFMPEG, destination)
+ encoder
+ .send_eof()
+ .map_err(|error| ffi::fail("encode", destination, error))?;
+ drain(&mut encoder, &mut output)?;
+ output
+ .write_trailer()
+ .map_err(|error| ffi::fail("write trailer", destination, error))
}
/// Joins an already-encoded video file and audio file into `output`.
///
/// Streams are copied, not re-encoded: the picture is already exactly as
/// asked, and a second pass would cost time and a generation of quality.
+/// The result ends with the shorter of the two.
pub fn mux(video: &Path, audio: &Path, output: &Path) -> Result<()> {
- let mut command = base_command(crate::binaries::ffmpeg());
- command
- .arg("-y")
- .arg("-i")
- .arg(video)
- .arg("-i")
- .arg(audio)
- .args(["-c:v", "copy", "-c:a", "copy", "-shortest"])
- .args(["-movflags", "+faststart"])
- .arg(output);
-
- run_to_completion(&mut command, FFMPEG, output)
+ ffi::init();
+ let mut video_in =
+ ffmpeg::format::input(video).map_err(|error| ffi::fail("open", video, error))?;
+ let mut audio_in =
+ ffmpeg::format::input(audio).map_err(|error| ffi::fail("open", audio, error))?;
+ let video_stream = video_in
+ .streams()
+ .best(ffmpeg::media::Type::Video)
+ .ok_or_else(|| Error::NoVideoStream {
+ path: video.to_path_buf(),
+ })?;
+ let audio_stream = audio_in
+ .streams()
+ .best(ffmpeg::media::Type::Audio)
+ .ok_or_else(|| Error::NoAudioStream {
+ path: audio.to_path_buf(),
+ })?;
+ let (video_index, video_tb) = (video_stream.index(), video_stream.time_base());
+ let (audio_index, audio_tb) = (audio_stream.index(), audio_stream.time_base());
+
+ let mut out =
+ ffmpeg::format::output(output).map_err(|error| ffi::fail("create", output, error))?;
+ {
+ let mut stream = out
+ .add_stream(encoder::find(ffmpeg::codec::Id::None))
+ .map_err(|error| ffi::fail("add stream", output, error))?;
+ stream.set_parameters(video_stream.parameters());
+ stream.set_time_base(video_tb);
+ }
+ {
+ let mut stream = out
+ .add_stream(encoder::find(ffmpeg::codec::Id::None))
+ .map_err(|error| ffi::fail("add stream", output, error))?;
+ stream.set_parameters(audio_stream.parameters());
+ stream.set_time_base(audio_tb);
+ }
+ out.write_header_with(ffmpeg::dict! { "movflags" => "+faststart" })
+ .map_err(|error| ffi::fail("write header", output, error))?;
+ let out_video_tb = out
+ .stream(0)
+ .map(|stream| stream.time_base())
+ .unwrap_or(video_tb);
+ let out_audio_tb = out
+ .stream(1)
+ .map(|stream| stream.time_base())
+ .unwrap_or(audio_tb);
+
+ // Alternate one packet from each; `write_interleaved` orders them by
+ // timestamp. When either input ends, the file ends: `-shortest`.
+ let mut copy = |input: &mut ffmpeg::format::context::Input,
+ wanted: usize,
+ from: ffmpeg::Rational,
+ to: ffmpeg::Rational,
+ stream: usize,
+ path: &Path|
+ -> Result<bool> {
+ loop {
+ let mut packet = ffmpeg::Packet::empty();
+ match packet.read(input) {
+ Ok(()) => {
+ if packet.stream() != wanted {
+ continue;
+ }
+ packet.set_stream(stream);
+ packet.rescale_ts(from, to);
+ packet.set_position(-1);
+ packet
+ .write_interleaved(&mut out)
+ .map_err(|error| ffi::fail("write packet", output, error))?;
+ return Ok(true);
+ }
+ Err(ffmpeg::Error::Eof) => return Ok(false),
+ Err(error) => return Err(ffi::fail("read", path, error)),
+ }
+ }
+ };
+ loop {
+ if !copy(&mut video_in, video_index, video_tb, out_video_tb, 0, video)? {
+ break;
+ }
+ if !copy(&mut audio_in, audio_index, audio_tb, out_audio_tb, 1, audio)? {
+ break;
+ }
+ }
+ out.write_trailer()
+ .map_err(|error| ffi::fail("write trailer", output, error))
}
#[cfg(test)]
@@ -305,7 +685,10 @@ mod tests {
#[test]
fn the_tape_effect_lands_on_the_mix_rate_before_shifting() {
let filters = speed_filters(2.0, false);
- assert_eq!(filters[0], "aresample=48000", "the shift must start from a known rate");
+ assert_eq!(
+ filters[0], "aresample=48000",
+ "the shift must start from a known rate"
+ );
assert!(filters[1].starts_with("asetrate=48000*"));
assert_eq!(filters[2], "aresample=48000");
}
@@ -320,8 +703,14 @@ mod tests {
#[test]
fn several_clips_mix_without_normalisation() {
let graph = mix_graph(&[clip("a.mp4"), clip("b.mp4")], 3.0).expect("valid");
- assert!(graph.contains("amix=inputs=2:normalize=0"), "graph was: {graph}");
- assert!(graph.contains("[0:a]") && graph.contains("[1:a]"), "graph was: {graph}");
+ assert!(
+ graph.contains("amix=inputs=2:normalize=0"),
+ "graph was: {graph}"
+ );
+ assert!(
+ graph.contains("[0:a]") && graph.contains("[1:a]"),
+ "graph was: {graph}"
+ );
}
#[test]
@@ -353,7 +742,10 @@ mod tests {
let chain_at = graph.find("highpass").expect("chain present");
let volume_at = graph.find(",volume=").expect("volume present");
let fade_at = graph.find(",afade=t=out").expect("fade present");
- assert!(chain_at < volume_at && volume_at < fade_at, "graph was: {graph}");
+ assert!(
+ chain_at < volume_at && volume_at < fade_at,
+ "graph was: {graph}"
+ );
}
#[test]
@@ -362,7 +754,10 @@ mod tests {
let mut hostile = clip("a.mp4");
hostile.filter_chain = bad.to_owned();
assert!(
- matches!(mix_graph(&[hostile], 2.0), Err(Error::InvalidFilterChain { .. })),
+ matches!(
+ mix_graph(&[hostile], 2.0),
+ Err(Error::InvalidFilterChain { .. })
+ ),
"{bad:?} should have been refused",
);
}
@@ -370,7 +765,11 @@ mod tests {
#[test]
fn ordinary_chains_pass_validation() {
- for good in ["", "highpass=f=80,equalizer=f=300:t=q:w=1.1:g=-1.4", "volume=0.5"] {
+ for good in [
+ "",
+ "highpass=f=80,equalizer=f=300:t=q:w=1.1:g=-1.4",
+ "volume=0.5",
+ ] {
assert!(validate_chain(good).is_ok(), "{good:?} should be fine");
}
}
diff --git a/engine/crates/concat-media/src/binaries.rs b/engine/crates/concat-media/src/binaries.rs
deleted file mode 100644
index ee26352c..00000000
--- a/engine/crates/concat-media/src/binaries.rs
+++ /dev/null
@@ -1,57 +0,0 @@
-// SPDX-License-Identifier: AGPL-3.0-or-later
-// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
-
-//! Where the FFmpeg binaries are.
-//!
-//! By default they are looked up on `PATH`, which is what a developer with
-//! FFmpeg installed expects. An application that ships its own copy calls
-//! [`set_binaries`] once at startup and everything downstream picks it up,
-//! because every process this crate spawns goes through [`ffmpeg`] or
-//! [`ffprobe`] rather than naming the program itself.
-//!
-//! Set once, at startup, before any decoding. Later calls are ignored rather
-//! than racing: a half-switched pair - old encoder, new decoder - would be a
-//! genuinely confusing thing to debug.
-
-use std::path::{Path, PathBuf};
-use std::sync::OnceLock;
-
-/// One cell for the pair, so two racing first-callers can never produce a
-/// mixed set - the decoder from one install, the prober from another.
-static BINARIES: OnceLock<(PathBuf, PathBuf)> = OnceLock::new();
-
-/// Points the crate at a specific pair of binaries.
-///
-/// Returns `false` if they were already resolved, in which case nothing
-/// changed. Call this before opening any media.
-pub fn set_binaries(ffmpeg: impl Into<PathBuf>, ffprobe: impl Into<PathBuf>) -> bool {
- BINARIES.set((ffmpeg.into(), ffprobe.into())).is_ok()
-}
-
-fn resolved() -> &'static (PathBuf, PathBuf) {
- BINARIES.get_or_init(|| (PathBuf::from("ffmpeg"), PathBuf::from("ffprobe")))
-}
-
-/// The `ffmpeg` binary to run. Defaults to whatever `PATH` finds.
-pub fn ffmpeg() -> &'static Path {
- &resolved().0
-}
-
-/// The `ffprobe` binary to run. Defaults to whatever `PATH` finds.
-pub fn ffprobe() -> &'static Path {
- &resolved().1
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
-
- #[test]
- fn defaults_to_the_bare_name_so_path_resolves_it() {
- // Deliberately does not call set_binaries: these statics are global,
- // and a test that set them would change the behaviour of every other
- // test in the binary depending on the order they happened to run in.
- assert_eq!(ffmpeg().as_os_str(), "ffmpeg");
- assert_eq!(ffprobe().as_os_str(), "ffprobe");
- }
-}
diff --git a/engine/crates/concat-media/src/decode.rs b/engine/crates/concat-media/src/decode.rs
index 7a508c78..72014f0e 100644
--- a/engine/crates/concat-media/src/decode.rs
+++ b/engine/crates/concat-media/src/decode.rs
@@ -2,25 +2,35 @@
// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
//! Pulling RGBA frames out of a file.
+//!
+//! One decoder does everything the old subprocess did with command-line
+//! flags: it seeks to a start point and discards up to it frame-accurately,
+//! turns the picture the way its rotation asks, scales, runs the clip's
+//! effect chain through libavfilter, paces frames to a requested output rate
+//! by duplicating and dropping, repeats a still forever, and stops after a
+//! frame budget. Every frame comes with its real presentation timestamp.
+//!
+//! ## On the unsafe in here
+//!
+//! There is none: the wrapper crate owns every pointer. What this module
+//! owns is the order things happen in.
-use std::io::Read;
use std::path::{Path, PathBuf};
-use std::process::{Child, ChildStdout, Stdio};
use concat_core::frame::Frame;
use concat_core::time::{FrameRate, Rational};
+use ffmpeg_the_third as ffmpeg;
+use ffmpeg_the_third::codec::decoder;
+use ffmpeg_the_third::filter;
+use ffmpeg_the_third::format::{self, Pixel};
+use ffmpeg_the_third::util::frame::video::Video;
use crate::error::{Error, Result};
-use crate::probe;
-use crate::process::{StderrTail, base_command};
-
-/// Name used in error messages. The binary actually run comes from
-/// [`crate::binaries::ffmpeg`], which may be a bundled copy.
-const FFMPEG: &str = "ffmpeg";
+use crate::ffi;
/// Anything that yields frames in order.
///
-/// The engine is written against this rather than against [`FfmpegDecoder`], so
+/// The engine is written against this rather than against [`Decoder`], so
/// that a GPU-decode backend can be dropped in later, and so that tests can
/// feed synthetic frames without touching the disk.
pub trait FrameSource {
@@ -33,23 +43,14 @@ pub trait FrameSource {
/// The next frame, or `Ok(None)` at the end of the stream.
fn next_frame(&mut self) -> Result<Option<Frame>>;
- /// When the frame just returned is meant to be shown.
- ///
- /// `None` means the source cannot say. The subprocess backend never can -
- /// raw video carries no timestamps - which is exactly why variable frame
- /// rate material desyncs through a pipe. A linked decoder reports the real
- /// presentation timestamp.
+ /// When the frame just returned is meant to be shown, in the media's
+ /// own clock. `None` when the container did not say.
fn position(&self) -> Option<Rational> {
None
}
}
/// A source that can jump to an arbitrary point.
-///
-/// Deliberately separate from [`FrameSource`]. Seeking over a pipe would mean
-/// respawning the process and decoding forward from a keyframe, which is not
-/// the same operation and should not be able to masquerade as one - so the
-/// subprocess decoder simply does not implement this.
pub trait SeekableSource: FrameSource {
/// Moves to `to`, or the nearest decodable point before it.
///
@@ -65,11 +66,12 @@ pub struct DecodeOptions {
pub start: Option<Rational>,
/// Stop after this many frames.
pub max_frames: Option<u64>,
- /// Scale to this size. Defaults to the file's own dimensions.
+ /// Scale to this size. Defaults to the file's own displayed dimensions.
pub size: Option<(u32, u32)>,
- /// Resample to this frame rate, duplicating or dropping frames.
+ /// Pace output to this frame rate, duplicating or dropping source frames
+ /// so that exactly one comes out per output frame instant.
pub frame_rate: Option<FrameRate>,
- /// Repeat the input forever.
+ /// Repeat the last frame forever.
///
/// A still image is a one-frame stream: decode it normally and you get a
/// single frame and then end-of-stream, which on a timeline means the
@@ -81,10 +83,9 @@ pub struct DecodeOptions {
///
/// This is how video effects reach the pixels: the same "one FFmpeg
/// string" design the audio filters use, so there is no second effect
- /// implementation to drift from. The decoder appends a final scale back
- /// to the requested size, because raw video has no framing - a chain that
- /// changed the frame size would not fail, it would silently shear every
- /// frame after the first.
+ /// implementation to drift from. A final scale back to the requested
+ /// size follows the chain, so an effect that changes the frame size
+ /// cannot change what the caller receives.
pub filter_chain: Option<String>,
}
@@ -107,13 +108,13 @@ impl DecodeOptions {
self
}
- /// Resamples output to `rate`.
+ /// Paces output to `rate`.
pub fn at_rate(mut self, rate: FrameRate) -> Self {
self.frame_rate = Some(rate);
self
}
- /// Repeats the input forever. See [`DecodeOptions::looping`].
+ /// Repeats the last frame forever. See [`DecodeOptions::looping`].
pub fn repeating(mut self) -> Self {
self.looping = true;
self
@@ -127,145 +128,351 @@ impl DecodeOptions {
}
}
-/// The `-vf` argument for these options, or `None` when nothing filters.
+/// The filtergraph between the decoder and the caller, as one string.
///
-/// Scale-to-fit first, then the effect chain at output resolution (cheaper
-/// than filtering the source size, and parameters mean the same thing at
-/// every export size), then the guard scale that pins the frame size the pipe
-/// was promised.
-fn video_filter(options: &DecodeOptions, width: u32, height: u32) -> Option<String> {
+/// Rotation first, so everything downstream sees the picture the way a
+/// player would; then scale-to-fit; then the effect chain at output
+/// resolution (cheaper than filtering the source size, and parameters mean
+/// the same thing at every export size); then the guard scale that pins the
+/// frame size the caller was promised; then RGBA.
+fn video_filter(rotation: i64, options: &DecodeOptions, width: u32, height: u32) -> String {
let mut parts: Vec<String> = Vec::new();
- if options.size.is_some() {
- parts.push(format!("scale={width}:{height}"));
+ if let Some(turn) = ffi::rotation_filters(rotation) {
+ parts.push(turn.to_owned());
}
+ parts.push(format!("scale={width}:{height}:flags=bilinear"));
if let Some(chain) = &options.filter_chain {
parts.push(chain.clone());
- parts.push(format!("scale={width}:{height}"));
+ parts.push(format!("scale={width}:{height}:flags=bilinear"));
}
- (!parts.is_empty()).then(|| parts.join(","))
+ parts.push("format=rgba".to_owned());
+ parts.join(",")
}
-/// Decodes a file by piping raw RGBA out of an `ffmpeg` child process.
-///
-/// The child's stderr is drained on its own thread and its tail is folded into
-/// [`Error::Exited`] - a packaged app has no terminal, so an inherited stderr
-/// would send the one useful line FFmpeg prints into the void.
-pub struct FfmpegDecoder {
+/// One decoded picture, before the filtergraph.
+struct Source {
+ frame: Video,
+ pts: Option<Rational>,
+}
+
+/// Decodes a file through libavcodec and libavfilter.
+pub struct Decoder {
path: PathBuf,
- child: Child,
- stdout: ChildStdout,
- stderr: StderrTail,
+ input: format::context::Input,
+ stream: usize,
+ time_base: ffmpeg::Rational,
+ decoder: decoder::Video,
+ rotation: i64,
+ options: DecodeOptions,
width: u32,
height: u32,
- /// Reused between frames so decoding does not allocate per frame.
- buffer: Vec<u8>,
+ /// Built on the first frame, once its format and size are known;
+ /// rebuilt if a later frame differs.
+ graph: Option<(Pixel, u32, u32, filter::Graph)>,
+ /// Frames before this instant are decoded and discarded - the exact
+ /// half of a seek, after the container landed on a keyframe.
+ discard_before: Option<Rational>,
+ /// The output frame instant the pacer is at, when pacing.
+ tick: u64,
+ origin: Rational,
+ /// The source frame the pacer is showing, and the one after it.
+ current: Option<Source>,
+ pending: Option<Source>,
+ source_done: bool,
produced: u64,
- limit: Option<u64>,
- finished: bool,
+ position: Option<Rational>,
}
-impl FfmpegDecoder {
+impl Decoder {
/// Opens `path` for decoding.
- ///
- /// If `options.size` is not set this probes the file first to learn its
- /// dimensions, because the pipe carries raw pixels with no header to say
- /// how big a frame is.
pub fn open(path: impl AsRef<Path>, options: &DecodeOptions) -> Result<Self> {
+ ffi::init();
// The same slot-escape rule the audio graph enforces: a chain with
- // `;` or `[..]` is no longer a filter applied to this clip. Video
- // rode without the guard only because every chain came from our own
- // catalogue; "validated, not trusted" should not depend on that.
+ // `;` or `[..]` is no longer a filter applied to this clip.
if let Some(chain) = &options.filter_chain {
crate::audio::validate_chain(chain)?;
}
let path = path.as_ref();
+ let input = ffmpeg::format::input(path).map_err(|error| ffi::fail("open", path, error))?;
+ let stream = input
+ .streams()
+ .best(ffmpeg::media::Type::Video)
+ .ok_or_else(|| Error::NoVideoStream {
+ path: path.to_path_buf(),
+ })?;
+ let stream_index = stream.index();
+ let time_base = stream.time_base();
+ let rotation = ffi::rotation(&stream);
+ let parameters = stream.parameters();
+ let coded = (parameters.width(), parameters.height());
+
+ let context = ffmpeg::codec::Context::from_parameters(parameters)
+ .map_err(|error| ffi::fail("codec parameters", path, error))?;
+ let decoder = context
+ .decoder()
+ .video()
+ .map_err(|error| ffi::fail("open decoder", path, error))?;
+
let (width, height) = match options.size {
Some(size) => size,
- None => {
- let info = probe::probe(path)?;
- let video = info.require_video()?;
- (video.width, video.height)
- }
+ None => ffi::displayed(coded.0, coded.1, rotation),
};
+ if width == 0 || height == 0 {
+ return Err(Error::Probe {
+ path: path.to_path_buf(),
+ detail: "video stream has no usable size".to_owned(),
+ });
+ }
- let mut command = base_command(crate::binaries::ffmpeg());
-
- // Both of these are *input* options and have to precede -i.
- if options.looping {
- command.args(["-loop", "1"]);
+ let mut this = Self {
+ path: path.to_path_buf(),
+ input,
+ stream: stream_index,
+ time_base,
+ decoder,
+ rotation,
+ options: options.clone(),
+ width,
+ height,
+ graph: None,
+ discard_before: None,
+ tick: 0,
+ origin: Rational::ZERO,
+ current: None,
+ pending: None,
+ source_done: false,
+ produced: 0,
+ position: None,
+ };
+ if let Some(start) = options.start
+ && !start.is_zero()
+ {
+ this.jump(start)?;
}
+ Ok(this)
+ }
- // -ss before -i is the fast seek: FFmpeg jumps to the nearest keyframe
- // and decodes forward, instead of decoding everything and discarding it.
- if let Some(start) = options.start {
- command.args(["-ss", &format!("{:.6}", start.as_f64())]);
+ /// How many frames have been produced so far.
+ pub const fn produced(&self) -> u64 {
+ self.produced
+ }
+
+ /// The container seek, and the state reset that goes with it.
+ fn jump(&mut self, to: Rational) -> Result<()> {
+ let target = ffi::av_ticks(to);
+ self.input
+ .seek(target, ..=target)
+ .map_err(|error| ffi::fail("seek", &self.path, error))?;
+ self.decoder.flush();
+ self.discard_before = Some(to);
+ self.origin = to;
+ self.tick = 0;
+ self.current = None;
+ self.pending = None;
+ self.source_done = false;
+ self.position = None;
+ Ok(())
+ }
+
+ /// The next picture out of the codec, or `None` at the end of the file.
+ /// Frames before the discard point never come out of here.
+ fn next_source(&mut self) -> Result<Option<Source>> {
+ loop {
+ let mut frame = Video::empty();
+ match self.decoder.receive_frame(&mut frame) {
+ Ok(()) => {
+ let pts = frame
+ .timestamp()
+ .and_then(|ticks| ffi::seconds(ticks, self.time_base));
+ if let (Some(before), Some(pts)) = (self.discard_before, pts)
+ && pts < before
+ {
+ continue;
+ }
+ self.discard_before = None;
+ return Ok(Some(Source { frame, pts }));
+ }
+ Err(ffmpeg::Error::Eof) => return Ok(None),
+ Err(error) if ffi::is_again(&error) => {}
+ Err(error) => return Err(ffi::fail("decode", &self.path, error)),
+ }
+
+ // The codec wants another packet. Packets for other streams are
+ // skipped; the end of the file flushes the codec so its last few
+ // frames come out at all.
+ loop {
+ let mut packet = ffmpeg::Packet::empty();
+ match packet.read(&mut self.input) {
+ Ok(()) => {
+ if packet.stream() != self.stream {
+ continue;
+ }
+ match self.decoder.send_packet(&packet) {
+ Ok(()) => break,
+ Err(error) if ffi::is_again(&error) => break,
+ Err(error) => return Err(ffi::fail("send packet", &self.path, error)),
+ }
+ }
+ Err(ffmpeg::Error::Eof) => {
+ let _ = self.decoder.send_eof();
+ break;
+ }
+ Err(error) => return Err(ffi::fail("read", &self.path, error)),
+ }
+ }
}
- command.arg("-i").arg(path);
+ }
- if let Some(limit) = options.max_frames {
- command.args(["-frames:v", &limit.to_string()]);
+ /// Runs one decoded picture through the graph and copies the RGBA out.
+ fn convert(&mut self, source: &Video) -> Result<Frame> {
+ let key = (source.format(), source.width(), source.height());
+ if self
+ .graph
+ .as_ref()
+ .is_none_or(|(format, width, height, _)| (*format, *width, *height) != key)
+ {
+ let graph = self.build_graph(key.0, key.1, key.2)?;
+ self.graph = Some((key.0, key.1, key.2, graph));
}
- if let Some(filters) = video_filter(options, width, height) {
- command.args(["-vf", &filters]);
+ let (.., graph) = self.graph.as_mut().expect("just built");
+
+ {
+ let mut context = graph.get("in").expect("the graph has an input");
+ context
+ .source()
+ .add(source)
+ .map_err(|error| ffi::fail("filter", &self.path, error))?;
}
- if let Some(rate) = options.frame_rate {
- command.args(["-r", &fraction(rate)]);
+ let mut filtered = Video::empty();
+ {
+ let mut context = graph.get("out").expect("the graph has an output");
+ context
+ .sink()
+ .frame(&mut filtered)
+ .map_err(|error| ffi::fail("filter output", &self.path, error))?;
}
- command
- // `pipe:1`, not `-`: the bare dash resolves to the `fd:` protocol
- // on modern FFmpeg, which a trimmed bundled build may not include.
- .args(["-f", "rawvideo", "-pix_fmt", "rgba", "pipe:1"])
- .stdout(Stdio::piped())
- .stdin(Stdio::null())
- .stderr(Stdio::piped());
-
- let mut child =
- command.spawn().map_err(|source| Error::Spawn { program: FFMPEG, source })?;
- let stdout = child.stdout.take().expect("stdout was piped");
- let stderr = StderrTail::drain(&mut child);
-
- Ok(Self {
- path: path.to_path_buf(),
- child,
- stdout,
- stderr,
- width,
- height,
- buffer: vec![0u8; Frame::byte_len(width, height)],
- produced: 0,
- limit: options.max_frames,
- finished: false,
+ // The sink's picture is RGBA already; only its row padding differs
+ // from the packed buffer a Frame is.
+ let width = filtered.width();
+ let height = filtered.height();
+ let row = width as usize * 4;
+ let stride = filtered.stride(0);
+ let data = filtered.data(0);
+ let mut pixels = Vec::with_capacity(row * height as usize);
+ for y in 0..height as usize {
+ pixels.extend_from_slice(&data[y * stride..y * stride + row]);
+ }
+ Frame::from_rgba(width, height, pixels).ok_or_else(|| Error::Probe {
+ path: self.path.clone(),
+ detail: "the filtergraph produced a frame of the wrong size".to_owned(),
})
}
- /// How many frames have been produced so far.
- pub const fn produced(&self) -> u64 {
- self.produced
+ fn build_graph(&self, format: Pixel, width: u32, height: u32) -> Result<filter::Graph> {
+ let mut graph = filter::Graph::new();
+ let args = format!(
+ "video_size={width}x{height}:pix_fmt={}:time_base={}/{}:pixel_aspect=1/1",
+ Into::<ffmpeg::sys::AVPixelFormat>::into(format).0,
+ self.time_base.numerator(),
+ self.time_base.denominator()
+ );
+ let missing = |name: &str| Error::Missing {
+ what: "filter",
+ name: name.to_owned(),
+ };
+ graph
+ .add(
+ &filter::find("buffer").ok_or_else(|| missing("buffer"))?,
+ "in",
+ &args,
+ )
+ .map_err(|error| ffi::fail("buffer source", &self.path, error))?;
+ graph
+ .add(
+ &filter::find("buffersink").ok_or_else(|| missing("buffersink"))?,
+ "out",
+ "",
+ )
+ .map_err(|error| ffi::fail("buffer sink", &self.path, error))?;
+ let spec = video_filter(self.rotation, &self.options, self.width, self.height);
+ graph
+ .output("in", 0)
+ .and_then(|parser| parser.input("out", 0))
+ .and_then(|parser| parser.parse(&spec))
+ .map_err(|error| ffi::fail("filter graph", &self.path, error))?;
+ graph
+ .validate()
+ .map_err(|error| ffi::fail("filter graph", &self.path, error))?;
+ Ok(graph)
}
- /// Reaps the child and turns a non-zero exit into an error.
- fn finish(&mut self) -> Result<()> {
- if self.finished {
- return Ok(());
+ /// Without pacing: every source frame, in order.
+ fn next_unpaced(&mut self) -> Result<Option<Frame>> {
+ let Some(source) = self.next_source()? else {
+ if self.options.looping
+ && let Some(last) = self.current.as_ref()
+ {
+ self.position = last.pts;
+ let frame = last.frame.clone();
+ return Ok(Some(self.convert(&frame)?));
+ }
+ return Ok(None);
+ };
+ self.position = source.pts;
+ let frame = self.convert(&source.frame)?;
+ self.current = Some(source);
+ Ok(Some(frame))
+ }
+
+ /// With pacing: the source frame on screen at the next output instant.
+ fn next_paced(&mut self, rate: FrameRate) -> Result<Option<Frame>> {
+ let target = self.origin + rate.frame_duration() * Rational::from_int(self.tick as i64);
+
+ // Advance until `pending` is the first frame after the target, so
+ // `current` is the one on screen at it. A frame with no timestamp
+ // counts as the next one in line.
+ while !self.source_done
+ && self
+ .pending
+ .as_ref()
+ .is_none_or(|next| next.pts.is_none_or(|pts| pts <= target))
+ {
+ if let Some(next) = self.pending.take() {
+ self.current = Some(next);
+ }
+ match self.next_source()? {
+ Some(source) => self.pending = Some(source),
+ None => self.source_done = true,
+ }
+ }
+ if self.current.is_none() {
+ // Nothing at or before the target: the first frame of the file
+ // sits after it, and shows from the first instant, as it would
+ // through any player.
+ self.current = self.pending.take();
}
- self.finished = true;
- let status = self.child.wait().map_err(|source| Error::Io { program: FFMPEG, source })?;
- if status.success() {
- Ok(())
- } else {
- Err(Error::Exited {
- program: FFMPEG,
- path: self.path.clone(),
- status,
- stderr: self.stderr.summary(),
- })
+ let Some(current) = self.current.as_ref() else {
+ return Ok(None);
+ };
+ if self.source_done && self.pending.is_none() && !self.options.looping {
+ // Past the last frame. It has already shown once at its own
+ // instant; the file is over.
+ if let Some(pts) = current.pts
+ && pts + rate.frame_duration() <= target
+ {
+ return Ok(None);
+ }
}
+ self.position = Some(target);
+ self.tick += 1;
+ let frame = current.frame.clone();
+ Ok(Some(self.convert(&frame)?))
}
}
-impl FrameSource for FfmpegDecoder {
+impl FrameSource for Decoder {
fn width(&self) -> u32 {
self.width
}
@@ -274,73 +481,39 @@ impl FrameSource for FfmpegDecoder {
self.height
}
- fn next_frame(&mut self) -> Result<Option<Frame>> {
- if self.finished || self.limit.is_some_and(|limit| self.produced >= limit) {
- return Ok(None);
- }
-
- // read_exact would collapse a clean end-of-stream and a truncated frame
- // into the same error, and those mean very different things here.
- let mut filled = 0;
- while filled < self.buffer.len() {
- match self.stdout.read(&mut self.buffer[filled..]) {
- Ok(0) => break,
- Ok(read) => filled += read,
- Err(error) if error.kind() == std::io::ErrorKind::Interrupted => continue,
- Err(source) => return Err(Error::Io { program: FFMPEG, source }),
- }
- }
+ fn position(&self) -> Option<Rational> {
+ self.position
+ }
- if filled == 0 {
- self.finish()?;
+ fn next_frame(&mut self) -> Result<Option<Frame>> {
+ if self
+ .options
+ .max_frames
+ .is_some_and(|limit| self.produced >= limit)
+ {
return Ok(None);
}
- if filled < self.buffer.len() {
- let error = Error::PartialFrame {
- path: self.path.clone(),
- got: filled,
- want: self.buffer.len(),
- };
- let _ = self.finish();
- return Err(error);
+ let frame = match self.options.frame_rate {
+ Some(rate) => self.next_paced(rate)?,
+ None => self.next_unpaced()?,
+ };
+ if frame.is_some() {
+ self.produced += 1;
}
-
- self.produced += 1;
- Ok(Some(
- Frame::from_rgba(self.width, self.height, self.buffer.clone())
- .expect("buffer is sized from the same width and height"),
- ))
+ Ok(frame)
}
}
-impl Drop for FfmpegDecoder {
- fn drop(&mut self) {
- // Dropping a decoder mid-stream is normal - a scrub replaces it. Kill
- // the child rather than leaking an ffmpeg process per seek.
- if !self.finished {
- let _ = self.child.kill();
- let _ = self.child.wait();
- }
+impl SeekableSource for Decoder {
+ fn seek(&mut self, to: Rational) -> Result<()> {
+ self.jump(to)
}
}
-/// Formats a frame rate as an exact `num/den` fraction. FFmpeg accepts these,
-/// so 29.97 stays 30000/1001 all the way through.
-fn fraction(rate: FrameRate) -> String {
- let fps = rate.fps();
- format!("{}/{}", fps.numerator(), fps.denominator())
-}
-
#[cfg(test)]
mod tests {
use super::*;
- #[test]
- fn frame_rates_reach_ffmpeg_as_exact_fractions() {
- assert_eq!(fraction(FrameRate::NTSC_30), "30000/1001");
- assert_eq!(fraction(FrameRate::THIRTY), "30/1");
- }
-
#[test]
fn options_build_up() {
let options = DecodeOptions::default()
@@ -354,16 +527,20 @@ mod tests {
#[test]
fn a_filter_chain_is_fenced_by_the_guard_scale() {
- let options = DecodeOptions::default().scaled_to(640, 360).filtered("hue=s=0");
+ let options = DecodeOptions::default()
+ .scaled_to(640, 360)
+ .filtered("hue=s=0");
assert_eq!(
- video_filter(&options, 640, 360).as_deref(),
- Some("scale=640:360,hue=s=0,scale=640:360"),
+ video_filter(0, &options, 640, 360),
+ "scale=640:360:flags=bilinear,hue=s=0,scale=640:360:flags=bilinear,format=rgba",
);
}
#[test]
- fn no_size_and_no_chain_means_no_vf_at_all() {
- assert_eq!(video_filter(&DecodeOptions::default(), 1920, 1080), None);
+ fn rotation_comes_first() {
+ let options = DecodeOptions::default();
+ assert!(video_filter(90, &options, 1080, 1920).starts_with("transpose=clock,"));
+ assert!(video_filter(0, &options, 1920, 1080).starts_with("scale="));
}
#[test]
@@ -372,24 +549,21 @@ mod tests {
}
#[test]
- fn a_chain_that_escapes_its_slot_is_refused_before_anything_spawns() {
+ fn a_chain_that_escapes_its_slot_is_refused_before_anything_opens() {
for bad in ["hue=s=0;movie=x", "split[a][b]", "a\nb"] {
- let result = FfmpegDecoder::open(
+ let result = Decoder::open(
"irrelevant.mp4",
&DecodeOptions::default().scaled_to(64, 64).filtered(bad),
);
assert!(
- matches!(result, Err(crate::error::Error::InvalidFilterChain { .. })),
+ matches!(result, Err(Error::InvalidFilterChain { .. })),
"{bad:?} should have been refused",
);
}
}
#[test]
- fn a_missing_ffmpeg_is_a_spawn_error_not_a_panic() {
- // Nothing to assert about the happy path without a fixture file, but
- // the error path must stay a Result.
- let result = FfmpegDecoder::open("does-not-exist.mp4", &DecodeOptions::default());
- assert!(result.is_err());
+ fn a_missing_file_is_an_error_not_a_panic() {
+ assert!(Decoder::open("does-not-exist.mp4", &DecodeOptions::default()).is_err());
}
}
diff --git a/engine/crates/concat-media/src/encode.rs b/engine/crates/concat-media/src/encode.rs
index e83824cc..33334639 100644
--- a/engine/crates/concat-media/src/encode.rs
+++ b/engine/crates/concat-media/src/encode.rs
@@ -3,19 +3,18 @@
//! Writing RGBA frames back out to a file.
-use std::io::Write;
use std::path::{Path, PathBuf};
-use std::process::{Child, ChildStdin, Stdio};
use concat_core::frame::Frame;
use concat_core::time::FrameRate;
+use ffmpeg_the_third as ffmpeg;
+use ffmpeg_the_third::codec::encoder;
+use ffmpeg_the_third::format::{self, Pixel};
+use ffmpeg_the_third::software::scaling;
+use ffmpeg_the_third::util::frame::video::Video;
use crate::error::{Error, Result};
-use crate::process::{StderrTail, base_command};
-
-/// Name used in error messages. The binary actually run comes from
-/// [`crate::binaries::ffmpeg`], which may be a bundled copy.
-const FFMPEG: &str = "ffmpeg";
+use crate::ffi;
/// Anything that accepts finished frames.
///
@@ -55,19 +54,35 @@ impl Default for EncodeOptions {
}
}
-/// Encodes by piping raw RGBA into an `ffmpeg` child process.
-pub struct FfmpegEncoder {
+/// The output pixel formats an encoder can be asked for, by FFmpeg's names.
+fn pixel_format(name: &str) -> Option<Pixel> {
+ Some(match name {
+ "yuv420p" => Pixel::YUV420P,
+ "yuv422p" => Pixel::YUV422P,
+ "yuv444p" => Pixel::YUV444P,
+ "nv12" => Pixel::NV12,
+ "rgba" => Pixel::RGBA,
+ _ => return None,
+ })
+}
+
+/// Encodes through libavcodec into a container libavformat writes.
+pub struct Encoder {
path: PathBuf,
- child: Child,
- /// Taken in `finish` - closing the pipe is what tells FFmpeg to stop.
- stdin: Option<ChildStdin>,
- stderr: StderrTail,
+ output: format::context::Output,
+ encoder: encoder::video::Encoder,
+ scaler: scaling::Context,
+ /// The encoder's time base, and the stream's after the header was
+ /// written - the muxer is free to pick its own.
+ encoder_time_base: ffmpeg::Rational,
+ stream_time_base: ffmpeg::Rational,
width: u32,
height: u32,
written: u64,
+ finished: bool,
}
-impl FfmpegEncoder {
+impl Encoder {
/// Opens `path` for writing, overwriting anything already there.
pub fn create(
path: impl AsRef<Path>,
@@ -76,44 +91,84 @@ impl FfmpegEncoder {
frame_rate: FrameRate,
options: &EncodeOptions,
) -> Result<Self> {
+ ffi::init();
let path = path.as_ref();
let fps = frame_rate.fps();
+ let rate = ffmpeg::Rational::new(fps.numerator() as i32, fps.denominator() as i32);
+ let time_base = rate.invert();
+
+ let codec = encoder::find_by_name(&options.codec).ok_or_else(|| Error::Missing {
+ what: "encoder",
+ name: options.codec.clone(),
+ })?;
+ let pixel_format = pixel_format(&options.pixel_format).ok_or_else(|| Error::Missing {
+ what: "pixel format",
+ name: options.pixel_format.clone(),
+ })?;
+
+ let mut output =
+ ffmpeg::format::output(path).map_err(|error| ffi::fail("create", path, error))?;
+ let global_header = output
+ .format()
+ .flags()
+ .contains(format::Flags::GLOBAL_HEADER);
+
+ let mut video = ffmpeg::codec::Context::new_with_codec(codec)
+ .encoder()
+ .video()
+ .map_err(|error| ffi::fail("video encoder", path, error))?;
+ video.set_width(width);
+ video.set_height(height);
+ video.set_format(pixel_format);
+ video.set_time_base(time_base);
+ video.set_frame_rate(Some(rate));
+ if global_header {
+ video.set_flags(ffmpeg::codec::Flags::GLOBAL_HEADER);
+ }
+ let encoder = video
+ .open_with(ffmpeg::dict! {
+ "preset" => options.preset.as_str(),
+ "crf" => &options.crf.to_string(),
+ })
+ .map_err(|error| ffi::fail("open encoder", path, error))?;
+
+ {
+ let mut stream = output
+ .add_stream(codec)
+ .map_err(|error| ffi::fail("add stream", path, error))?;
+ stream.copy_parameters_from_context(&encoder);
+ stream.set_time_base(time_base);
+ }
+ output
+ .write_header_with(ffmpeg::dict! { "movflags" => "+faststart" })
+ .map_err(|error| ffi::fail("write header", path, error))?;
+ let stream_time_base = output
+ .stream(0)
+ .map(|stream| stream.time_base())
+ .unwrap_or(time_base);
- let mut command = base_command(crate::binaries::ffmpeg());
- command
- .arg("-y")
- // Input: what is coming down the pipe.
- .args(["-f", "rawvideo", "-pix_fmt", "rgba"])
- .args(["-s", &format!("{width}x{height}")])
- .args(["-r", &format!("{}/{}", fps.numerator(), fps.denominator())])
- // `pipe:0`, not `-`: since FFmpeg 6 the bare dash resolves to the
- // `fd:` protocol, which a trimmed build (like the one we bundle)
- // may not include. The pipe protocol is what we actually mean.
- .args(["-i", "pipe:0"])
- // Output.
- .args(["-c:v", &options.codec])
- .args(["-preset", &options.preset])
- .args(["-crf", &options.crf.to_string()])
- .args(["-pix_fmt", &options.pixel_format])
- .args(["-movflags", "+faststart"])
- .arg(path)
- .stdin(Stdio::piped())
- .stdout(Stdio::null())
- .stderr(Stdio::piped());
-
- let mut child =
- command.spawn().map_err(|source| Error::Spawn { program: FFMPEG, source })?;
- let stdin = child.stdin.take().expect("stdin was piped");
- let stderr = StderrTail::drain(&mut child);
+ let scaler = scaling::Context::get(
+ Pixel::RGBA,
+ width,
+ height,
+ pixel_format,
+ width,
+ height,
+ scaling::Flags::BILINEAR,
+ )
+ .map_err(|error| ffi::fail("scaler", path, error))?;
Ok(Self {
path: path.to_path_buf(),
- child,
- stdin: Some(stdin),
- stderr,
+ output,
+ encoder,
+ scaler,
+ encoder_time_base: time_base,
+ stream_time_base,
width,
height,
written: 0,
+ finished: false,
})
}
@@ -121,13 +176,30 @@ impl FfmpegEncoder {
pub const fn written(&self) -> u64 {
self.written
}
+
+ /// Writes every packet the encoder has ready.
+ fn drain(&mut self) -> Result<()> {
+ loop {
+ let mut packet = ffmpeg::Packet::empty();
+ match self.encoder.receive_packet(&mut packet) {
+ Ok(()) => {
+ packet.set_stream(0);
+ packet.rescale_ts(self.encoder_time_base, self.stream_time_base);
+ packet
+ .write_interleaved(&mut self.output)
+ .map_err(|error| ffi::fail("write packet", &self.path, error))?;
+ }
+ Err(ffmpeg::Error::Eof) => return Ok(()),
+ Err(error) if ffi::is_again(&error) => return Ok(()),
+ Err(error) => return Err(ffi::fail("encode", &self.path, error)),
+ }
+ }
+ }
}
-impl FrameSink for FfmpegEncoder {
+impl FrameSink for Encoder {
fn write_frame(&mut self, frame: &Frame) -> Result<()> {
if frame.width() != self.width || frame.height() != self.height {
- // Raw video has no framing, so a wrong-sized frame would not fail
- // here - it would silently shear every frame after it.
return Err(Error::FrameSizeMismatch {
want_width: self.width,
want_height: self.height,
@@ -135,94 +207,157 @@ impl FrameSink for FfmpegEncoder {
got_height: frame.height(),
});
}
-
- let Some(stdin) = self.stdin.as_mut() else {
+ if self.finished {
return Err(Error::Io {
- program: FFMPEG,
+ path: self.path.clone(),
source: std::io::Error::other("encoder was already finished"),
});
- };
-
- if let Err(source) = stdin.write_all(frame.pixels()) {
- // A broken pipe means the encoder died; what it printed on the way
- // out is the informative error, not the EPIPE it left behind.
- if source.kind() == std::io::ErrorKind::BrokenPipe {
- drop(self.stdin.take());
- if let Ok(status) = self.child.wait()
- && !status.success()
- {
- return Err(Error::Exited {
- program: FFMPEG,
- path: self.path.clone(),
- status,
- stderr: self.stderr.summary(),
- });
- }
+ }
+
+ let mut rgba = Video::new(Pixel::RGBA, self.width, self.height);
+ {
+ let stride = rgba.stride(0);
+ let row = self.width as usize * 4;
+ let data = rgba.data_mut(0);
+ for (y, source) in frame.pixels().chunks_exact(row).enumerate() {
+ data[y * stride..y * stride + row].copy_from_slice(source);
}
- return Err(Error::Io { program: FFMPEG, source });
}
+ let mut converted = Video::empty();
+ self.scaler
+ .run(&rgba, &mut converted)
+ .map_err(|error| ffi::fail("convert", &self.path, error))?;
+ converted.set_pts(Some(self.written as i64));
+
+ self.encoder
+ .send_frame(&converted)
+ .map_err(|error| ffi::fail("encode", &self.path, error))?;
self.written += 1;
- Ok(())
+ self.drain()
}
fn finish(&mut self) -> Result<()> {
- // Closing stdin is the signal to flush and write the trailer.
- drop(self.stdin.take());
-
- let status = self.child.wait().map_err(|source| Error::Io { program: FFMPEG, source })?;
- if status.success() {
- Ok(())
- } else {
- Err(Error::Exited {
- program: FFMPEG,
- path: self.path.clone(),
- status,
- stderr: self.stderr.summary(),
- })
+ if self.finished {
+ return Ok(());
}
+ self.finished = true;
+ self.encoder
+ .send_eof()
+ .map_err(|error| ffi::fail("encode", &self.path, error))?;
+ self.drain()?;
+ self.output
+ .write_trailer()
+ .map_err(|error| ffi::fail("write trailer", &self.path, error))
}
}
-impl Drop for FfmpegEncoder {
- fn drop(&mut self) {
- if self.stdin.is_some() {
- // finish() was never called, so the output is garbage anyway.
- // Kill the child instead of blocking a drop on an encode flush.
- drop(self.stdin.take());
- let _ = self.child.kill();
- let _ = self.child.wait();
+/// Encodes one frame as a JPEG, for posters and thumbnails written to disk.
+///
+/// `quality` is the JPEG quantiser scale, 2 (best) to 31 (worst) - the
+/// `-q:v` of the command line.
+pub fn jpeg(frame: &Frame, quality: u8) -> Result<Vec<u8>> {
+ ffi::init();
+ let path = Path::new("<jpeg>");
+ let codec = encoder::find_by_name("mjpeg").ok_or_else(|| Error::Missing {
+ what: "encoder",
+ name: "mjpeg".to_owned(),
+ })?;
+ let mut video = ffmpeg::codec::Context::new_with_codec(codec)
+ .encoder()
+ .video()
+ .map_err(|error| ffi::fail("jpeg encoder", path, error))?;
+ video.set_width(frame.width());
+ video.set_height(frame.height());
+ video.set_format(Pixel::YUVJ420P);
+ video.set_time_base(ffmpeg::Rational::new(1, 25));
+ let quality = i32::from(quality.clamp(2, 31));
+ video.set_qmin(quality);
+ video.set_qmax(quality);
+ let mut encoder = video
+ .open()
+ .map_err(|error| ffi::fail("open jpeg encoder", path, error))?;
+
+ let mut rgba = Video::new(Pixel::RGBA, frame.width(), frame.height());
+ {
+ let stride = rgba.stride(0);
+ let row = frame.width() as usize * 4;
+ let data = rgba.data_mut(0);
+ for (y, source) in frame.pixels().chunks_exact(row).enumerate() {
+ data[y * stride..y * stride + row].copy_from_slice(source);
+ }
+ }
+ let mut scaler = scaling::Context::get(
+ Pixel::RGBA,
+ frame.width(),
+ frame.height(),
+ Pixel::YUVJ420P,
+ frame.width(),
+ frame.height(),
+ scaling::Flags::BILINEAR,
+ )
+ .map_err(|error| ffi::fail("scaler", path, error))?;
+ let mut converted = Video::empty();
+ scaler
+ .run(&rgba, &mut converted)
+ .map_err(|error| ffi::fail("convert", path, error))?;
+ converted.set_pts(Some(0));
+
+ encoder
+ .send_frame(&converted)
+ .map_err(|error| ffi::fail("encode", path, error))?;
+ encoder
+ .send_eof()
+ .map_err(|error| ffi::fail("encode", path, error))?;
+ let mut bytes = Vec::new();
+ loop {
+ let mut packet = ffmpeg::Packet::empty();
+ match encoder.receive_packet(&mut packet) {
+ Ok(()) => bytes.extend_from_slice(packet.data().unwrap_or(&[])),
+ Err(ffmpeg::Error::Eof) => break,
+ Err(error) if ffi::is_again(&error) => break,
+ Err(error) => return Err(ffi::fail("encode", path, error)),
}
}
+ Ok(bytes)
}
#[cfg(test)]
mod tests {
use super::*;
+ #[test]
+ fn a_frame_becomes_a_jpeg() {
+ let mut frame = Frame::black(32, 32);
+ frame.fill([200, 40, 40, 255]);
+ let bytes = jpeg(&frame, 4).expect("encodes");
+ assert!(bytes.starts_with(&[0xFF, 0xD8]), "a JPEG starts with SOI");
+ }
+
#[test]
fn defaults_are_a_playable_h264_file() {
let options = EncodeOptions::default();
assert_eq!(options.codec, "libx264");
- assert_eq!(options.pixel_format, "yuv420p", "yuv444 will not play in browsers");
+ assert_eq!(
+ options.pixel_format, "yuv420p",
+ "yuv444 will not play in browsers"
+ );
}
#[test]
fn a_wrong_sized_frame_is_rejected() {
- let Ok(mut encoder) = FfmpegEncoder::create(
- std::env::temp_dir().join("concat-encode-size-test.mp4"),
- 64,
- 64,
- FrameRate::THIRTY,
- &EncodeOptions::default(),
- ) else {
- // No FFmpeg on this machine; nothing to assert.
- return;
- };
+ let path = std::env::temp_dir().join("concat-encode-size-test.mp4");
+ let mut encoder =
+ Encoder::create(&path, 64, 64, FrameRate::THIRTY, &EncodeOptions::default())
+ .expect("the linked FFmpeg encodes h264");
let wrong = Frame::black(32, 32);
assert!(matches!(
encoder.write_frame(&wrong),
Err(Error::FrameSizeMismatch { got_width: 32, .. })
));
+ encoder.write_frame(&Frame::black(64, 64)).expect("writes");
+ encoder.finish().expect("finishes");
+ assert!(std::fs::metadata(&path).is_ok_and(|meta| meta.len() > 0));
+ let _ = std::fs::remove_file(&path);
}
}
diff --git a/engine/crates/concat-media/src/error.rs b/engine/crates/concat-media/src/error.rs
index f078e4aa..ec55c267 100644
--- a/engine/crates/concat-media/src/error.rs
+++ b/engine/crates/concat-media/src/error.rs
@@ -4,45 +4,32 @@
//! What can go wrong when talking to FFmpeg.
use std::path::PathBuf;
-use std::process::ExitStatus;
/// Shorthand for results in this crate.
pub type Result<T> = std::result::Result<T, Error>;
/// A media IO failure.
///
-/// The messages name the program and the file, because the first question you
-/// will ask six months from now is "which file, and what did FFmpeg say".
+/// The messages name the operation and the file, because the first question
+/// you will ask six months from now is "which file, and what did FFmpeg say".
#[derive(Debug, thiserror::Error)]
pub enum Error {
- /// The `ffmpeg` or `ffprobe` binary could not be started at all.
- #[error("could not run `{program}` - is FFmpeg installed and on PATH?")]
- Spawn {
- /// Which binary we tried to start.
- program: &'static str,
- /// The underlying OS error.
- #[source]
- source: std::io::Error,
- },
-
- /// The child process ran and exited unhappily.
- #[error("`{program}` exited with {status} while handling {path}: {stderr}")]
- Exited {
- /// Which binary failed.
- program: &'static str,
- /// The file it was working on.
+ /// A libav* call failed.
+ #[error("{operation} failed for {path}: {detail}")]
+ Ffi {
+ /// The libav function or step that failed.
+ operation: &'static str,
+ /// The file being worked on.
path: PathBuf,
- /// Its exit status.
- status: ExitStatus,
- /// The tail of what it printed - the actual reason, when there is one.
- stderr: String,
+ /// FFmpeg's own description of the error.
+ detail: String,
},
- /// An IO error moving bytes over the pipe.
- #[error("io error talking to `{program}`")]
+ /// An IO error on a file this crate reads or writes itself.
+ #[error("io error on {path}")]
Io {
- /// Which binary we were talking to.
- program: &'static str,
+ /// The file.
+ path: PathBuf,
/// The underlying OS error.
#[source]
source: std::io::Error,
@@ -58,8 +45,8 @@ pub enum Error {
detail: String,
},
- /// `ffprobe` returned something we could not make sense of.
- #[error("could not understand ffprobe output for {path}: {detail}")]
+ /// The container said something we could not make sense of.
+ #[error("could not understand {path}: {detail}")]
Probe {
/// The file being probed.
path: PathBuf,
@@ -74,26 +61,18 @@ pub enum Error {
path: PathBuf,
},
- /// The decoder stopped mid-frame, which means FFmpeg died partway through.
- #[error("{path} produced a partial frame ({got} of {want} bytes)")]
- PartialFrame {
- /// The file being decoded.
+ /// The file has no audio stream, but we were asked to decode sound.
+ #[error("{path} has no audio stream")]
+ NoAudioStream {
+ /// The file in question.
path: PathBuf,
- /// How many bytes arrived.
- got: usize,
- /// How many bytes a frame needs.
- want: usize,
},
- /// A libav* call failed. Only produced by the linked backend.
- #[error("{operation} failed for {path}: {detail}")]
- Ffi {
- /// The libav function that failed.
- operation: &'static str,
- /// The file being worked on.
+ /// A seek landed past every decodable frame.
+ #[error("{path} has no frame to show at the requested time")]
+ NoFrame {
+ /// The file being decoded.
path: PathBuf,
- /// FFmpeg's own description of the error code.
- detail: String,
},
/// A frame handed to an encoder was not the size the encoder was opened for.
@@ -108,4 +87,14 @@ pub enum Error {
/// Height of the offending frame.
got_height: u32,
},
+
+ /// The linked FFmpeg was built without something this operation needs -
+ /// an encoder, a filter, a muxer.
+ #[error("the linked FFmpeg has no {what} {name:?}")]
+ Missing {
+ /// "encoder", "filter", "muxer".
+ what: &'static str,
+ /// Its name.
+ name: String,
+ },
}
diff --git a/engine/crates/concat-media/src/ffi.rs b/engine/crates/concat-media/src/ffi.rs
index aa60cebd..a24f2184 100644
--- a/engine/crates/concat-media/src/ffi.rs
+++ b/engine/crates/concat-media/src/ffi.rs
@@ -1,34 +1,31 @@
// SPDX-License-Identifier: AGPL-3.0-or-later
// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
-//! Linked FFmpeg, as opposed to spawned FFmpeg.
-//!
-//! This backend exists for the two things a pipe fundamentally cannot do:
-//! report a frame's real presentation timestamp, and seek to an exact point.
-//! Both are needed for playback and scrubbing; neither matters for probing or
-//! export, which is why the subprocess backend is not going anywhere.
-//!
-//! Only compiled with the `ffi` feature. See
-//! `docs/decisions/0002-ffmpeg-over-a-pipe.md`.
-//!
-//! ## On the unsafe in here
-//!
-//! Every raw pointer this module owns is allocated in [`FfiDecoder::open`] and
-//! freed in exactly one place, [`FfiDecoder::drop`]. Nothing escapes: the only
-//! thing callers ever receive is an owned [`Frame`] of copied pixels. That
-//! containment is what keeps the `unsafe` auditable - it is a handful of
-//! functions, not a lifetime that leaks across the crate.
+//! The small things every linked-FFmpeg module needs: one-time
+//! initialisation, error conversion, timestamp arithmetic and the display
+//! rotation a stream carries.
-use std::ffi::{CStr, CString};
-use std::path::{Path, PathBuf};
-use std::ptr;
+use std::ffi::CStr;
+use std::path::Path;
+use std::sync::Once;
-use concat_core::frame::Frame;
use concat_core::time::Rational;
-use rusty_ffmpeg::ffi;
-
-use crate::decode::{FrameSource, SeekableSource};
-use crate::error::{Error, Result};
+use ffmpeg_the_third as ffmpeg;
+
+use crate::error::Error;
+
+/// Makes sure the libraries are initialised and quiet. Cheap after the
+/// first call; every entry point calls it.
+pub fn init() {
+ static ONCE: Once = Once::new();
+ ONCE.call_once(|| {
+ let _ = ffmpeg::init();
+ // Errors only: a GUI has no terminal, and FFmpeg's warnings are
+ // noise about files we are about to describe precisely in our own
+ // error messages.
+ ffmpeg::util::log::set_level(ffmpeg::util::log::Level::Error);
+ });
+}
/// The version of FFmpeg this binary is linked against.
///
@@ -38,451 +35,115 @@ use crate::error::{Error, Result};
pub fn linked_version() -> String {
// SAFETY: av_version_info returns a pointer to a static NUL-terminated
// string compiled into the library. Valid for the process lifetime.
- let raw = unsafe { ffi::av_version_info() };
+ let raw = unsafe { ffmpeg::sys::av_version_info() };
if raw.is_null() {
return "unknown".to_owned();
}
// SAFETY: checked non-null; contract is a NUL-terminated C string.
- unsafe { CStr::from_ptr(raw) }.to_string_lossy().into_owned()
-}
-
-/// Major version of the linked `libavcodec`.
-pub fn avcodec_major() -> u32 {
- // SAFETY: pure accessor returning a packed integer, no pointers involved.
- unsafe { ffi::avcodec_version() >> 16 }
-}
-
-/// Turns a libav error code into the message FFmpeg itself would print.
-fn describe(code: i32) -> String {
- let mut buffer = [0i8; ffi::AV_ERROR_MAX_STRING_SIZE as usize];
- // SAFETY: av_strerror writes at most `len` bytes into the buffer and
- // NUL-terminates. The buffer outlives the call.
- let written = unsafe { ffi::av_strerror(code, buffer.as_mut_ptr(), buffer.len()) };
- if written < 0 {
- return format!("error {code}");
- }
- // SAFETY: av_strerror NUL-terminates on success.
- unsafe { CStr::from_ptr(buffer.as_ptr()) }.to_string_lossy().into_owned()
-}
-
-/// `AVERROR(EAGAIN)`: the decoder wants more input before it can produce output.
-///
-/// Spelled out rather than taken from the bindings because `EAGAIN` comes from
-/// the C library's `errno.h`, which is not part of the FFmpeg headers and is
-/// therefore not always present in generated bindings. The value is *not* the
-/// same everywhere: 11 on Linux and on Windows FFmpeg builds, 35 on macOS,
-/// iOS and the BSDs - discovered the honest way, as "Resource temporarily
-/// unavailable" errors from a perfectly healthy decoder.
-#[cfg(any(
- target_os = "macos",
- target_os = "ios",
- target_os = "freebsd",
- target_os = "openbsd",
- target_os = "netbsd",
-))]
-const AVERROR_EAGAIN: i32 = -35;
-#[cfg(not(any(
- target_os = "macos",
- target_os = "ios",
- target_os = "freebsd",
- target_os = "openbsd",
- target_os = "netbsd",
-)))]
-const AVERROR_EAGAIN: i32 = -11;
-
-/// A decoder that links FFmpeg rather than spawning it.
-///
-/// Produces frames as RGBA8 scaled to a fixed size, alongside the real
-/// presentation timestamp of each.
-pub struct FfiDecoder {
- path: PathBuf,
- format: *mut ffi::AVFormatContext,
- codec: *mut ffi::AVCodecContext,
- scaler: *mut ffi::SwsContext,
- decoded: *mut ffi::AVFrame,
- packet: *mut ffi::AVPacket,
- stream: i32,
- /// Ticks per second for this stream's timestamps.
- time_base: ffi::AVRational,
- width: u32,
- height: u32,
- position: Option<Rational>,
- finished: bool,
+ unsafe { CStr::from_ptr(raw) }
+ .to_string_lossy()
+ .into_owned()
}
-// SAFETY: every pointer here is exclusively owned by this struct - none are
-// shared, and `&mut self` is required to touch any of them. FFmpeg contexts
-// may be used from any single thread, just not concurrently, which is exactly
-// what Rust's borrow rules already enforce for the owner.
-unsafe impl Send for FfiDecoder {}
-
-impl FfiDecoder {
- /// Opens `path`, producing frames scaled to `width` by `height`.
- pub fn open(path: impl AsRef<Path>, width: u32, height: u32) -> Result<Self> {
- let path = path.as_ref().to_path_buf();
- let fail = |operation: &'static str, code: i32| Error::Ffi {
- operation,
- path: path.clone(),
- detail: describe(code),
- };
-
- let c_path = CString::new(path.to_string_lossy().as_bytes()).map_err(|_| Error::Ffi {
- operation: "path conversion",
- path: path.clone(),
- detail: "path contains an interior NUL byte".to_owned(),
- })?;
-
- // Built up in pieces, and every early return from here on has to undo
- // what came before it - hence `Guard`, which owns the half-built state
- // until construction succeeds.
- let mut guard = Guard::default();
-
- unsafe {
- // SAFETY: `format` is null, so avformat_open_input allocates it and
- // writes the pointer back. On failure it leaves it null.
- let code = ffi::avformat_open_input(
- &mut guard.format,
- c_path.as_ptr(),
- ptr::null(),
- ptr::null_mut(),
- );
- if code < 0 {
- return Err(fail("avformat_open_input", code));
- }
-
- // SAFETY: format is open; this reads packets to fill in stream info.
- let code = ffi::avformat_find_stream_info(guard.format, ptr::null_mut());
- if code < 0 {
- return Err(fail("avformat_find_stream_info", code));
- }
-
- let mut decoder: *const ffi::AVCodec = ptr::null();
- // SAFETY: format is open; decoder is written on success.
- let stream = ffi::av_find_best_stream(
- guard.format,
- ffi::AVMEDIA_TYPE_VIDEO,
- -1,
- -1,
- &mut decoder,
- 0,
- );
- if stream < 0 {
- return Err(Error::NoVideoStream { path: path.clone() });
- }
-
- // SAFETY: av_find_best_stream returned a valid index into streams.
- let stream_ptr = *(*guard.format).streams.offset(stream as isize);
- let time_base = (*stream_ptr).time_base;
-
- // SAFETY: decoder was written by av_find_best_stream.
- guard.codec = ffi::avcodec_alloc_context3(decoder);
- if guard.codec.is_null() {
- return Err(fail("avcodec_alloc_context3", 0));
- }
-
- // SAFETY: both pointers are valid and the codec context is fresh.
- let code = ffi::avcodec_parameters_to_context(guard.codec, (*stream_ptr).codecpar);
- if code < 0 {
- return Err(fail("avcodec_parameters_to_context", code));
- }
-
- // SAFETY: context is populated from the stream's parameters.
- let code = ffi::avcodec_open2(guard.codec, decoder, ptr::null_mut());
- if code < 0 {
- return Err(fail("avcodec_open2", code));
- }
-
- let source_width = (*guard.codec).width;
- let source_height = (*guard.codec).height;
- let source_format = (*guard.codec).pix_fmt;
-
- // SAFETY: dimensions and format come from the opened codec.
- guard.scaler = ffi::sws_getContext(
- source_width,
- source_height,
- source_format,
- width as i32,
- height as i32,
- ffi::AV_PIX_FMT_RGBA,
- ffi::SWS_BILINEAR as i32,
- ptr::null_mut(),
- ptr::null_mut(),
- ptr::null(),
- );
- if guard.scaler.is_null() {
- return Err(fail("sws_getContext", 0));
- }
-
- // SAFETY: plain allocations, checked for null.
- guard.decoded = ffi::av_frame_alloc();
- guard.packet = ffi::av_packet_alloc();
- if guard.decoded.is_null() || guard.packet.is_null() {
- return Err(fail("av_frame_alloc", 0));
- }
-
- Ok(Self {
- path,
- format: guard.take_format(),
- codec: guard.take_codec(),
- scaler: guard.take_scaler(),
- decoded: guard.take_decoded(),
- packet: guard.take_packet(),
- stream,
- time_base,
- width,
- height,
- position: None,
- finished: false,
- })
- }
- }
-
- /// Converts a stream timestamp into seconds, exactly.
- ///
- /// This is the whole point of the linked backend: `time_base` is a
- /// rational, the timestamp is an integer count of those ticks, and
- /// `concat-core` speaks rationals - so the value survives with no rounding
- /// anywhere along the way.
- ///
- /// `None` when the container's numbers cannot form a representable value -
- /// a degenerate time base, or a product that overflows. Both come straight
- /// from an arbitrary file, so they must degrade to "position unknown"
- /// rather than panic the process.
- fn to_seconds(&self, ticks: i64) -> Option<Rational> {
- if self.time_base.den == 0 {
- return None;
- }
- Rational::checked_new(
- i128::from(ticks) * i128::from(self.time_base.num),
- i128::from(self.time_base.den),
- )
- }
-
- /// Scales the decoded frame into a fresh RGBA buffer.
- fn convert(&mut self) -> Frame {
- let stride = self.width as usize * 4;
- let mut pixels = vec![0u8; stride * self.height as usize];
-
- let destination: [*mut u8; 4] = [pixels.as_mut_ptr(), ptr::null_mut(), ptr::null_mut(), ptr::null_mut()];
- let strides: [i32; 4] = [stride as i32, 0, 0, 0];
-
- // SAFETY: `decoded` holds a complete frame from the codec, the scaler
- // was built for exactly that source format and this destination size,
- // and `pixels` is stride * height bytes - the size sws_scale writes.
- unsafe {
- ffi::sws_scale(
- self.scaler,
- (*self.decoded).data.as_ptr() as *const *const u8,
- (*self.decoded).linesize.as_ptr(),
- 0,
- (*self.decoded).height,
- destination.as_ptr(),
- strides.as_ptr(),
- );
- }
-
- Frame::from_rgba(self.width, self.height, pixels)
- .expect("buffer is sized from the same width and height")
+/// Wraps a libav error with the operation and the file it happened to.
+pub(crate) fn fail(operation: &'static str, path: &Path, error: ffmpeg::Error) -> Error {
+ Error::Ffi {
+ operation,
+ path: path.to_path_buf(),
+ detail: error.to_string(),
}
}
-impl FrameSource for FfiDecoder {
- fn width(&self) -> u32 {
- self.width
- }
-
- fn height(&self) -> u32 {
- self.height
- }
-
- fn position(&self) -> Option<Rational> {
- self.position
- }
-
- fn next_frame(&mut self) -> Result<Option<Frame>> {
- if self.finished {
- return Ok(None);
- }
-
- loop {
- // SAFETY: codec is open; `decoded` is a valid, reusable frame.
- let code = unsafe { ffi::avcodec_receive_frame(self.codec, self.decoded) };
-
- if code == 0 {
- // SAFETY: receive_frame succeeded, so the frame is populated.
- let ticks = unsafe { (*self.decoded).best_effort_timestamp };
- self.position =
- (ticks != ffi::AV_NOPTS_VALUE).then(|| self.to_seconds(ticks)).flatten();
- return Ok(Some(self.convert()));
- }
-
- if code == ffi::AVERROR_EOF {
- self.finished = true;
- return Ok(None);
- }
-
- if code != AVERROR_EAGAIN {
- return Err(Error::Ffi {
- operation: "avcodec_receive_frame",
- path: self.path.clone(),
- detail: describe(code),
- });
- }
-
- // EAGAIN: the decoder needs another packet before it can produce
- // anything. Packets for other streams are skipped, not decoded.
- loop {
- // SAFETY: format is open; av_read_frame fills the packet, and
- // we unref it below whatever happens.
- let code = unsafe { ffi::av_read_frame(self.format, self.packet) };
-
- if code == ffi::AVERROR_EOF {
- // Flush: a null packet tells the decoder to emit whatever
- // it is still holding, which is how the final few frames
- // of a file come out at all.
- // SAFETY: passing null is the documented flush signal.
- unsafe { ffi::avcodec_send_packet(self.codec, ptr::null()) };
- break;
- }
- if code < 0 {
- return Err(Error::Ffi {
- operation: "av_read_frame",
- path: self.path.clone(),
- detail: describe(code),
- });
- }
-
- // SAFETY: packet is populated by av_read_frame.
- let belongs = unsafe { (*self.packet).stream_index } == self.stream;
- if belongs {
- // SAFETY: sending a valid packet to an open decoder.
- let sent = unsafe { ffi::avcodec_send_packet(self.codec, self.packet) };
- // SAFETY: releases the packet's buffer for reuse.
- unsafe { ffi::av_packet_unref(self.packet) };
-
- if sent < 0 && sent != AVERROR_EAGAIN {
- return Err(Error::Ffi {
- operation: "avcodec_send_packet",
- path: self.path.clone(),
- detail: describe(sent),
- });
- }
- break;
- }
-
- // SAFETY: not our stream, so drop it and read the next.
- unsafe { ffi::av_packet_unref(self.packet) };
- }
- }
- }
+/// `AVERROR(EAGAIN)`: the codec or filter wants more input before it can
+/// produce output.
+pub(crate) fn is_again(error: &ffmpeg::Error) -> bool {
+ matches!(error, ffmpeg::Error::Other { errno } if *errno == libc::EAGAIN)
}
-impl SeekableSource for FfiDecoder {
- fn seek(&mut self, to: Rational) -> Result<()> {
- // Convert seconds back into this stream's tick count. The time base
- // comes from the file, so a degenerate one is an error, not a panic.
- if self.time_base.num == 0 || self.time_base.den == 0 {
- return Err(Error::Ffi {
- operation: "seek time base",
- path: self.path.clone(),
- detail: format!(
- "stream has a degenerate time base {}/{}",
- self.time_base.num, self.time_base.den
- ),
- });
- }
- let ticks_per_second = Rational::new(self.time_base.den as i64, self.time_base.num as i64);
- let target = (to * ticks_per_second).floor();
-
- // BACKWARD lands on the keyframe at or before the target. Decoding
- // forward from there is what makes the seek frame-accurate rather
- // than keyframe-accurate - which is precisely what the pipe cannot do.
- // SAFETY: format is open and the stream index came from it.
- let code = unsafe {
- ffi::av_seek_frame(self.format, self.stream, target, ffi::AVSEEK_FLAG_BACKWARD as i32)
- };
- if code < 0 {
- return Err(Error::Ffi {
- operation: "av_seek_frame",
- path: self.path.clone(),
- detail: describe(code),
- });
- }
-
- // The decoder is still holding frames from before the jump.
- // SAFETY: codec is open.
- unsafe { ffi::avcodec_flush_buffers(self.codec) };
-
- self.finished = false;
- self.position = None;
- Ok(())
+/// Converts a stream timestamp into seconds, exactly.
+///
+/// `time_base` is a rational, the timestamp is an integer count of those
+/// ticks, and `concat-core` speaks rationals - so the value survives with no
+/// rounding anywhere along the way. `None` when the container's numbers
+/// cannot form a representable value - a degenerate time base, or a product
+/// that overflows. Both come straight from an arbitrary file, so they must
+/// degrade to "position unknown" rather than panic the process.
+pub(crate) fn seconds(ticks: i64, time_base: ffmpeg::Rational) -> Option<Rational> {
+ if time_base.denominator() == 0 {
+ return None;
}
+ Rational::checked_new(
+ i128::from(ticks) * i128::from(time_base.numerator()),
+ i128::from(time_base.denominator()),
+ )
}
-impl Drop for FfiDecoder {
- fn drop(&mut self) {
- // SAFETY: each pointer was allocated in `open` and is freed exactly
- // once, here. The struct cannot be used after this.
- unsafe {
- ffi::av_packet_free(&mut self.packet);
- ffi::av_frame_free(&mut self.decoded);
- ffi::sws_freeContext(self.scaler);
- ffi::avcodec_free_context(&mut self.codec);
- ffi::avformat_close_input(&mut self.format);
- }
- }
+/// Seconds in `AV_TIME_BASE` units, which is what container-level seeks take.
+pub(crate) fn av_ticks(seconds: Rational) -> i64 {
+ let scaled = seconds * Rational::from_int(i64::from(ffmpeg::sys::AV_TIME_BASE));
+ scaled.floor()
}
-/// Owns half-built state so that an early return from `open` still frees it.
+/// The rotation a stream asks players to apply before showing its frames,
+/// in whole degrees clockwise, normalised to `0..360`.
///
-/// Without this, every `?` between the first allocation and the last would
-/// leak whatever had been built up to that point - and there are six of them.
-#[derive(Default)]
-struct Guard {
- format: *mut ffi::AVFormatContext,
- codec: *mut ffi::AVCodecContext,
- scaler: *mut ffi::SwsContext,
- decoded: *mut ffi::AVFrame,
- packet: *mut ffi::AVPacket,
-}
-
-macro_rules! take {
- ($name:ident, $field:ident, $ty:ty) => {
- fn $name(&mut self) -> $ty {
- std::mem::replace(&mut self.$field, ptr::null_mut())
+/// Phones record portrait video sideways and store a display matrix saying
+/// so; older files carry a `rotate` metadata tag instead. Both are read here
+/// so the decoder can turn the picture the way every player does, and the
+/// probe can report the dimensions as displayed.
+pub(crate) fn rotation(stream: &ffmpeg::format::stream::Stream<'_>) -> i64 {
+ // The display matrix moved from the stream to its codec parameters in
+ // FFmpeg 7.0, which is the oldest this crate builds against; the wrapper
+ // has no accessor for that field yet, so it is read directly.
+ // SAFETY: the stream pointer is valid for the stream's lifetime, and
+ // `coded_side_data` is an array of `nb_coded_side_data` entries owned by
+ // the codec parameters; a display matrix entry holds nine `i32`s.
+ let from_matrix = unsafe {
+ let parameters = (*stream.as_ptr()).codecpar;
+ let count = (*parameters).nb_coded_side_data;
+ let entries = (*parameters).coded_side_data;
+ let mut found = None;
+ for index in 0..count {
+ let entry = entries.offset(index as isize);
+ if (*entry).type_ == ffmpeg::sys::AVPacketSideDataType::DISPLAYMATRIX
+ && (*entry).size >= 9 * 4
+ {
+ // Players rotate by the negative of what the matrix encodes;
+ // `get_rotation` in ffmpeg's own tools does exactly this.
+ let angle = -ffmpeg::sys::av_display_rotation_get((*entry).data as *const i32);
+ found = Some(angle);
+ break;
+ }
}
+ found
};
+ let degrees = from_matrix.or_else(|| {
+ stream
+ .metadata()
+ .get("rotate")
+ .and_then(|value| value.parse::<f64>().ok())
+ });
+ degrees.map_or(0, |angle| (angle.round() as i64).rem_euclid(360))
}
-impl Guard {
- take!(take_format, format, *mut ffi::AVFormatContext);
- take!(take_codec, codec, *mut ffi::AVCodecContext);
- take!(take_scaler, scaler, *mut ffi::SwsContext);
- take!(take_decoded, decoded, *mut ffi::AVFrame);
- take!(take_packet, packet, *mut ffi::AVPacket);
+/// The filters that turn a decoded frame the way its rotation asks - the
+/// same three shapes ffmpeg's autorotate inserts - or nothing at all.
+pub(crate) fn rotation_filters(degrees: i64) -> Option<&'static str> {
+ match degrees {
+ 90 => Some("transpose=clock"),
+ 180 => Some("hflip,vflip"),
+ 270 => Some("transpose=cclock"),
+ _ => None,
+ }
}
-impl Drop for Guard {
- fn drop(&mut self) {
- // SAFETY: each free is a no-op on null, and a successful `open` has
- // already taken every pointer out, leaving all fields null.
- unsafe {
- if !self.packet.is_null() {
- ffi::av_packet_free(&mut self.packet);
- }
- if !self.decoded.is_null() {
- ffi::av_frame_free(&mut self.decoded);
- }
- if !self.scaler.is_null() {
- ffi::sws_freeContext(self.scaler);
- }
- if !self.codec.is_null() {
- ffi::avcodec_free_context(&mut self.codec);
- }
- if !self.format.is_null() {
- ffi::avformat_close_input(&mut self.format);
- }
- }
+/// The displayed size of a picture coded at `width` by `height` with the
+/// given rotation: a quarter turn swaps the two.
+pub(crate) fn displayed(width: u32, height: u32, degrees: i64) -> (u32, u32) {
+ if degrees.rem_euclid(180) == 90 {
+ (height, width)
+ } else {
+ (width, height)
}
}
@@ -491,24 +152,32 @@ mod tests {
use super::*;
#[test]
- fn links_against_the_expected_ffmpeg() {
- assert!(!linked_version().is_empty(), "av_version_info returned nothing");
+ fn links_against_something() {
+ init();
+ assert!(!linked_version().is_empty());
+ }
- // The bindings are generated for avcodec 62 (FFmpeg 8.x). A different
- // major version means the struct layouts we compiled against do not
- // match the library that loaded - which corrupts silently rather than
- // failing, so it is worth an explicit check.
- assert_eq!(avcodec_major(), 62, "linked libavcodec mismatch");
+ #[test]
+ fn a_quarter_turn_swaps_the_dimensions_and_a_half_turn_does_not() {
+ assert_eq!(displayed(1920, 1080, 90), (1080, 1920));
+ assert_eq!(displayed(1920, 1080, 270), (1080, 1920));
+ assert_eq!(displayed(1920, 1080, 180), (1920, 1080));
+ assert_eq!(displayed(1920, 1080, 0), (1920, 1080));
}
#[test]
- fn describes_error_codes() {
- // Whatever the wording, it should not be the numeric fallback.
- assert!(!describe(ffi::AVERROR_EOF).starts_with("error "));
+ fn rotation_filters_match_autorotate() {
+ assert_eq!(rotation_filters(0), None);
+ assert_eq!(rotation_filters(90), Some("transpose=clock"));
+ assert_eq!(rotation_filters(180), Some("hflip,vflip"));
+ assert_eq!(rotation_filters(270), Some("transpose=cclock"));
}
#[test]
- fn a_missing_file_is_an_error_not_a_panic() {
- assert!(FfiDecoder::open("does-not-exist.mp4", 64, 64).is_err());
+ fn timestamps_convert_exactly() {
+ let tb = ffmpeg::Rational::new(1, 30000);
+ assert_eq!(seconds(1001, tb), Some(Rational::new(1001, 30000)));
+ assert_eq!(seconds(5, ffmpeg::Rational::new(1, 0)), None);
+ assert_eq!(av_ticks(Rational::new(3, 2)), 1_500_000);
}
}
diff --git a/engine/crates/concat-media/src/lib.rs b/engine/crates/concat-media/src/lib.rs
index 0ac8e141..ec4ecbc1 100644
--- a/engine/crates/concat-media/src/lib.rs
+++ b/engine/crates/concat-media/src/lib.rs
@@ -1,35 +1,33 @@
// SPDX-License-Identifier: AGPL-3.0-or-later
// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
-//! Media IO: reading frames out of files and writing them back.
+//! Media IO: reading frames and samples out of files and writing them back.
//!
-//! This is the only crate that knows FFmpeg exists, and it talks to it as a
-//! child process rather than through FFI. See
-//! `docs/decisions/0002-ffmpeg-over-a-pipe.md` for the reasoning and the exit
-//! criteria.
+//! This is the only crate that knows FFmpeg exists, and it links it -
+//! libavformat, libavcodec, libavfilter, libswscale and libswresample -
+//! rather than spawning the `ffmpeg` binary. Nothing here needs a program on
+//! `PATH`; a build needs the FFmpeg development libraries, and the binary
+//! carries the rest.
//!
-//! Everything process-shaped hides behind two traits, [`FrameSource`] and
-//! [`FrameSink`]. A future GPU-decode or FFI backend implements those two and
-//! nothing else in the workspace changes.
+//! Everything codec-shaped hides behind two traits, [`FrameSource`] and
+//! [`FrameSink`]. A GPU-decode backend would implement those two and nothing
+//! else in the workspace changes.
pub mod audio;
-pub mod binaries;
pub mod decode;
pub mod encode;
-#[cfg(feature = "ffi")]
-pub mod ffi;
pub mod error;
+mod ffi;
pub mod peaks;
pub mod pool;
pub mod probe;
+pub mod samples;
-mod process;
-
-pub use binaries::{ffmpeg, ffprobe, set_binaries};
-pub use decode::{DecodeOptions, FfmpegDecoder, FrameSource, SeekableSource};
-pub use encode::{EncodeOptions, FfmpegEncoder, FrameSink};
+pub use decode::{DecodeOptions, Decoder, FrameSource, SeekableSource};
+pub use encode::{EncodeOptions, Encoder, FrameSink, jpeg};
pub use error::{Error, Result};
+pub use ffi::{init, linked_version};
pub use peaks::Peaks;
pub use pool::{FrameCache, ReaderPool};
pub use probe::{AudioStream, MediaInfo, VideoStream, probe};
-pub use process::command;
+pub use samples::{AudioDecoder, AudioOptions, SampleFormat};
diff --git a/engine/crates/concat-media/src/peaks.rs b/engine/crates/concat-media/src/peaks.rs
index 26861a3f..b43e833c 100644
--- a/engine/crates/concat-media/src/peaks.rs
+++ b/engine/crates/concat-media/src/peaks.rs
@@ -5,21 +5,14 @@
//!
//! The timeline draws a clip's waveform from a few hundred buckets per
//! second, not from the samples themselves. This module produces those
-//! buckets by streaming FFmpeg's decode of the file's audio - the samples
-//! are folded into buckets as they arrive and never accumulate, so an
-//! hour-long recording costs the same memory as a jingle.
-//!
-//! This used to live in the UI, which read the entire file over IPC and
-//! decoded it with WebAudio on the main thread. Moving it here removed the
-//! app's last whole-file read across the IPC boundary.
+//! buckets by streaming the file's decoded audio - the samples are folded
+//! into buckets as they arrive and never accumulate, so an hour-long
+//! recording costs the same memory as a jingle.
-use std::io::Read;
use std::path::Path;
-use std::process::Stdio;
-use crate::binaries::ffmpeg;
-use crate::error::{Error, Result};
-use crate::process::{StderrTail, base_command};
+use crate::error::Result;
+use crate::samples::{AudioDecoder, AudioOptions, SampleFormat};
/// The rate audio is resampled to before bucketing.
///
@@ -62,174 +55,107 @@ impl Peaks {
}
bytes
}
+
+ /// The inverse of [`Peaks::encode`]: `None` for bytes that do not have
+ /// that shape, so a corrupt cache entry regenerates instead of drawing.
+ pub fn decode(bytes: &[u8]) -> Option<Peaks> {
+ if bytes.len() < 8 {
+ return None;
+ }
+ let buckets_per_second = f32::from_le_bytes([bytes[0], bytes[1], bytes[2], bytes[3]]);
+ let count = u32::from_le_bytes([bytes[4], bytes[5], bytes[6], bytes[7]]) as usize;
+ if bytes.len() != 8 + count * 8 {
+ return None;
+ }
+ let floats = |offset: usize| -> Vec<f32> {
+ bytes[offset..offset + count * 4]
+ .chunks_exact(4)
+ .map(|chunk| f32::from_le_bytes([chunk[0], chunk[1], chunk[2], chunk[3]]))
+ .collect()
+ };
+ Some(Peaks {
+ min: floats(8),
+ max: floats(8 + count * 4),
+ buckets_per_second,
+ })
+ }
}
/// Decodes one file's audio and reduces it to peaks.
///
-/// The decode is mono 16-bit at [`PEAK_RATE`], piped back as WAV - `wav`
-/// rather than raw `s16le` because the bundled FFmpeg is a trimmed build
-/// and the raw muxer is not in it. A file with no audio stream fails the
-/// FFmpeg run, and the error carries what FFmpeg said.
+/// The decode is mono 16-bit at [`PEAK_RATE`]. A file with no audio stream
+/// is an error that says so.
pub fn extract(path: &Path, buckets_per_second: u32) -> Result<Peaks> {
- let mut child = base_command(ffmpeg())
- .arg("-i")
- .arg(path)
- .args(["-vn", "-ac", "1"])
- .args(["-ar", &PEAK_RATE.to_string()])
- .args(["-c:a", "pcm_s16le", "-f", "wav", "pipe:1"])
- .stdin(Stdio::null())
- .stdout(Stdio::piped())
- .stderr(Stdio::piped())
- .spawn()
- .map_err(|source| Error::Spawn { program: "ffmpeg", source })?;
-
- let mut tail = StderrTail::drain(&mut child);
- let mut stdout = child.stdout.take().expect("stdout was piped");
- let parsed = peaks_from_wav(&mut stdout, path, buckets_per_second);
-
- if parsed.is_err() {
- // A parse failure leaves FFmpeg still writing into a pipe nobody
- // reads; waiting on it now would deadlock on the full pipe.
- let _ = child.kill();
+ let mut decoder = AudioDecoder::open(
+ path,
+ &AudioOptions {
+ rate: PEAK_RATE,
+ channels: 1,
+ format: SampleFormat::I16,
+ ..AudioOptions::default()
+ },
+ )?;
+ let mut folder = Folder::new(buckets_per_second);
+ while let Some(chunk) = decoder.next_i16()? {
+ folder.fold_all(chunk.iter().map(|sample| f32::from(*sample) / 32768.0));
}
- let status = child.wait().map_err(|source| Error::Io { program: "ffmpeg", source })?;
-
- // FFmpeg's own account of the failure - "no audio stream", "no such
- // file" - beats whatever the parser tripped over downstream of it.
- if !status.success() {
- return Err(Error::Exited {
- program: "ffmpeg",
- path: path.to_path_buf(),
- status,
- stderr: tail.summary(),
- });
- }
- parsed
+ Ok(folder.finish())
}
-/// Folds a WAV stream into peaks without holding the samples.
-///
-/// A plain chunk walk, not a format library: the stream comes from our own
-/// FFmpeg invocation above, so the only variability is which bookkeeping
-/// chunks precede `data`. Sizes written to a pipe are `u32::MAX`
-/// placeholders - the muxer could not seek back to patch them - in which
-/// case the payload is simply the rest of the stream.
-fn peaks_from_wav(
- reader: &mut impl Read,
- path: &Path,
- buckets_per_second: u32,
-) -> Result<Peaks> {
- let malformed = |detail: &str| Error::Probe {
- path: path.to_path_buf(),
- detail: format!("ffmpeg piped back something other than the wav asked for: {detail}"),
- };
- let io = |source| Error::Io { program: "ffmpeg", source };
-
- let buckets_per_second = buckets_per_second.clamp(1, PEAK_RATE);
- let bucket_size = (PEAK_RATE / buckets_per_second) as usize;
-
- let mut riff = [0u8; 12];
- reader.read_exact(&mut riff).map_err(io)?;
- if &riff[0..4] != b"RIFF" || &riff[8..12] != b"WAVE" {
- return Err(malformed("missing RIFF/WAVE header"));
- }
+/// Folds a sample stream into peaks without holding the samples.
+struct Folder {
+ bucket_size: usize,
+ min: Vec<f32>,
+ max: Vec<f32>,
+ low: f32,
+ high: f32,
+ filled: usize,
+}
- // Walk the bookkeeping chunks until `data`. Its payload runs to EOF
- // when the size is a placeholder.
- let mut remaining: Option<usize> = loop {
- let mut header = [0u8; 8];
- reader.read_exact(&mut header).map_err(io)?;
- let size = u32::from_le_bytes([header[4], header[5], header[6], header[7]]);
- if &header[0..4] == b"data" {
- break if size == u32::MAX { None } else { Some(size as usize) };
+impl Folder {
+ fn new(buckets_per_second: u32) -> Self {
+ let buckets_per_second = buckets_per_second.clamp(1, PEAK_RATE);
+ Self {
+ bucket_size: (PEAK_RATE / buckets_per_second) as usize,
+ min: Vec::new(),
+ max: Vec::new(),
+ low: 0.0,
+ high: 0.0,
+ filled: 0,
}
- // A placeholder size on any other chunk means the walk cannot
- // continue - refuse rather than misread samples as headers.
- if size == u32::MAX {
- return Err(malformed("placeholder size before the data chunk"));
- }
- // Chunks are word-aligned; an odd size is followed by a pad byte.
- let skip = size as usize + (size as usize & 1);
- std::io::copy(&mut reader.take(skip as u64), &mut std::io::sink()).map_err(io)?;
- };
-
- let mut min = Vec::new();
- let mut max = Vec::new();
- let mut low = 0.0f32;
- let mut high = 0.0f32;
- let mut filled = 0usize;
+ }
- let mut chunk = [0u8; 8192];
- // A sample split across two reads: its first byte waits here.
- let mut half: Option<u8> = None;
- loop {
- let want = remaining.map_or(chunk.len(), |left| left.min(chunk.len()));
- if want == 0 {
- break;
- }
- let read = reader.read(&mut chunk[..want]).map_err(io)?;
- if read == 0 {
- break;
- }
- if let Some(left) = remaining.as_mut() {
- *left -= read;
+ fn fold_all(&mut self, samples: impl Iterator<Item = f32>) {
+ for sample in samples {
+ if sample < self.low {
+ self.low = sample;
+ }
+ if sample > self.high {
+ self.high = sample;
+ }
+ self.filled += 1;
+ if self.filled == self.bucket_size {
+ self.min.push(self.low);
+ self.max.push(self.high);
+ self.low = 0.0;
+ self.high = 0.0;
+ self.filled = 0;
+ }
}
+ }
- let mut bytes = &chunk[..read];
- if let Some(first) = half.take() {
- let sample = f32::from(i16::from_le_bytes([first, bytes[0]])) / 32768.0;
- fold(sample, &mut low, &mut high);
- bump(&mut filled, bucket_size, &mut min, &mut max, &mut low, &mut high);
- bytes = &bytes[1..];
+ fn finish(mut self) -> Peaks {
+ // The trailing partial bucket still counts - dropping it would shave
+ // the last fraction of a second off every waveform.
+ if self.filled > 0 {
+ self.min.push(self.low);
+ self.max.push(self.high);
}
- for pair in bytes.chunks_exact(2) {
- let sample = f32::from(i16::from_le_bytes([pair[0], pair[1]])) / 32768.0;
- fold(sample, &mut low, &mut high);
- bump(&mut filled, bucket_size, &mut min, &mut max, &mut low, &mut high);
+ Peaks {
+ min: self.min,
+ max: self.max,
+ buckets_per_second: PEAK_RATE as f32 / self.bucket_size as f32,
}
- if bytes.len() & 1 == 1 {
- half = Some(bytes[bytes.len() - 1]);
- }
- }
-
- // The trailing partial bucket still counts - dropping it would shave
- // the last fraction of a second off every waveform.
- if filled > 0 {
- min.push(low);
- max.push(high);
- }
-
- Ok(Peaks {
- min,
- max,
- buckets_per_second: PEAK_RATE as f32 / bucket_size as f32,
- })
-}
-
-fn fold(sample: f32, low: &mut f32, high: &mut f32) {
- if sample < *low {
- *low = sample;
- }
- if sample > *high {
- *high = sample;
- }
-}
-
-fn bump(
- filled: &mut usize,
- bucket_size: usize,
- min: &mut Vec<f32>,
- max: &mut Vec<f32>,
- low: &mut f32,
- high: &mut f32,
-) {
- *filled += 1;
- if *filled == bucket_size {
- min.push(*low);
- max.push(*high);
- *low = 0.0;
- *high = 0.0;
- *filled = 0;
}
}
@@ -237,22 +163,10 @@ fn bump(
mod tests {
use super::*;
- /// A WAV as FFmpeg writes one to a pipe: placeholder sizes, an `fmt `
- /// chunk, then `data` running to EOF.
- fn piped_wav(samples: &[i16]) -> Vec<u8> {
- let mut bytes = Vec::new();
- bytes.extend_from_slice(b"RIFF");
- bytes.extend_from_slice(&u32::MAX.to_le_bytes());
- bytes.extend_from_slice(b"WAVE");
- bytes.extend_from_slice(b"fmt ");
- bytes.extend_from_slice(&16u32.to_le_bytes());
- bytes.extend_from_slice(&[0u8; 16]);
- bytes.extend_from_slice(b"data");
- bytes.extend_from_slice(&u32::MAX.to_le_bytes());
- for sample in samples {
- bytes.extend_from_slice(&sample.to_le_bytes());
- }
- bytes
+ fn fold(samples: &[i16], buckets_per_second: u32) -> Peaks {
+ let mut folder = Folder::new(buckets_per_second);
+ folder.fold_all(samples.iter().map(|sample| f32::from(*sample) / 32768.0));
+ folder.finish()
}
#[test]
@@ -264,9 +178,7 @@ mod tests {
samples[100] = 16384;
samples.extend(std::iter::repeat_n(-8192i16, 10));
- let wav = piped_wav(&samples);
- let peaks = peaks_from_wav(&mut wav.as_slice(), Path::new("test.wav"), 200).unwrap();
-
+ let peaks = fold(&samples, 200);
assert_eq!(peaks.buckets_per_second, 200.0);
assert_eq!(peaks.min.len(), 2);
assert_eq!(peaks.min[0], -1.0);
@@ -278,27 +190,32 @@ mod tests {
#[test]
fn silence_is_flat_zeroes() {
- let wav = piped_wav(&[0i16; 480]);
- let peaks = peaks_from_wav(&mut wav.as_slice(), Path::new("test.wav"), 200).unwrap();
+ let peaks = fold(&[0i16; 480], 200);
assert_eq!(peaks.min, vec![0.0, 0.0]);
assert_eq!(peaks.max, vec![0.0, 0.0]);
}
#[test]
- fn a_stream_that_is_not_wav_is_refused() {
- let junk = b"MPEG something else entirely, long enough to read".to_vec();
- let error = peaks_from_wav(&mut junk.as_slice(), Path::new("test.mp3"), 200);
- assert!(error.is_err());
+ fn a_file_with_no_audio_is_an_error() {
+ assert!(extract(Path::new("does-not-exist.mp3"), 200).is_err());
}
#[test]
- fn encode_lays_out_rate_count_min_max() {
- let peaks = Peaks { min: vec![-0.5, 0.0], max: vec![0.25, 0.0], buckets_per_second: 200.0 };
+ fn encode_lays_out_rate_count_min_max_and_decode_reads_it_back() {
+ let peaks = Peaks {
+ min: vec![-0.5, 0.0],
+ max: vec![0.25, 0.0],
+ buckets_per_second: 200.0,
+ };
let bytes = peaks.encode();
assert_eq!(bytes.len(), 8 + 2 * 8);
assert_eq!(f32::from_le_bytes(bytes[0..4].try_into().unwrap()), 200.0);
assert_eq!(u32::from_le_bytes(bytes[4..8].try_into().unwrap()), 2);
assert_eq!(f32::from_le_bytes(bytes[8..12].try_into().unwrap()), -0.5);
assert_eq!(f32::from_le_bytes(bytes[16..20].try_into().unwrap()), 0.25);
+ let back = Peaks::decode(&bytes).expect("decodes");
+ assert_eq!(back.min, peaks.min);
+ assert_eq!(back.max, peaks.max);
+ assert!(Peaks::decode(&bytes[..10]).is_none());
}
}
diff --git a/engine/crates/concat-media/src/pool.rs b/engine/crates/concat-media/src/pool.rs
index 56e1b530..05f6451c 100644
--- a/engine/crates/concat-media/src/pool.rs
+++ b/engine/crates/concat-media/src/pool.rs
@@ -10,11 +10,10 @@
//! module is the piece the decision log has called "the next real piece of
//! work" since the first CLI commit:
//!
-//! - **One reader per (file, size)**, kept warm between requests. A request
-//! near the reader's current position rolls forward (cheap, exact); a
-//! request elsewhere seeks - frame-accurately through the FFI decoder when
-//! the `ffi` feature is on, by respawning the subprocess decoder at the
-//! target otherwise.
+//! - **One reader per (file, size, chain)**, kept warm between requests. A
+//! request near the reader's current position rolls forward (cheap,
+//! exact); a request elsewhere seeks - frame-accurately, guided by the
+//! real timestamps the linked decoder reports.
//! - **A byte-budgeted LRU frame cache** in front of the readers. Scrubbing
//! back over ground just covered, or dwelling on one frame, costs a hash
//! lookup. Frames roll into the cache as decoding passes them, so rolling
@@ -22,7 +21,7 @@
//!
//! The pool is deliberately not used by export: export decodes every frame
//! exactly once in order, and a cache in that path is pure overhead. This is
-//! playback infrastructure - see `docs/decisions/0002`.
+//! playback infrastructure.
use std::collections::HashMap;
use std::path::{Path, PathBuf};
@@ -31,7 +30,7 @@ use std::sync::Arc;
use concat_core::frame::Frame;
use concat_core::time::{FrameRate, Rational};
-use crate::decode::{DecodeOptions, FfmpegDecoder, FrameSource};
+use crate::decode::{DecodeOptions, Decoder, FrameSource, SeekableSource};
use crate::error::Result;
use crate::probe;
@@ -68,7 +67,12 @@ pub struct FrameCache {
impl FrameCache {
/// An empty cache holding at most `budget` bytes of frames.
pub fn new(budget: usize) -> Self {
- Self { budget, held: 0, tick: 0, frames: HashMap::new() }
+ Self {
+ budget,
+ held: 0,
+ tick: 0,
+ frames: HashMap::new(),
+ }
}
fn get(&mut self, key: &FrameKey) -> Option<Arc<Frame>> {
@@ -147,21 +151,12 @@ fn plan_access(current_next: Option<i64>, target: i64) -> Access {
/// One warm reader: a decoder plus where its *next* frame will land.
struct Reader {
- backend: Backend,
- /// The frame index `next_frame` will produce, in the media's own rate.
+ decoder: Decoder,
+ /// The frame index `next_frame` will produce, in the media's own rate,
+ /// or `i64::MIN` right after a seek, when only the timestamps know.
next_index: i64,
}
-enum Backend {
- /// Frame-accurate seeks, real timestamps. Only with the `ffi` feature.
- #[cfg(feature = "ffi")]
- Linked(crate::ffi::FfiDecoder),
- /// The subprocess pipe: no seeking, so "seek" means respawn at the
- /// target's keyframe-fast `-ss` and count ordinally from there. Exact for
- /// constant-frame-rate material, the pipe's known limit otherwise.
- Pipe { decoder: FfmpegDecoder, path: PathBuf, width: u32, height: u32, chain: Option<String> },
-}
-
impl Reader {
fn open(
path: &Path,
@@ -171,90 +166,41 @@ impl Reader {
rate: FrameRate,
index: i64,
) -> Result<Self> {
- #[cfg(feature = "ffi")]
- // The linked decoder produces raw frames; a clip with an effect chain
- // needs FFmpeg's filters, so it stays on the pipe.
- if chain.is_none() {
- if let Ok(mut linked) = crate::ffi::FfiDecoder::open(path, width, height) {
- use crate::decode::SeekableSource;
- linked.seek(rate.time_of_frame(index))?;
- // The linked decoder lands on the keyframe at or before the
- // target; rolling forward from there is what `frame_at` does,
- // guided by real timestamps. Report where decoding resumes.
- return Ok(Self { backend: Backend::Linked(linked), next_index: i64::MIN });
- }
- // A file the linked build cannot open falls through to the pipe.
- }
-
+ // Unpaced, so every source frame comes out with its own timestamp
+ // and the index is read from that; a seek lands on the keyframe at
+ // or before the target and `frame_at` rolls forward from there.
let mut options = DecodeOptions::default()
.starting_at(rate.time_of_frame(index))
- .scaled_to(width, height)
- .at_rate(rate);
+ .scaled_to(width, height);
if let Some(chain) = chain {
options = options.filtered(chain);
}
- let decoder = FfmpegDecoder::open(path, &options)?;
+ let decoder = Decoder::open(path, &options)?;
Ok(Self {
- backend: Backend::Pipe {
- decoder,
- path: path.to_path_buf(),
- width,
- height,
- chain: chain.map(str::to_owned),
- },
- next_index: index,
+ decoder,
+ next_index: i64::MIN,
})
}
- #[cfg_attr(not(feature = "ffi"), allow(unused_variables))]
fn next_frame(&mut self, rate: FrameRate) -> Result<Option<(i64, Frame)>> {
- match &mut self.backend {
- #[cfg(feature = "ffi")]
- Backend::Linked(decoder) => {
- let Some(frame) = decoder.next_frame()? else { return Ok(None) };
- // The linked decoder tells us where the frame really sits; a
- // half-frame nudge keeps exact boundary timestamps from
- // rounding down a frame.
- let index = decoder
- .position()
- .map(|position| rate.frame_at(position + rate.frame_duration() / Rational::from_int(2)))
- .unwrap_or(self.next_index.max(0));
- self.next_index = index + 1;
- Ok(Some((index, frame)))
- }
- Backend::Pipe { decoder, .. } => {
- let Some(frame) = decoder.next_frame()? else { return Ok(None) };
- let index = self.next_index;
- self.next_index = index + 1;
- Ok(Some((index, frame)))
- }
- }
+ let Some(frame) = self.decoder.next_frame()? else {
+ return Ok(None);
+ };
+ // The decoder says where the frame really sits; a half-frame nudge
+ // keeps exact boundary timestamps from rounding down a frame.
+ let index = self
+ .decoder
+ .position()
+ .map(|position| rate.frame_at(position + rate.frame_duration() / Rational::from_int(2)))
+ .unwrap_or(self.next_index.max(0));
+ self.next_index = index + 1;
+ Ok(Some((index, frame)))
}
fn seek(&mut self, rate: FrameRate, index: i64) -> Result<()> {
- match &mut self.backend {
- #[cfg(feature = "ffi")]
- Backend::Linked(decoder) => {
- use crate::decode::SeekableSource;
- decoder.seek(rate.time_of_frame(index))?;
- self.next_index = i64::MIN;
- Ok(())
- }
- Backend::Pipe { decoder, path, width, height, chain } => {
- // The pipe cannot seek; a new process at the target is the
- // seek. This is the cost the FFI backend exists to remove.
- let mut options = DecodeOptions::default()
- .starting_at(rate.time_of_frame(index))
- .scaled_to(*width, *height)
- .at_rate(rate);
- if let Some(chain) = chain {
- options = options.filtered(chain.clone());
- }
- *decoder = FfmpegDecoder::open(path.as_path(), &options)?;
- self.next_index = index;
- Ok(())
- }
- }
+ self.decoder.seek(rate.time_of_frame(index))?;
+ self.next_index = i64::MIN;
+ Ok(())
}
}
@@ -318,7 +264,11 @@ impl ReaderPool {
) -> Result<Arc<Frame>> {
let facts = self.facts_for(path, still)?;
let rate = facts.rate;
- let mut target = if facts.still { 0 } else { rate.frame_at(time).max(0) };
+ let mut target = if facts.still {
+ 0
+ } else {
+ rate.frame_at(time).max(0)
+ };
// A clip can outlive its media - an end-trim past the file's length -
// and the honest picture for any time past the end is the last frame,
// exactly what the roll-forward case already serves. Clamp before
@@ -406,17 +356,19 @@ impl ReaderPool {
}
}
- latest.ok_or_else(|| crate::error::Error::PartialFrame {
+ latest.ok_or_else(|| crate::error::Error::NoFrame {
path: path.to_path_buf(),
- got: 0,
- want: Frame::byte_len(width, height),
})
}
fn facts_for(&mut self, path: &Path, still: bool) -> Result<&MediaFacts> {
if !self.facts.contains_key(path) {
let facts = if still {
- MediaFacts { rate: FrameRate::THIRTY, still: true, frames: None }
+ MediaFacts {
+ rate: FrameRate::THIRTY,
+ still: true,
+ frames: None,
+ }
} else {
let info = probe::probe(path)?;
let rate = info
@@ -430,7 +382,11 @@ impl ReaderPool {
.duration
.map(|duration| (duration * rate.fps()).ceil())
.filter(|count| *count > 0);
- MediaFacts { rate, still: false, frames }
+ MediaFacts {
+ rate,
+ still: false,
+ frames,
+ }
};
self.facts.insert(path.to_path_buf(), facts);
}
@@ -488,19 +444,31 @@ impl ReaderPool {
#[cfg(test)]
mod tests {
use super::*;
- use crate::encode::{EncodeOptions, FfmpegEncoder, FrameSink};
+ use crate::encode::{EncodeOptions, Encoder, FrameSink};
#[test]
fn the_access_policy_rolls_forward_and_seeks_backward() {
- assert_eq!(plan_access(Some(10), 10), Access::Roll(0), "the very next frame");
- assert_eq!(plan_access(Some(10), 30), Access::Roll(20), "a short hop ahead");
+ assert_eq!(
+ plan_access(Some(10), 10),
+ Access::Roll(0),
+ "the very next frame"
+ );
+ assert_eq!(
+ plan_access(Some(10), 30),
+ Access::Roll(20),
+ "a short hop ahead"
+ );
assert_eq!(plan_access(Some(10), 9), Access::Seek, "behind means seek");
assert_eq!(
plan_access(Some(10), 10 + ROLL_FORWARD_FRAMES + 1),
Access::Seek,
"too far ahead means seek"
);
- assert_eq!(plan_access(None, 5), Access::Seek, "unknown position means seek");
+ assert_eq!(
+ plan_access(None, 5),
+ Access::Seek,
+ "unknown position means seek"
+ );
}
#[test]
@@ -533,23 +501,27 @@ mod tests {
fn an_oversized_frame_is_served_but_never_cached() {
let mut cache = FrameCache::new(8);
cache.insert(
- FrameKey { path: PathBuf::from("a"), width: 2, height: 2, chain: None, index: 0 },
+ FrameKey {
+ path: PathBuf::from("a"),
+ width: 2,
+ height: 2,
+ chain: None,
+ index: 0,
+ },
Arc::new(Frame::black(2, 2)),
);
assert!(cache.is_empty(), "16 bytes cannot fit an 8 byte budget");
}
- /// End to end against a real FFmpeg: encode a tiny video whose frames are
- /// identifiable by colour, then read arbitrary frames back out of order.
- /// Skips silently on machines without FFmpeg, like the encoder's own test.
+ /// End to end against the linked FFmpeg: encode a tiny video whose frames
+ /// are identifiable by colour, then read arbitrary frames back out of
+ /// order.
#[test]
fn random_access_returns_the_right_frames() {
let path = std::env::temp_dir().join("concat-pool-test.mp4");
- let Ok(mut encoder) =
- FfmpegEncoder::create(&path, 64, 64, FrameRate::THIRTY, &EncodeOptions::default())
- else {
- return; // no ffmpeg here
- };
+ let mut encoder =
+ Encoder::create(&path, 64, 64, FrameRate::THIRTY, &EncodeOptions::default())
+ .expect("the linked FFmpeg encodes h264");
// 90 frames; the red channel encodes the frame index (x2 to survive
// compression rounding).
for index in 0..90u32 {
@@ -577,14 +549,23 @@ mod tests {
let back_at_20 = red_at(&mut pool, 20);
assert!(near(back_at_20, 20), "backward to 20 read {back_at_20}");
let again_at_50 = red_at(&mut pool, 50);
- assert_eq!(again_at_50, at_50, "revisit must come from cache, identical");
- assert!(!pool.cache.is_empty(), "rolling forward populated the cache");
+ assert_eq!(
+ again_at_50, at_50,
+ "revisit must come from cache, identical"
+ );
+ assert!(
+ !pool.cache.is_empty(),
+ "rolling forward populated the cache"
+ );
// Far past the end - a clip that outlives its media. The last real
// frame is the answer, not an error: a seek there decodes nothing,
// which used to surface as PartialFrame(0 bytes) and a black monitor.
let past_end = red_at(&mut pool, 300);
- assert!(near(past_end, 89), "past the end read {past_end}, wanted the last frame");
+ assert!(
+ near(past_end, 89),
+ "past the end read {past_end}, wanted the last frame"
+ );
let _ = std::fs::remove_file(&path);
}
diff --git a/engine/crates/concat-media/src/probe.rs b/engine/crates/concat-media/src/probe.rs
index 3f7935ed..f2b7f1d2 100644
--- a/engine/crates/concat-media/src/probe.rs
+++ b/engine/crates/concat-media/src/probe.rs
@@ -1,17 +1,15 @@
// SPDX-License-Identifier: AGPL-3.0-or-later
// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
-//! Asking `ffprobe` what is inside a file.
+//! Asking libavformat what is inside a file.
use std::path::{Path, PathBuf};
use concat_core::time::{FrameRate, Rational};
-use serde_json::Value;
+use ffmpeg_the_third as ffmpeg;
use crate::error::{Error, Result};
-
-/// Name used in error messages; see [`crate::binaries::ffprobe`].
-const FFPROBE: &str = "ffprobe";
+use crate::ffi;
/// What a video stream looks like.
#[derive(Clone, Debug)]
@@ -62,238 +60,111 @@ pub struct MediaInfo {
impl MediaInfo {
/// The video stream, or [`Error::NoVideoStream`] if the file has none.
pub fn require_video(&self) -> Result<&VideoStream> {
- self.video.as_ref().ok_or_else(|| Error::NoVideoStream { path: self.path.clone() })
+ self.video.as_ref().ok_or_else(|| Error::NoVideoStream {
+ path: self.path.clone(),
+ })
}
}
-/// Runs `ffprobe` and summarises what it found.
+/// Opens the file and summarises what it found.
pub fn probe(path: impl AsRef<Path>) -> Result<MediaInfo> {
+ ffi::init();
let path = path.as_ref();
- let output = crate::process::command(crate::binaries::ffprobe())
- .args(["-v", "error", "-show_streams", "-show_format", "-of", "json"])
- .arg(path)
- .output()
- .map_err(|source| Error::Spawn { program: FFPROBE, source })?;
+ let input = ffmpeg::format::input(path).map_err(|error| ffi::fail("open", path, error))?;
- if !output.status.success() {
- return Err(Error::Exited {
- program: FFPROBE,
- path: path.to_path_buf(),
- status: output.status,
- stderr: crate::process::summarize(&output.stderr),
- });
- }
-
- let root: Value = serde_json::from_slice(&output.stdout).map_err(|error| Error::Probe {
- path: path.to_path_buf(),
- detail: format!("output was not valid json: {error}"),
- })?;
+ let duration = input.duration();
+ let duration =
+ (duration > 0).then(|| Rational::new(duration, i64::from(ffmpeg::sys::AV_TIME_BASE)));
- let streams = root.get("streams").and_then(Value::as_array).ok_or_else(|| Error::Probe {
- path: path.to_path_buf(),
- detail: "no `streams` array".to_owned(),
- })?;
+ let video = match input.streams().best(ffmpeg::media::Type::Video) {
+ Some(stream) => Some(video_stream(&stream, path)?),
+ None => None,
+ };
+ let audio = input
+ .streams()
+ .best(ffmpeg::media::Type::Audio)
+ .map(|stream| {
+ let parameters = stream.parameters();
+ AudioStream {
+ index: stream.index() as u32,
+ codec: parameters.id().name().to_owned(),
+ sample_rate: parameters.sample_rate(),
+ channels: parameters.ch_layout().channels(),
+ }
+ });
Ok(MediaInfo {
path: path.to_path_buf(),
- duration: root
- .get("format")
- .and_then(|format| format.get("duration"))
- .and_then(Value::as_str)
- .and_then(Rational::parse),
- video: streams.iter().find_map(|stream| parse_video(stream, path)).transpose()?,
- audio: streams.iter().find_map(parse_audio),
+ duration,
+ video,
+ audio,
})
}
-/// Returns `None` for non-video streams, `Some(Err(..))` for a video stream we
-/// cannot make sense of. Skipping a malformed video stream silently would show
-/// up much later as a blank preview.
-fn parse_video(stream: &Value, path: &Path) -> Option<Result<VideoStream>> {
- if stream.get("codec_type").and_then(Value::as_str) != Some("video") {
- return None;
+fn video_stream(stream: &ffmpeg::format::stream::Stream<'_>, path: &Path) -> Result<VideoStream> {
+ let parameters = stream.parameters();
+ let (width, height) = (parameters.width(), parameters.height());
+ if width == 0 || height == 0 {
+ return Err(Error::Probe {
+ path: path.to_path_buf(),
+ detail: "video stream has no usable size".to_owned(),
+ });
}
+ let (width, height) = ffi::displayed(width, height, ffi::rotation(stream));
- let missing = |field: &str| Error::Probe {
- path: path.to_path_buf(),
- detail: format!("video stream has no usable `{field}`"),
- };
-
- Some((|| {
- let width = u32_field(stream, "width").ok_or_else(|| missing("width"))?;
- let height = u32_field(stream, "height").ok_or_else(|| missing("height"))?;
-
- // Portrait phone footage is commonly coded sideways, with a display
- // rotation every player applies before showing the frame - as does
- // our own decoder, which leaves ffmpeg's autorotate on. What the rest
- // of the app must see are the displayed dimensions, so a quarter turn
- // swaps them here; ignoring it treats a portrait clip as landscape.
- let (width, height) = if display_rotation(stream).rem_euclid(180) == 90 {
- (height, width)
- } else {
- (width, height)
- };
-
- // `avg_frame_rate` is 0/0 for streams with no constant rate, in which
- // case `r_frame_rate` carries the best guess FFmpeg has.
- let frame_rate = ["avg_frame_rate", "r_frame_rate"]
- .iter()
- .filter_map(|field| stream.get(field).and_then(Value::as_str))
- .find_map(Rational::parse)
- .filter(|rate| !rate.is_zero() && !rate.is_negative())
- .ok_or_else(|| missing("frame rate"))?;
-
- Ok(VideoStream {
- index: u32_field(stream, "index").unwrap_or(0),
- codec: string_field(stream, "codec_name"),
- width,
- height,
- frame_rate: FrameRate::new(frame_rate),
- })
- })())
+ let frame_rate = pick_rate(rational(stream.avg_frame_rate()), rational(stream.rate()))
+ .ok_or_else(|| Error::Probe {
+ path: path.to_path_buf(),
+ detail: "video stream has no usable frame rate".to_owned(),
+ })?;
+
+ Ok(VideoStream {
+ index: stream.index() as u32,
+ codec: parameters.id().name().to_owned(),
+ width,
+ height,
+ frame_rate: FrameRate::new(frame_rate),
+ })
}
-fn parse_audio(stream: &Value) -> Option<AudioStream> {
- if stream.get("codec_type").and_then(Value::as_str) != Some("audio") {
- return None;
- }
- Some(AudioStream {
- index: u32_field(stream, "index").unwrap_or(0),
- codec: string_field(stream, "codec_name"),
- sample_rate: u32_field(stream, "sample_rate").unwrap_or(0),
- channels: u32_field(stream, "channels").unwrap_or(0),
- })
+fn rational(value: ffmpeg::Rational) -> Option<Rational> {
+ (value.denominator() != 0)
+ .then(|| Rational::new(i64::from(value.numerator()), i64::from(value.denominator())))
}
-/// Display rotation in degrees: a Display Matrix in the stream's side data
-/// (how phones record portrait video), or the legacy `rotate` tag older
-/// files carry. Zero when neither says anything.
-fn display_rotation(stream: &Value) -> i64 {
- stream
- .get("side_data_list")
- .and_then(Value::as_array)
+/// `avg_frame_rate` is 0/0 for streams with no constant rate, in which case
+/// `r_frame_rate` carries the best guess FFmpeg has.
+fn pick_rate(average: Option<Rational>, guess: Option<Rational>) -> Option<Rational> {
+ [average, guess]
.into_iter()
.flatten()
- .find_map(|entry| i64_field(entry, "rotation"))
- .or_else(|| stream.get("tags").and_then(|tags| i64_field(tags, "rotate")))
- .unwrap_or(0)
-}
-
-/// Like [`u32_field`], but signed, and tolerant of the float form some
-/// ffprobe builds use for the display matrix angle.
-fn i64_field(value: &Value, field: &str) -> Option<i64> {
- match value.get(field)? {
- Value::Number(number) => {
- number.as_i64().or_else(|| number.as_f64().map(|angle| angle.round() as i64))
- }
- Value::String(text) => text.parse().ok(),
- _ => None,
- }
-}
-
-/// ffprobe is inconsistent about quoting numbers, so accept both forms.
-fn u32_field(stream: &Value, field: &str) -> Option<u32> {
- let value = stream.get(field)?;
- match value {
- Value::Number(number) => number.as_u64().and_then(|n| u32::try_from(n).ok()),
- Value::String(text) => text.parse().ok(),
- _ => None,
- }
-}
-
-fn string_field(stream: &Value, field: &str) -> String {
- stream.get(field).and_then(Value::as_str).unwrap_or("unknown").to_owned()
+ .find(|rate| !rate.is_zero() && !rate.is_negative())
}
#[cfg(test)]
mod tests {
use super::*;
- fn json(text: &str) -> Value {
- serde_json::from_str(text).expect("test json is valid")
- }
-
- #[test]
- fn reads_a_video_stream() {
- let stream = json(
- r#"{"codec_type":"video","index":0,"codec_name":"h264",
- "width":1920,"height":1080,"avg_frame_rate":"30000/1001"}"#,
- );
- let video = parse_video(&stream, Path::new("a.mp4")).expect("is video").expect("parses");
- assert_eq!((video.width, video.height), (1920, 1080));
- assert_eq!(video.frame_rate, FrameRate::NTSC_30);
- assert_eq!(video.codec, "h264");
- }
-
#[test]
fn falls_back_when_avg_frame_rate_is_zero() {
- let stream = json(
- r#"{"codec_type":"video","width":640,"height":480,
- "avg_frame_rate":"0/0","r_frame_rate":"25/1"}"#,
- );
- let video = parse_video(&stream, Path::new("a.mp4")).expect("is video").expect("parses");
- assert_eq!(video.frame_rate, FrameRate::PAL);
- }
-
- #[test]
- fn a_quarter_turn_display_rotation_swaps_the_dimensions() {
- // iPhone portrait HEVC: coded sideways, display matrix says -90.
- let stream = json(
- r#"{"codec_type":"video","width":1920,"height":1080,"avg_frame_rate":"30/1",
- "side_data_list":[{"side_data_type":"Display Matrix","rotation":-90}]}"#,
- );
- let video = parse_video(&stream, Path::new("a.mp4")).expect("is video").expect("parses");
- assert_eq!((video.width, video.height), (1080, 1920));
- }
-
- #[test]
- fn a_half_turn_rotation_keeps_the_dimensions() {
- let stream = json(
- r#"{"codec_type":"video","width":1920,"height":1080,"avg_frame_rate":"30/1",
- "side_data_list":[{"side_data_type":"Display Matrix","rotation":180}]}"#,
- );
- let video = parse_video(&stream, Path::new("a.mp4")).expect("is video").expect("parses");
- assert_eq!((video.width, video.height), (1920, 1080));
- }
-
- #[test]
- fn the_legacy_rotate_tag_counts_too() {
- let stream = json(
- r#"{"codec_type":"video","width":640,"height":480,"avg_frame_rate":"30/1",
- "tags":{"rotate":"90"}}"#,
+ let zero = Some(Rational::ZERO);
+ let pal = Some(Rational::from_int(25));
+ assert_eq!(pick_rate(zero, pal), pal);
+ assert_eq!(pick_rate(None, pal), pal);
+ assert_eq!(
+ pick_rate(Some(Rational::new(30000, 1001)), pal),
+ Some(Rational::new(30000, 1001))
);
- let video = parse_video(&stream, Path::new("a.mp4")).expect("is video").expect("parses");
- assert_eq!((video.width, video.height), (480, 640));
}
#[test]
- fn accepts_numbers_quoted_or_not() {
- let quoted = json(r#"{"width":"640"}"#);
- let bare = json(r#"{"width":640}"#);
- assert_eq!(u32_field(&quoted, "width"), Some(640));
- assert_eq!(u32_field(&bare, "width"), Some(640));
+ fn no_usable_rate_is_none() {
+ assert_eq!(pick_rate(Some(Rational::ZERO), Some(Rational::ZERO)), None);
+ assert_eq!(pick_rate(None, None), None);
}
#[test]
- fn a_video_stream_with_no_usable_rate_is_an_error_not_a_skip() {
- let stream = json(r#"{"codec_type":"video","width":64,"height":64,"avg_frame_rate":"0/0"}"#);
- let result = parse_video(&stream, Path::new("a.mp4")).expect("is video");
- assert!(matches!(result, Err(Error::Probe { .. })));
- }
-
- #[test]
- fn ignores_non_video_streams() {
- assert!(parse_video(&json(r#"{"codec_type":"audio"}"#), Path::new("a.mp4")).is_none());
- }
-
- #[test]
- fn reads_an_audio_stream() {
- let stream = json(
- r#"{"codec_type":"audio","index":1,"codec_name":"aac",
- "sample_rate":"48000","channels":2}"#,
- );
- let audio = parse_audio(&stream).expect("is audio");
- assert_eq!((audio.sample_rate, audio.channels), (48000, 2));
- assert_eq!(audio.codec, "aac");
+ fn a_missing_file_is_an_error_not_a_panic() {
+ assert!(probe("does-not-exist.mp4").is_err());
}
}
diff --git a/engine/crates/concat-media/src/process.rs b/engine/crates/concat-media/src/process.rs
deleted file mode 100644
index 75b98fbb..00000000
--- a/engine/crates/concat-media/src/process.rs
+++ /dev/null
@@ -1,153 +0,0 @@
-// SPDX-License-Identifier: AGPL-3.0-or-later
-// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
-
-//! Shared plumbing for the FFmpeg children this crate spawns.
-//!
-//! Every invocation goes through [`base_command`], and every long-lived child
-//! drains its stderr through [`StderrTail`]. The tail is why: a packaged app
-//! has no terminal, so an inherited stderr sends FFmpeg's one useful line into
-//! the void and leaves the user with "exited with status 1". Draining on a
-//! thread also means the pipe can never fill up and stall the child.
-
-use std::ffi::OsStr;
-use std::io::Read;
-use std::path::Path;
-use std::process::{Child, Command, Stdio};
-use std::sync::{Arc, Mutex};
-use std::thread::JoinHandle;
-
-use crate::error::{Error, Result};
-
-/// How much of the child's stderr to keep. At `-loglevel error` FFmpeg says
-/// little; this holds the message that matters plus its context.
-const STDERR_KEEP: usize = 8 * 1024;
-
-/// A `Command` for `binary` that never flashes a console window.
-///
-/// On Windows, a GUI-subsystem parent has no console, so each console-subsystem
-/// child (FFmpeg, ffprobe, ...) gets a fresh one that pops up over the app.
-/// `CREATE_NO_WINDOW` suppresses that. Elsewhere this is just `Command::new`.
-pub fn command(binary: impl AsRef<OsStr>) -> Command {
- #[allow(unused_mut)]
- let mut command = Command::new(binary.as_ref());
- #[cfg(windows)]
- {
- use std::os::windows::process::CommandExt;
- const CREATE_NO_WINDOW: u32 = 0x0800_0000;
- command.creation_flags(CREATE_NO_WINDOW);
- }
- command
-}
-
-/// A command with the flags every invocation wants: no banner, no terminal
-/// interaction, errors only.
-pub(crate) fn base_command(binary: &Path) -> Command {
- let mut command = command(binary);
- command.args(["-hide_banner", "-nostdin", "-loglevel", "error"]);
- command
-}
-
-/// Collects a child's stderr on a drain thread, keeping the tail for error
-/// messages.
-pub(crate) struct StderrTail {
- buffer: Arc<Mutex<Vec<u8>>>,
- handle: Option<JoinHandle<()>>,
-}
-
-impl StderrTail {
- /// Starts draining `child`'s stderr, which must have been spawned with
- /// `Stdio::piped()`. A child whose stderr was not piped yields an empty
- /// tail rather than an error.
- pub(crate) fn drain(child: &mut Child) -> Self {
- let buffer = Arc::new(Mutex::new(Vec::new()));
- let handle = child.stderr.take().map(|mut stderr| {
- let buffer = Arc::clone(&buffer);
- std::thread::spawn(move || {
- let mut chunk = [0u8; 4096];
- loop {
- match stderr.read(&mut chunk) {
- Ok(0) => break,
- Ok(read) => {
- let mut buffer = buffer.lock().expect("stderr buffer poisoned");
- buffer.extend_from_slice(&chunk[..read]);
- let excess = buffer.len().saturating_sub(STDERR_KEEP);
- if excess > 0 {
- buffer.drain(..excess);
- }
- }
- Err(_) => break,
- }
- }
- })
- });
- Self { buffer, handle }
- }
-
- /// The last few lines the child printed, joined onto one line.
- ///
- /// Call only after the child has been waited on: EOF on the pipe is what
- /// lets the drain thread finish, and joining it first is what makes the
- /// buffer complete.
- pub(crate) fn summary(&mut self) -> String {
- if let Some(handle) = self.handle.take() {
- let _ = handle.join();
- }
- let buffer = self.buffer.lock().expect("stderr buffer poisoned");
- summarize(&buffer)
- }
-}
-
-/// The last three non-empty lines, joined onto one. What an error message has
-/// room for, and with `-loglevel error` usually the whole story.
-pub(crate) fn summarize(stderr: &[u8]) -> String {
- let text = String::from_utf8_lossy(stderr);
- let mut lines: Vec<&str> = text.lines().map(str::trim).filter(|line| !line.is_empty()).collect();
- let tail = lines.split_off(lines.len().saturating_sub(3));
- if tail.is_empty() {
- "no error output captured".to_owned()
- } else {
- tail.join(" / ")
- }
-}
-
-/// Runs a prepared command to completion, capturing stderr, and turns a
-/// non-zero exit into [`Error::Exited`] that carries what the child said.
-pub(crate) fn run_to_completion(
- command: &mut Command,
- program: &'static str,
- path: &Path,
-) -> Result<()> {
- let output = command
- .stdin(Stdio::null())
- .stdout(Stdio::null())
- .stderr(Stdio::piped())
- .output()
- .map_err(|source| Error::Spawn { program, source })?;
-
- if output.status.success() {
- return Ok(());
- }
- Err(Error::Exited {
- program,
- path: path.to_path_buf(),
- status: output.status,
- stderr: summarize(&output.stderr),
- })
-}
-
-#[cfg(test)]
-mod tests {
- use super::*;
-
- #[test]
- fn summarize_keeps_the_last_three_lines() {
- let text = b"one\ntwo\n\nthree\nfour\n";
- assert_eq!(summarize(text), "two / three / four");
- }
-
- #[test]
- fn summarize_of_nothing_says_so() {
- assert_eq!(summarize(b""), "no error output captured");
- assert_eq!(summarize(b"\n \n"), "no error output captured");
- }
-}
diff --git a/engine/crates/concat-media/src/samples.rs b/engine/crates/concat-media/src/samples.rs
new file mode 100644
index 00000000..ad04977b
--- /dev/null
+++ b/engine/crates/concat-media/src/samples.rs
@@ -0,0 +1,413 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Pulling samples out of a file.
+//!
+//! One audio decoder for everything that needs sound as numbers: the
+//! waveform peaks, the playback cache, the transcriber. Each asks for a
+//! window of the file, a filter chain to run it through, and the rate,
+//! channel count and sample format it wants back; libavfilter does the
+//! trimming, the filtering and the resampling in one graph, so a clip's
+//! speed and effects mean exactly the same thing here as in the export mix.
+
+use std::path::{Path, PathBuf};
+
+use ffmpeg_the_third as ffmpeg;
+use ffmpeg_the_third::codec::decoder;
+use ffmpeg_the_third::filter;
+use ffmpeg_the_third::format;
+use ffmpeg_the_third::util::frame::audio::Audio;
+
+use crate::error::{Error, Result};
+use crate::ffi;
+
+/// The sample format decoded audio comes back in. Interleaved either way.
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum SampleFormat {
+ /// Signed 16-bit.
+ I16,
+ /// 32-bit float in [-1, 1].
+ F32,
+}
+
+impl SampleFormat {
+ fn name(self) -> &'static str {
+ match self {
+ SampleFormat::I16 => "s16",
+ SampleFormat::F32 => "flt",
+ }
+ }
+
+ fn bytes(self) -> usize {
+ match self {
+ SampleFormat::I16 => 2,
+ SampleFormat::F32 => 4,
+ }
+ }
+}
+
+/// What to decode and how it should come back.
+#[derive(Clone, Debug)]
+pub struct AudioOptions {
+ /// Seconds into the file to start. The cut is exact: the container
+ /// seeks near it, the graph trims to it.
+ pub start: Option<f64>,
+ /// Seconds of source to decode from `start`, or the rest of the file.
+ pub duration: Option<f64>,
+ /// FFmpeg audio filters to run the window through, in order, before
+ /// resampling. Each is validated like a clip's chain.
+ pub filters: Vec<String>,
+ /// Samples per second to come back at.
+ pub rate: u32,
+ /// One for mono, two for stereo.
+ pub channels: u16,
+ /// The sample format to come back in.
+ pub format: SampleFormat,
+}
+
+impl Default for AudioOptions {
+ fn default() -> Self {
+ Self {
+ start: None,
+ duration: None,
+ filters: Vec::new(),
+ rate: 48_000,
+ channels: 2,
+ format: SampleFormat::I16,
+ }
+ }
+}
+
+/// The graph between the codec and the caller, as one string.
+fn audio_filter(options: &AudioOptions) -> String {
+ let mut parts: Vec<String> = Vec::new();
+ match (options.start, options.duration) {
+ (Some(start), Some(duration)) => {
+ parts.push(format!("atrim=start={start:.6}:duration={duration:.6}"));
+ parts.push("asetpts=PTS-STARTPTS".to_owned());
+ }
+ (Some(start), None) => {
+ parts.push(format!("atrim=start={start:.6}"));
+ parts.push("asetpts=PTS-STARTPTS".to_owned());
+ }
+ (None, Some(duration)) => {
+ parts.push(format!("atrim=duration={duration:.6}"));
+ }
+ (None, None) => {}
+ }
+ parts.extend(
+ options
+ .filters
+ .iter()
+ .filter(|filter| !filter.is_empty())
+ .cloned(),
+ );
+ parts.push(format!("aresample={}", options.rate));
+ parts.push(format!(
+ "aformat=sample_fmts={}:sample_rates={}:channel_layouts={}",
+ options.format.name(),
+ options.rate,
+ if options.channels == 1 {
+ "mono"
+ } else {
+ "stereo"
+ }
+ ));
+ parts.join(",")
+}
+
+/// Decodes a file's audio through libavcodec and libavfilter.
+pub struct AudioDecoder {
+ path: PathBuf,
+ input: format::context::Input,
+ stream: usize,
+ time_base: ffmpeg::Rational,
+ decoder: decoder::Audio,
+ options: AudioOptions,
+ graph: Option<filter::Graph>,
+ /// The codec has been drained and the graph flushed; only what the sink
+ /// still holds is left.
+ flushed: bool,
+ done: bool,
+}
+
+impl AudioDecoder {
+ /// Opens `path`'s first audio stream.
+ pub fn open(path: impl AsRef<Path>, options: &AudioOptions) -> Result<Self> {
+ ffi::init();
+ for chain in &options.filters {
+ crate::audio::validate_chain(chain)?;
+ }
+ if options.channels != 1 && options.channels != 2 {
+ return Err(Error::Probe {
+ path: path.as_ref().to_path_buf(),
+ detail: format!(
+ "{} channels requested; mono and stereo are the choices",
+ options.channels
+ ),
+ });
+ }
+
+ let path = path.as_ref();
+ let mut input =
+ ffmpeg::format::input(path).map_err(|error| ffi::fail("open", path, error))?;
+ let stream = input
+ .streams()
+ .best(ffmpeg::media::Type::Audio)
+ .ok_or_else(|| Error::NoAudioStream {
+ path: path.to_path_buf(),
+ })?;
+ let stream_index = stream.index();
+ let time_base = stream.time_base();
+ let context = ffmpeg::codec::Context::from_parameters(stream.parameters())
+ .map_err(|error| ffi::fail("codec parameters", path, error))?;
+ let decoder = context
+ .decoder()
+ .audio()
+ .map_err(|error| ffi::fail("open decoder", path, error))?;
+
+ // Near the start, not at it: the graph's atrim does the exact cut,
+ // and a seek only saves decoding what comes before.
+ if let Some(start) = options.start
+ && start > 0.0
+ {
+ let target = (start * f64::from(ffmpeg::sys::AV_TIME_BASE)) as i64;
+ input
+ .seek(target, ..=target)
+ .map_err(|error| ffi::fail("seek", path, error))?;
+ }
+
+ Ok(Self {
+ path: path.to_path_buf(),
+ input,
+ stream: stream_index,
+ time_base,
+ decoder,
+ options: options.clone(),
+ graph: None,
+ flushed: false,
+ done: false,
+ })
+ }
+
+ /// Samples per second of what comes back.
+ pub fn rate(&self) -> u32 {
+ self.options.rate
+ }
+
+ /// Channels of what comes back.
+ pub fn channels(&self) -> u16 {
+ self.options.channels
+ }
+
+ fn build_graph(&self, first: &Audio) -> Result<filter::Graph> {
+ let mut graph = filter::Graph::new();
+ let args = format!(
+ "time_base={}/{}:sample_rate={}:sample_fmt={}:channel_layout={}",
+ self.time_base.numerator(),
+ self.time_base.denominator(),
+ first.rate(),
+ first.format().name(),
+ first.ch_layout().description()
+ );
+ let missing = |name: &str| Error::Missing {
+ what: "filter",
+ name: name.to_owned(),
+ };
+ graph
+ .add(
+ &filter::find("abuffer").ok_or_else(|| missing("abuffer"))?,
+ "in",
+ &args,
+ )
+ .map_err(|error| ffi::fail("buffer source", &self.path, error))?;
+ graph
+ .add(
+ &filter::find("abuffersink").ok_or_else(|| missing("abuffersink"))?,
+ "out",
+ "",
+ )
+ .map_err(|error| ffi::fail("buffer sink", &self.path, error))?;
+ graph
+ .output("in", 0)
+ .and_then(|parser| parser.input("out", 0))
+ .and_then(|parser| parser.parse(&audio_filter(&self.options)))
+ .map_err(|error| ffi::fail("filter graph", &self.path, error))?;
+ graph
+ .validate()
+ .map_err(|error| ffi::fail("filter graph", &self.path, error))?;
+ Ok(graph)
+ }
+
+ /// One decoded frame out of the codec, or `None` when the file is over.
+ fn next_source(&mut self) -> Result<Option<Audio>> {
+ loop {
+ let mut frame = Audio::empty();
+ match self.decoder.receive_frame(&mut frame) {
+ Ok(()) => return Ok(Some(frame)),
+ Err(ffmpeg::Error::Eof) => return Ok(None),
+ Err(error) if ffi::is_again(&error) => {}
+ Err(error) => return Err(ffi::fail("decode", &self.path, error)),
+ }
+ loop {
+ let mut packet = ffmpeg::Packet::empty();
+ match packet.read(&mut self.input) {
+ Ok(()) => {
+ if packet.stream() != self.stream {
+ continue;
+ }
+ match self.decoder.send_packet(&packet) {
+ Ok(()) => break,
+ Err(error) if ffi::is_again(&error) => break,
+ Err(error) => return Err(ffi::fail("send packet", &self.path, error)),
+ }
+ }
+ Err(ffmpeg::Error::Eof) => {
+ let _ = self.decoder.send_eof();
+ break;
+ }
+ Err(error) => return Err(ffi::fail("read", &self.path, error)),
+ }
+ }
+ }
+ }
+
+ /// The next frame out of the graph: interleaved samples at the requested
+ /// rate and layout, or `None` once the window is exhausted.
+ pub fn next_frame(&mut self) -> Result<Option<Audio>> {
+ loop {
+ if self.done {
+ return Ok(None);
+ }
+ if let Some(graph) = self.graph.as_mut() {
+ let mut out = Audio::empty();
+ let pulled = {
+ let mut context = graph.get("out").expect("the graph has an output");
+ context.sink().frame(&mut out)
+ };
+ match pulled {
+ Ok(()) => return Ok(Some(out)),
+ Err(ffmpeg::Error::Eof) => {
+ self.done = true;
+ return Ok(None);
+ }
+ Err(error) if ffi::is_again(&error) => {}
+ Err(error) => return Err(ffi::fail("filter output", &self.path, error)),
+ }
+ if self.flushed {
+ self.done = true;
+ return Ok(None);
+ }
+ }
+
+ // The graph wants more. Feed it the next decoded frame, or tell
+ // it the file is over.
+ match self.next_source()? {
+ Some(frame) => {
+ if self.graph.is_none() {
+ self.graph = Some(self.build_graph(&frame)?);
+ }
+ let graph = self.graph.as_mut().expect("just built");
+ let mut context = graph.get("in").expect("the graph has an input");
+ context
+ .source()
+ .add(&frame)
+ .map_err(|error| ffi::fail("filter", &self.path, error))?;
+ }
+ None => {
+ let Some(graph) = self.graph.as_mut() else {
+ // No audio frame at all - a stream that claims sound
+ // and delivers none.
+ self.done = true;
+ return Ok(None);
+ };
+ let mut context = graph.get("in").expect("the graph has an input");
+ let _ = context.source().flush();
+ self.flushed = true;
+ }
+ }
+ }
+ }
+
+ /// The interleaved bytes of one frame from the sink: samples times
+ /// channels times the sample width, without the row padding libav adds.
+ pub fn bytes_of(&self, frame: &Audio) -> Vec<u8> {
+ let wanted =
+ frame.samples() * usize::from(self.options.channels) * self.options.format.bytes();
+ let data = frame.data(0);
+ data[..wanted.min(data.len())].to_vec()
+ }
+
+ /// The next frame as interleaved 16-bit samples. Only for decoders
+ /// opened with [`SampleFormat::I16`].
+ pub fn next_i16(&mut self) -> Result<Option<Vec<i16>>> {
+ let Some(frame) = self.next_frame()? else {
+ return Ok(None);
+ };
+ let bytes = self.bytes_of(&frame);
+ Ok(Some(
+ bytes
+ .chunks_exact(2)
+ .map(|pair| i16::from_le_bytes([pair[0], pair[1]]))
+ .collect(),
+ ))
+ }
+
+ /// The next frame as interleaved floats. Only for decoders opened with
+ /// [`SampleFormat::F32`].
+ pub fn next_f32(&mut self) -> Result<Option<Vec<f32>>> {
+ let Some(frame) = self.next_frame()? else {
+ return Ok(None);
+ };
+ let bytes = self.bytes_of(&frame);
+ Ok(Some(
+ bytes
+ .chunks_exact(4)
+ .map(|chunk| f32::from_le_bytes([chunk[0], chunk[1], chunk[2], chunk[3]]))
+ .collect(),
+ ))
+ }
+
+ /// Everything left in the window, as interleaved floats.
+ pub fn collect_f32(&mut self) -> Result<Vec<f32>> {
+ let mut all = Vec::new();
+ while let Some(chunk) = self.next_f32()? {
+ all.extend_from_slice(&chunk);
+ }
+ Ok(all)
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn the_graph_trims_filters_and_lands_on_the_requested_shape() {
+ let options = AudioOptions {
+ start: Some(1.5),
+ duration: Some(2.0),
+ filters: vec!["atempo=2.000000".to_owned(), String::new()],
+ rate: 16_000,
+ channels: 1,
+ format: SampleFormat::F32,
+ };
+ assert_eq!(
+ audio_filter(&options),
+ "atrim=start=1.500000:duration=2.000000,asetpts=PTS-STARTPTS,atempo=2.000000,\
+ aresample=16000,aformat=sample_fmts=flt:sample_rates=16000:channel_layouts=mono"
+ );
+ }
+
+ #[test]
+ fn a_whole_file_has_no_trim() {
+ let graph = audio_filter(&AudioOptions::default());
+ assert!(graph.starts_with("aresample=48000,"), "{graph}");
+ assert!(graph.ends_with("channel_layouts=stereo"), "{graph}");
+ }
+
+ #[test]
+ fn a_missing_file_is_an_error_not_a_panic() {
+ assert!(AudioDecoder::open("does-not-exist.mp3", &AudioOptions::default()).is_err());
+ }
+}
diff --git a/engine/crates/concat-media/tests/ffi_decode.rs b/engine/crates/concat-media/tests/ffi_decode.rs
deleted file mode 100644
index 7d7cb91a..00000000
--- a/engine/crates/concat-media/tests/ffi_decode.rs
+++ /dev/null
@@ -1,151 +0,0 @@
-// SPDX-License-Identifier: AGPL-3.0-or-later
-// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
-
-//! End-to-end checks for the linked decoder.
-//!
-//! These are the claims that justify the FFI backend existing at all - real
-//! presentation timestamps and frame-accurate seeking - so they are tested
-//! against an actual file rather than asserted in a doc comment.
-//!
-//! The fixture is generated with the `ffmpeg` binary at test time. If FFmpeg
-//! is not installed the tests skip rather than fail: a missing tool is a
-//! missing tool, not a broken decoder.
-
-#![cfg(feature = "ffi")]
-
-use std::path::PathBuf;
-use std::process::Command;
-
-use concat_core::time::Rational;
-use concat_media::decode::{FrameSource, SeekableSource};
-use concat_media::ffi::FfiDecoder;
-
-const RATE: i64 = 30;
-const SECONDS: i64 = 4;
-
-/// Renders a deterministic clip and returns its path, or `None` without FFmpeg.
-fn fixture() -> Option<PathBuf> {
- let path = std::env::temp_dir().join("concat-ffi-fixture.mp4");
- if path.exists() {
- return Some(path);
- }
-
- let status = Command::new("ffmpeg")
- .args(["-hide_banner", "-loglevel", "error", "-y"])
- .args(["-f", "lavfi", "-i", &format!("testsrc2=size=320x240:rate={RATE}")])
- .args(["-t", &SECONDS.to_string()])
- // A short keyframe interval, so a seek has something nearby to land on
- // and the test measures our decode-forward rather than FFmpeg's luck.
- .args(["-g", "15", "-c:v", "libx264", "-pix_fmt", "yuv420p"])
- .arg(&path)
- .status()
- .ok()?;
-
- status.success().then_some(path)
-}
-
-/// One frame's duration, as the tolerance for "landed on the right frame".
-fn frame_duration() -> Rational {
- Rational::new(1, RATE)
-}
-
-#[test]
-fn reports_real_presentation_timestamps() {
- let Some(path) = fixture() else { return };
-
- let mut decoder = FfiDecoder::open(&path, 160, 120).expect("open");
- let mut times = Vec::new();
-
- for _ in 0..10 {
- let frame = decoder.next_frame().expect("decode").expect("a frame");
- assert_eq!((frame.width(), frame.height()), (160, 120), "scaled to the requested size");
-
- let position = decoder.position().expect("a linked decoder always knows the time");
- times.push(position);
- }
-
- // The whole point: these come from the container, not from counting.
- assert_eq!(times[0], Rational::ZERO, "first frame starts at zero");
-
- for (index, time) in times.iter().enumerate() {
- let expected = frame_duration() * Rational::from_int(index as i64);
- assert_eq!(*time, expected, "frame {index} should sit exactly on its boundary");
- }
-
- assert!(times.windows(2).all(|pair| pair[0] < pair[1]), "timestamps must increase");
-}
-
-#[test]
-fn seeks_to_an_exact_frame() {
- let Some(path) = fixture() else { return };
-
- let mut decoder = FfiDecoder::open(&path, 160, 120).expect("open");
-
- // Deliberately not a keyframe: at -g 15 the keyframes are every half
- // second, so 2.4s sits between two of them. Landing there is only possible
- // by decoding forward from the previous one, which is the thing a pipe
- // cannot do.
- let target = Rational::new(24, 10);
- decoder.seek(target).expect("seek");
-
- let mut landed = None;
- for _ in 0..64 {
- decoder.next_frame().expect("decode").expect("a frame");
- let position = decoder.position().expect("position");
- if position >= target {
- landed = Some(position);
- break;
- }
- }
-
- let landed = landed.expect("never reached the target after a seek");
- let drift = landed - target;
- assert!(
- drift >= Rational::ZERO && drift < frame_duration(),
- "landed at {landed} for a target of {target}, which is more than one frame out",
- );
-}
-
-#[test]
-fn seeking_backwards_works_too() {
- let Some(path) = fixture() else { return };
-
- let mut decoder = FfiDecoder::open(&path, 64, 64).expect("open");
-
- // Run well into the file first, so the seek genuinely goes backwards and
- // has stale decoder state to flush.
- for _ in 0..60 {
- decoder.next_frame().expect("decode");
- }
- let before = decoder.position().expect("position");
- assert!(before > Rational::ONE, "should be past a second by now");
-
- decoder.seek(Rational::new(1, 2)).expect("seek back");
- decoder.next_frame().expect("decode").expect("a frame");
-
- let after = decoder.position().expect("position");
- assert!(after < before, "seeking back should move the position back");
- assert!(after <= Rational::new(6, 10), "should land near the half-second mark, got {after}");
-}
-
-#[test]
-fn runs_to_the_end_and_stops() {
- let Some(path) = fixture() else { return };
-
- let mut decoder = FfiDecoder::open(&path, 64, 64).expect("open");
- let mut count = 0u32;
- while decoder.next_frame().expect("decode").is_some() {
- count += 1;
- assert!(count < 1000, "decoder never reported end of stream");
- }
-
- // Flushing at EOF is what makes the last few frames come out at all; a
- // decoder that forgets it silently loses them.
- let expected = (RATE * SECONDS) as u32;
- assert!(
- count.abs_diff(expected) <= 1,
- "decoded {count} frames, expected about {expected}",
- );
-
- assert!(decoder.next_frame().expect("decode").is_none(), "end of stream is sticky");
-}
diff --git a/engine/crates/concat-project/src/doc.rs b/engine/crates/concat-project/src/doc.rs
index 76ce3b1d..79023281 100644
--- a/engine/crates/concat-project/src/doc.rs
+++ b/engine/crates/concat-project/src/doc.rs
@@ -4,7 +4,7 @@
//! Reading and writing `concat.json`.
//!
//! Projects saved before the renames carry the same document as
-//! `wolfcut.json` or `relay.json`. The name on disk is the host's concern:
+//! `wolfcut.json`. The name on disk is the host's concern:
//! it keeps whichever name a project already has, and this module never
//! looks at a filename.
//!
diff --git a/engine/crates/concat-speech/Cargo.toml b/engine/crates/concat-speech/Cargo.toml
new file mode 100644
index 00000000..f2eb7a15
--- /dev/null
+++ b/engine/crates/concat-speech/Cargo.toml
@@ -0,0 +1,35 @@
+[package]
+name = "concat-speech"
+version.workspace = true
+edition.workspace = true
+rust-version.workspace = true
+license.workspace = true
+description = "Speech in and out: on-device transcription with whisper.cpp, and text to speech with Kokoro."
+
+[dependencies]
+concat-host.workspace = true
+concat-media.workspace = true
+serde = { version = "1", features = ["derive"] }
+serde_json.workspace = true
+
+# Model downloads. Blocking on purpose: downloads run on their own thread,
+# and an async HTTP stack would be a second runtime's worth of dependencies
+# for one GET.
+ureq = "2"
+
+# Text to speech: the official k2-fsa bindings, linked statically so the
+# binary carries no extra dylibs. Kokoro models arrive as tar.bz2 archives,
+# hence tar + bzip2 (pure-Rust decoder) for unpacking them.
+sherpa-onnx = "1.13"
+tar = "0.4"
+bzip2 = "0.6"
+
+# Transcription: whisper.cpp in-process. Built from source by cmake at build
+# time; on macOS with Metal so the encoder runs on the GPU.
+whisper-rs = "0.16"
+
+[target.'cfg(target_os = "macos")'.dependencies]
+whisper-rs = { version = "0.16", features = ["metal"] }
+
+[lints]
+workspace = true
diff --git a/engine/crates/concat-speech/src/lib.rs b/engine/crates/concat-speech/src/lib.rs
new file mode 100644
index 00000000..032bc43f
--- /dev/null
+++ b/engine/crates/concat-speech/src/lib.rs
@@ -0,0 +1,108 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Speech in and out, entirely on this machine.
+//!
+//! - [`transcribe`] - whisper.cpp, in-process, turns a clip's audio into
+//! timed caption segments.
+//! - [`tts`] - Kokoro, through sherpa-onnx, turns typed narration into a
+//! WAV in the project folder.
+//!
+//! Both download their models on demand into the app's data directory and
+//! never bundle them. Both are one-at-a-time: a [`concat_host::SingleFlight`]
+//! refuses a second concurrent run rather than letting two share a cancel
+//! flag.
+//!
+//! A separate crate from `concat-host` so the heavy native dependency
+//! (sherpa-onnx's static libraries) stays out of everything that does not
+//! speak.
+
+pub mod transcribe;
+pub mod tts;
+
+pub use transcribe::Transcriber;
+pub use tts::Speech;
+
+/// Progress for one model download.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct DownloadProgress {
+ /// Which model.
+ pub id: String,
+ /// Bytes received so far.
+ pub received: u64,
+ /// Content-Length when the server sent one, the table estimate otherwise.
+ pub total: u64,
+ /// True while an archive is being unpacked - bytes stop moving but the
+ /// job is far from done, and the bar should say so.
+ pub unpacking: bool,
+ /// True on the final report.
+ pub done: bool,
+}
+
+/// Streams `url` into `partial`, reporting every couple of megabytes and
+/// stopping when `cancel` is set. Shared by both model downloaders.
+fn download_to(
+ url: &str,
+ partial: &std::path::Path,
+ id: &str,
+ estimate: u64,
+ cancel: &std::sync::atomic::AtomicBool,
+ progress: &mut dyn FnMut(DownloadProgress),
+) -> Result<(u64, u64), String> {
+ use std::io::Read;
+ use std::sync::atomic::Ordering;
+
+ // A read timeout, or a stalled connection blocks in `read` forever
+ // with the cancel flag unreachable - the flag is only checked between
+ // reads. Thirty seconds without a byte means the download is dead.
+ let agent = ureq::AgentBuilder::new()
+ .timeout_connect(std::time::Duration::from_secs(20))
+ .timeout_read(std::time::Duration::from_secs(30))
+ .build();
+ let response = agent
+ .get(url)
+ .call()
+ .map_err(|error| format!("download failed: {error}"))?;
+ let total = response
+ .header("Content-Length")
+ .and_then(|value| value.parse::<u64>().ok())
+ .unwrap_or(estimate);
+
+ let mut file = std::fs::File::create(partial)
+ .map_err(|error| format!("could not create {}: {error}", partial.display()))?;
+
+ let mut reader = response.into_reader();
+ let mut buffer = [0u8; 64 * 1024];
+ let mut received: u64 = 0;
+ let mut last_report: u64 = 0;
+
+ loop {
+ if cancel.load(Ordering::Relaxed) {
+ drop(file);
+ let _ = std::fs::remove_file(partial);
+ return Err("download cancelled".to_owned());
+ }
+ let read = reader
+ .read(&mut buffer)
+ .map_err(|error| format!("download interrupted: {error}"))?;
+ if read == 0 {
+ break;
+ }
+ std::io::Write::write_all(&mut file, &buffer[..read])
+ .map_err(|error| format!("could not write model: {error}"))?;
+ received += read as u64;
+
+ // Every 2 MB, not every chunk: a progress bar cannot use more.
+ if received - last_report >= 2 * 1024 * 1024 {
+ last_report = received;
+ progress(DownloadProgress {
+ id: id.to_owned(),
+ received,
+ total,
+ unpacking: false,
+ done: false,
+ });
+ }
+ }
+ Ok((received, received.max(total)))
+}
diff --git a/engine/crates/concat-speech/src/transcribe.rs b/engine/crates/concat-speech/src/transcribe.rs
new file mode 100644
index 00000000..82f987e9
--- /dev/null
+++ b/engine/crates/concat-speech/src/transcribe.rs
@@ -0,0 +1,439 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Whisper transcription: audio in, timed caption segments out.
+//!
+//! whisper.cpp runs in-process through `whisper-rs`, the way Kokoro runs
+//! through sherpa-onnx: no binary to ship or find, and cancellation is a
+//! callback returning `true`. On macOS the Metal build is used, so the
+//! encoder runs on the GPU.
+//!
+//! Two pieces:
+//!
+//! 1. **Models.** Downloaded on demand from Hugging Face into the app data
+//! folder as plain `.bin` files, streamed with progress, written to a
+//! `.part` and renamed - a torn download must never look usable. A loaded
+//! model is cached until another is asked for, because loading one takes
+//! seconds.
+//! 2. **Transcription.** The engine's audio decoder hands over the clip's
+//! window as 16 kHz mono floats - the input whisper wants - and the
+//! segments come back relative to the window.
+
+use std::path::PathBuf;
+use std::sync::atomic::Ordering;
+use std::sync::{Arc, Mutex};
+
+use concat_host::{AppDirs, SingleFlight};
+use concat_media::{AudioDecoder, AudioOptions, SampleFormat};
+use serde::{Deserialize, Serialize};
+use whisper_rs::{FullParams, SamplingStrategy, WhisperContext, WhisperContextParameters};
+
+use crate::DownloadProgress;
+
+/// One downloadable Whisper model. Sizes are approximate, for progress bars
+/// when the server declines to send a Content-Length.
+struct KnownModel {
+ id: &'static str,
+ label: &'static str,
+ /// One line for the settings row: what this size trades away.
+ blurb: &'static str,
+ approx_bytes: u64,
+ english_only: bool,
+}
+
+/// The models the settings panel offers, fastest first.
+///
+/// All from the official `ggerganov/whisper.cpp` conversions. Nothing bigger
+/// than `small`: `medium` is 1.5 GB and minutes-per-minute on CPU, which is
+/// the opposite of what this feature is for.
+const KNOWN_MODELS: &[KnownModel] = &[
+ KnownModel {
+ id: "tiny.en",
+ label: "Tiny (English)",
+ blurb: "Fastest draft. Rough on names and punctuation.",
+ approx_bytes: 77_700_000,
+ english_only: true,
+ },
+ KnownModel {
+ id: "tiny",
+ label: "Tiny (Multilingual)",
+ blurb: "Fastest draft, any language.",
+ approx_bytes: 77_700_000,
+ english_only: false,
+ },
+ KnownModel {
+ id: "base.en",
+ label: "Base (English)",
+ blurb: "The sweet spot: solid captions at ~10x realtime.",
+ approx_bytes: 147_400_000,
+ english_only: true,
+ },
+ KnownModel {
+ id: "base",
+ label: "Base (Multilingual)",
+ blurb: "Solid captions, any language.",
+ approx_bytes: 147_500_000,
+ english_only: false,
+ },
+ KnownModel {
+ id: "small.en",
+ label: "Small (English)",
+ blurb: "Noticeably better wording; a few times slower.",
+ approx_bytes: 487_600_000,
+ english_only: true,
+ },
+ KnownModel {
+ id: "small",
+ label: "Small (Multilingual)",
+ blurb: "Best quality offered, any language.",
+ approx_bytes: 487_600_000,
+ english_only: false,
+ },
+];
+
+fn known(id: &str) -> Option<&'static KnownModel> {
+ KNOWN_MODELS.iter().find(|model| model.id == id)
+}
+
+fn model_url(id: &str) -> String {
+ format!("https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-{id}.bin")
+}
+
+/// Where downloaded models live: `<app data>/whisper-models/ggml-<id>.bin`.
+fn models_dir(dirs: &AppDirs) -> Result<PathBuf, String> {
+ let dir = dirs.data.join("whisper-models");
+ std::fs::create_dir_all(&dir)
+ .map_err(|error| format!("could not create {}: {error}", dir.display()))?;
+ Ok(dir)
+}
+
+fn model_file(dirs: &AppDirs, id: &str) -> Result<PathBuf, String> {
+ // Ids come from our own table; anything else is a bug, not input.
+ known(id).ok_or_else(|| format!("unknown model {id:?}"))?;
+ Ok(models_dir(dirs)?.join(format!("ggml-{id}.bin")))
+}
+
+/// One model, as the settings panel shows it.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct ModelStatus {
+ /// The model id, e.g. "base.en".
+ pub id: String,
+ /// Display name.
+ pub label: String,
+ /// One line on what this size trades away.
+ pub blurb: String,
+ /// Whether the model only understands English.
+ pub english_only: bool,
+ /// Approximate download size in bytes, for display.
+ pub size_bytes: u64,
+ /// Whether the file is on disk.
+ pub downloaded: bool,
+}
+
+/// What the settings panel shows.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct TranscriberStatus {
+ /// Where models are stored, so the settings panel can say so.
+ pub models_dir: String,
+ /// Every model the panel offers.
+ pub models: Vec<ModelStatus>,
+}
+
+/// One caption, in seconds relative to the transcribed window's start.
+#[derive(Serialize, Clone, Debug, PartialEq)]
+#[serde(rename_all = "camelCase")]
+pub struct Segment {
+ /// Seconds from the window's start.
+ pub start: f64,
+ /// Seconds from the window's start.
+ pub end: f64,
+ /// What was said.
+ pub text: String,
+}
+
+/// What to transcribe.
+#[derive(Deserialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct TranscribeRequest {
+ /// The media file to transcribe.
+ pub path: String,
+ /// Seconds into the file where the clip's source window begins.
+ pub source_start: f64,
+ /// How much source the clip covers, in seconds (`duration * speed`).
+ pub window: f64,
+ /// Whisper language code, or "auto".
+ pub language: String,
+ /// Which model to use, e.g. "base.en".
+ pub model_id: String,
+}
+
+struct LoadedModel {
+ id: String,
+ context: WhisperContext,
+}
+
+/// The transcriber: the one transcription and the one model download that
+/// can run at a time, and the model kept loaded between runs.
+pub struct Transcriber {
+ gate: Arc<SingleFlight>,
+ downloads: Arc<SingleFlight>,
+ /// Held for the whole transcription, which is what lets
+ /// [`Transcriber::delete_model`] refuse to delete a model mid-use.
+ model: Mutex<Option<LoadedModel>>,
+}
+
+impl Default for Transcriber {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl Transcriber {
+ /// An idle transcriber, loading nothing until asked to transcribe.
+ pub fn new() -> Self {
+ Self {
+ gate: Arc::new(SingleFlight::new()),
+ downloads: Arc::new(SingleFlight::new()),
+ model: Mutex::new(None),
+ }
+ }
+
+ /// Per-model state, for the settings panel.
+ pub fn status(dirs: &AppDirs) -> Result<TranscriberStatus, String> {
+ let dir = models_dir(dirs)?;
+ Ok(TranscriberStatus {
+ models_dir: dir.to_string_lossy().into_owned(),
+ models: KNOWN_MODELS
+ .iter()
+ .map(|model| ModelStatus {
+ id: model.id.to_owned(),
+ label: model.label.to_owned(),
+ blurb: model.blurb.to_owned(),
+ english_only: model.english_only,
+ size_bytes: model.approx_bytes,
+ downloaded: dir.join(format!("ggml-{}.bin", model.id)).is_file(),
+ })
+ .collect(),
+ })
+ }
+
+ /// Streams one model from Hugging Face into the models folder. Blocks
+ /// for the whole download: run it on its own thread.
+ ///
+ /// Written to `<file>.part` and renamed at the end: a download killed
+ /// halfway must never be mistaken for a model, because whisper would fail
+ /// on it with a message that blames the model rather than the download.
+ pub fn download_model(
+ &self,
+ dirs: &AppDirs,
+ id: &str,
+ mut progress: impl FnMut(DownloadProgress),
+ ) -> Result<(), String> {
+ let job = self.downloads.begin("model download")?;
+ let destination = model_file(dirs, id)?;
+ if destination.is_file() {
+ return Ok(());
+ }
+ let estimate = known(id).map(|model| model.approx_bytes).unwrap_or(0);
+ let partial = destination.with_extension("bin.part");
+ let (received, total) = crate::download_to(
+ &model_url(id),
+ &partial,
+ id,
+ estimate,
+ job.cancel_flag(),
+ &mut progress,
+ )?;
+ std::fs::rename(&partial, &destination)
+ .map_err(|error| format!("could not finish {}: {error}", destination.display()))?;
+ progress(DownloadProgress {
+ id: id.to_owned(),
+ received,
+ total,
+ unpacking: false,
+ done: true,
+ });
+ Ok(())
+ }
+
+ /// Asks the running download to stop. Idle is a harmless no-op.
+ pub fn cancel_download(&self) {
+ self.downloads.cancel();
+ }
+
+ /// Removes a downloaded model file, unless a transcription is using it.
+ pub fn delete_model(&self, dirs: &AppDirs, id: &str) -> Result<(), String> {
+ let file = model_file(dirs, id)?;
+ let mut loaded = self.model.try_lock().map_err(|_| {
+ "a transcription is running - wait for it or cancel it first".to_owned()
+ })?;
+ if loaded.as_ref().is_some_and(|model| model.id == id) {
+ *loaded = None;
+ }
+ std::fs::remove_file(&file)
+ .map_err(|error| format!("could not remove {}: {error}", file.display()))
+ }
+
+ /// Transcribes one clip's audio window into caption segments. Blocks
+ /// for the whole run: run it on its own thread. `progress` is called
+ /// with a 0..100 percentage as whisper works.
+ ///
+ /// Timestamps come back relative to the window: 0 is `source_start`. The
+ /// caller owns the clip, so the map onto the timeline (divide by speed,
+ /// add the clip's start) happens there, next to the numbers it needs.
+ pub fn transcribe(
+ &self,
+ dirs: &AppDirs,
+ request: &TranscribeRequest,
+ progress: impl FnMut(i32) + Send + 'static,
+ ) -> Result<Vec<Segment>, String> {
+ let job = self.gate.begin("transcription")?;
+ let cancel = job.cancel_handle();
+
+ let model_path = model_file(dirs, &request.model_id)?;
+ if !model_path.is_file() {
+ return Err(format!(
+ "model {} is not downloaded - see Settings > Transcriber",
+ request.model_id
+ ));
+ }
+ // NaN is refused along with zero and negative windows.
+ if request.window.is_nan() || request.window <= 0.0 {
+ return Err("nothing to transcribe: the clip covers no time".to_owned());
+ }
+
+ // The window, as whisper wants it: 16 kHz mono floats.
+ let mut decoder = AudioDecoder::open(
+ &request.path,
+ &AudioOptions {
+ start: Some(request.source_start),
+ duration: Some(request.window),
+ filters: Vec::new(),
+ rate: 16_000,
+ channels: 1,
+ format: SampleFormat::F32,
+ },
+ )
+ .map_err(|error| error.to_string())?;
+ let samples = decoder.collect_f32().map_err(|error| error.to_string())?;
+ if samples.is_empty() {
+ return Err("nothing to transcribe: the clip has no audio".to_owned());
+ }
+ if cancel.load(Ordering::Relaxed) {
+ return Err("transcription cancelled".to_owned());
+ }
+
+ let mut loaded = self
+ .model
+ .lock()
+ .map_err(|_| "transcriber state poisoned".to_owned())?;
+ if loaded.as_ref().map(|model| model.id.as_str()) != Some(request.model_id.as_str()) {
+ // Load before overwriting: a failed load keeps the old model.
+ let context =
+ WhisperContext::new_with_params(&model_path, WhisperContextParameters::default())
+ .map_err(|error| format!("could not load the model: {error}"))?;
+ *loaded = Some(LoadedModel {
+ id: request.model_id.clone(),
+ context,
+ });
+ }
+ let context = &loaded.as_ref().expect("model loaded above").context;
+ let mut state = context
+ .create_state()
+ .map_err(|error| format!("could not start whisper: {error}"))?;
+
+ let threads = std::thread::available_parallelism()
+ .map(|count| count.get().min(8))
+ .unwrap_or(4);
+ let mut params = FullParams::new(SamplingStrategy::Greedy { best_of: 1 });
+ params.set_n_threads(threads as i32);
+ params.set_language(Some(if request.language.is_empty() {
+ "auto"
+ } else {
+ &request.language
+ }));
+ params.set_print_special(false);
+ params.set_print_progress(false);
+ params.set_print_realtime(false);
+ params.set_print_timestamps(false);
+ params.set_suppress_blank(true);
+ let abort = Arc::clone(&cancel);
+ params.set_abort_callback_safe(move || abort.load(Ordering::Relaxed));
+ params.set_progress_callback_safe(progress);
+
+ state
+ .full(params, &samples)
+ .map_err(|error| format!("transcription failed: {error}"))?;
+ if cancel.load(Ordering::Relaxed) {
+ return Err("transcription cancelled".to_owned());
+ }
+
+ Ok(state
+ .as_iter()
+ .filter_map(|segment| {
+ // Whisper's timestamps are centiseconds.
+ let start = segment.start_timestamp() as f64 / 100.0;
+ let end = segment.end_timestamp() as f64 / 100.0;
+ let text = segment.to_str_lossy().ok()?.trim().to_owned();
+ keep_segment(start, end, text)
+ })
+ .collect())
+ }
+
+ /// Asks the running transcription to stop at its next check.
+ pub fn cancel(&self) {
+ self.gate.cancel();
+ }
+
+ /// Whether a transcription is running.
+ pub fn is_busy(&self) -> bool {
+ self.gate.is_busy()
+ }
+}
+
+/// Whisper marks silence and music with bracketed stage directions
+/// ("[BLANK_AUDIO]", "(upbeat music)"). Those are not captions, and neither
+/// is an empty or degenerate span.
+fn keep_segment(start: f64, end: f64, text: String) -> Option<Segment> {
+ if text.is_empty() || (text.starts_with('[') && text.ends_with(']')) {
+ return None;
+ }
+ (end > start).then_some(Segment { start, end, text })
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn stage_directions_and_degenerate_spans_are_not_captions() {
+ assert_eq!(
+ keep_segment(0.0, 2.5, " Hello there.".trim().to_owned()).map(|s| s.text),
+ Some("Hello there.".to_owned())
+ );
+ assert!(keep_segment(2.5, 2.5, "degenerate".to_owned()).is_none());
+ assert!(keep_segment(3.0, 4.0, "[BLANK_AUDIO]".to_owned()).is_none());
+ assert!(keep_segment(3.0, 4.0, String::new()).is_none());
+ }
+
+ #[test]
+ fn every_known_model_has_a_url_and_file_name() {
+ for model in KNOWN_MODELS {
+ assert!(model_url(model.id).ends_with(&format!("ggml-{}.bin", model.id)));
+ assert!(model.approx_bytes > 0);
+ }
+ }
+
+ #[test]
+ fn status_lists_every_model_undownloaded_in_an_empty_data_dir() {
+ let scratch =
+ std::env::temp_dir().join(format!("concat-whisper-test-{}", std::process::id()));
+ let _ = std::fs::remove_dir_all(&scratch);
+ let status = Transcriber::status(&AppDirs::under(&scratch)).expect("status");
+ assert_eq!(status.models.len(), KNOWN_MODELS.len());
+ assert!(status.models.iter().all(|model| !model.downloaded));
+ let _ = std::fs::remove_dir_all(&scratch);
+ }
+}
diff --git a/engine/crates/concat-speech/src/tts.rs b/engine/crates/concat-speech/src/tts.rs
new file mode 100644
index 00000000..208eb541
--- /dev/null
+++ b/engine/crates/concat-speech/src/tts.rs
@@ -0,0 +1,608 @@
+// SPDX-License-Identifier: AGPL-3.0-or-later
+// SPDX-FileCopyrightText: 2026 Jareer and Concat contributors
+
+//! Kokoro text-to-speech: text in, a narration WAV in the project folder out.
+//!
+//! The same local-first shape as transcription, with one difference:
+//! whisper runs as a child process, Kokoro runs in-process through
+//! sherpa-onnx (the official k2-fsa bindings, statically linked).
+//! In-process because there is no `kokoro-cli` worth shipping, and because
+//! sherpa's progress callback gives us cancellation anyway - returning
+//! `false` from it stops synthesis mid-sentence.
+//!
+//! Three pieces:
+//!
+//! 1. **Models.** Kokoro ships as a tar.bz2 bundle (the network, the voice
+//! bank, espeak-ng data, lexicons), downloaded on demand from the
+//! sherpa-onnx releases into `<app data>/tts-models/<id>/`. Streamed to a
+//! `.part`, unpacked through a staging folder and renamed - a torn
+//! download or unpack must never look usable.
+//! 2. **Voices.** Kokoro is one model with many built-in speakers, addressed
+//! by integer id. The table below names the ones we offer - the English
+//! and Chinese speakers, the languages this model's lexicons actually
+//! cover.
+//! 3. **Synthesis.** The engine loads once and is cached until the model
+//! changes or is deleted; each request writes a WAV into the project's
+//! `audio/` folder (not `cache/` - cache is regenerable, narration the
+//! user placed on the timeline is not) and returns its path for the
+//! caller to import like any other media file.
+
+use std::path::{Path, PathBuf};
+use std::sync::atomic::Ordering;
+use std::sync::{Arc, Mutex};
+
+use concat_host::{AppDirs, SingleFlight, projects};
+use serde::{Deserialize, Serialize};
+use sherpa_onnx::{
+ GenerationConfig, OfflineTts, OfflineTtsConfig, OfflineTtsKokoroModelConfig,
+ OfflineTtsModelConfig,
+};
+
+use crate::DownloadProgress;
+
+/// One downloadable Kokoro bundle. Sizes are approximate, for progress bars
+/// when the server declines to send a Content-Length.
+struct KnownModel {
+ id: &'static str,
+ label: &'static str,
+ /// One line for the settings row: what this size trades away.
+ blurb: &'static str,
+ approx_bytes: u64,
+}
+
+/// The models the settings panel offers, smallest first.
+///
+/// Both are Kokoro v1.0 multi-language (English + Chinese) from the official
+/// sherpa-onnx conversions; they differ only in quantisation. The int8 build
+/// is the default recommendation - a third of the download for a difference
+/// most ears never find.
+const KNOWN_MODELS: &[KnownModel] = &[
+ KnownModel {
+ id: "kokoro-int8-multi-lang-v1_0",
+ label: "Kokoro (compact)",
+ blurb: "The recommended build: same voices, a third of the download.",
+ approx_bytes: 131_839_838,
+ },
+ KnownModel {
+ id: "kokoro-multi-lang-v1_0",
+ label: "Kokoro (full precision)",
+ blurb: "Bit-perfect weights for the skeptical; rarely audibly better.",
+ approx_bytes: 349_418_188,
+ },
+];
+
+/// The speakers we offer, by Kokoro v1.0 speaker id.
+///
+/// The model bundles 53 voices across nine languages, but its lexicons (and
+/// sherpa's text frontend) only genuinely cover English and Chinese, so only
+/// those are listed. The name encodes accent and gender - `af` American
+/// female, `bm` British male, `zf` Chinese female - which the caller decodes
+/// for display rather than us shipping 36 label strings.
+const VOICES: &[(i32, &str)] = &[
+ (0, "af_alloy"),
+ (1, "af_aoede"),
+ (2, "af_bella"),
+ (3, "af_heart"),
+ (4, "af_jessica"),
+ (5, "af_kore"),
+ (6, "af_nicole"),
+ (7, "af_nova"),
+ (8, "af_river"),
+ (9, "af_sarah"),
+ (10, "af_sky"),
+ (11, "am_adam"),
+ (12, "am_echo"),
+ (13, "am_eric"),
+ (14, "am_fenrir"),
+ (15, "am_liam"),
+ (16, "am_michael"),
+ (17, "am_onyx"),
+ (18, "am_puck"),
+ (19, "am_santa"),
+ (20, "bf_alice"),
+ (21, "bf_emma"),
+ (22, "bf_isabella"),
+ (23, "bf_lily"),
+ (24, "bm_daniel"),
+ (25, "bm_fable"),
+ (26, "bm_george"),
+ (27, "bm_lewis"),
+ (45, "zf_xiaobei"),
+ (46, "zf_xiaoni"),
+ (47, "zf_xiaoxiao"),
+ (48, "zf_xiaoyi"),
+ (49, "zm_yunjian"),
+ (50, "zm_yunxi"),
+ (51, "zm_yunxia"),
+ (52, "zm_yunyang"),
+];
+
+fn known(id: &str) -> Option<&'static KnownModel> {
+ KNOWN_MODELS.iter().find(|model| model.id == id)
+}
+
+fn model_url(id: &str) -> String {
+ format!("https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/{id}.tar.bz2")
+}
+
+/// Where downloaded models live: `<app data>/tts-models/<id>/`.
+fn models_dir(dirs: &AppDirs) -> Result<PathBuf, String> {
+ let dir = dirs.data.join("tts-models");
+ std::fs::create_dir_all(&dir)
+ .map_err(|error| format!("could not create {}: {error}", dir.display()))?;
+ Ok(dir)
+}
+
+fn model_dir(dirs: &AppDirs, id: &str) -> Result<PathBuf, String> {
+ // Ids come from our own table; anything else is a bug, not input.
+ known(id).ok_or_else(|| format!("unknown model {id:?}"))?;
+ Ok(models_dir(dirs)?.join(id))
+}
+
+/// The `.onnx` network inside an unpacked bundle. Located by scanning rather
+/// than by name because the archives disagree - `model.onnx` in the full
+/// build, `model.int8.onnx` in the quantised one.
+fn onnx_file(dir: &Path) -> Option<PathBuf> {
+ std::fs::read_dir(dir).ok()?.flatten().find_map(|entry| {
+ let path = entry.path();
+ (path.extension().is_some_and(|ext| ext == "onnx")).then_some(path)
+ })
+}
+
+/// A bundle counts as downloaded once its network is in place. The rename at
+/// the end of unpacking makes this atomic: no folder, or a complete one.
+fn model_downloaded(dirs: &AppDirs, id: &str) -> bool {
+ model_dir(dirs, id).is_ok_and(|dir| onnx_file(&dir).is_some())
+}
+
+/// One model, as the settings panel shows it.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct ModelStatus {
+ /// The bundle id.
+ pub id: String,
+ /// Display name.
+ pub label: String,
+ /// One line on what this size trades away.
+ pub blurb: String,
+ /// Approximate download size in bytes, for display.
+ pub size_bytes: u64,
+ /// Whether the bundle is unpacked on disk.
+ pub downloaded: bool,
+}
+
+/// One speaker.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct VoiceInfo {
+ /// Kokoro speaker id, what synthesis wants back.
+ pub id: i32,
+ /// The upstream voice name, e.g. "af_heart"; the caller decodes the
+ /// accent/gender prefix and title-cases the rest.
+ pub name: String,
+}
+
+/// What the settings panel and the speech dialog show.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct TtsStatus {
+ /// Where models are stored, so the settings panel can say so.
+ pub models_dir: String,
+ /// Every model the panel offers.
+ pub models: Vec<ModelStatus>,
+ /// Every speaker on offer.
+ pub voices: Vec<VoiceInfo>,
+}
+
+/// What to say, and where the file goes.
+#[derive(Deserialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct SpeakRequest {
+ /// Which model to use, e.g. "kokoro-int8-multi-lang-v1_0".
+ pub model_id: String,
+ /// Kokoro speaker id, from the voices table.
+ pub voice: i32,
+ /// What to say.
+ pub text: String,
+ /// Speaking rate; 1.0 is the voice's natural pace.
+ pub speed: f32,
+ /// The project folder the WAV should land in.
+ pub project: String,
+}
+
+/// What synthesis produced.
+#[derive(Serialize, Clone, Debug)]
+#[serde(rename_all = "camelCase")]
+pub struct SpeakResult {
+ /// Absolute path of the written WAV, ready for the media import path.
+ pub path: String,
+ /// Seconds of audio, so the caller can say something useful.
+ pub duration: f64,
+}
+
+struct CachedEngine {
+ model_id: String,
+ tts: OfflineTts,
+}
+
+/// The one synthesis and the one download that can run at a time, plus the
+/// loaded engine.
+///
+/// The engine is cached because loading means parsing a 100-300 MB network;
+/// doing that per sentence would make the feature feel broken. The mutex is
+/// held for the whole synthesis, which is what lets [`Speech::delete_model`]
+/// use `try_lock` to refuse deleting a model mid-use instead of yanking
+/// mapped files out from under the session.
+pub struct Speech {
+ gate: Arc<SingleFlight>,
+ downloads: Arc<SingleFlight>,
+ engine: Mutex<Option<CachedEngine>>,
+}
+
+impl Default for Speech {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl Speech {
+ /// An idle speech engine, loading nothing until asked to speak.
+ pub fn new() -> Self {
+ Self {
+ gate: Arc::new(SingleFlight::new()),
+ downloads: Arc::new(SingleFlight::new()),
+ engine: Mutex::new(None),
+ }
+ }
+
+ /// Models and voices, for the settings panel and the speech dialog.
+ pub fn status(dirs: &AppDirs) -> Result<TtsStatus, String> {
+ let dir = models_dir(dirs)?;
+ Ok(TtsStatus {
+ models_dir: dir.to_string_lossy().into_owned(),
+ models: KNOWN_MODELS
+ .iter()
+ .map(|model| ModelStatus {
+ id: model.id.to_owned(),
+ label: model.label.to_owned(),
+ blurb: model.blurb.to_owned(),
+ size_bytes: model.approx_bytes,
+ downloaded: model_downloaded(dirs, model.id),
+ })
+ .collect(),
+ voices: VOICES
+ .iter()
+ .map(|(id, name)| VoiceInfo {
+ id: *id,
+ name: (*name).to_owned(),
+ })
+ .collect(),
+ })
+ }
+
+ /// Streams one bundle from the sherpa-onnx releases and unpacks it.
+ /// Blocks for the whole download: run it on its own thread.
+ ///
+ /// The archive lands in a `.part`, unpacks into a `.staging-<id>` folder,
+ /// and only the final rename puts `<id>/` in place - killed at any
+ /// earlier point, nothing is left that could be mistaken for a model.
+ pub fn download_model(
+ &self,
+ dirs: &AppDirs,
+ id: &str,
+ mut progress: impl FnMut(DownloadProgress),
+ ) -> Result<(), String> {
+ let job = self.downloads.begin("voice model download")?;
+ let cancel = job.cancel_flag();
+ let destination = model_dir(dirs, id)?;
+ if onnx_file(&destination).is_some() {
+ return Ok(());
+ }
+ let parent = models_dir(dirs)?;
+ let estimate = known(id).map(|model| model.approx_bytes).unwrap_or(0);
+
+ let archive = parent.join(format!("{id}.tar.bz2.part"));
+ let staging = parent.join(format!(".staging-{id}"));
+ let result = (|| {
+ let (received, total) = crate::download_to(
+ &model_url(id),
+ &archive,
+ id,
+ estimate,
+ cancel,
+ &mut progress,
+ )?;
+
+ // Unpacking a 130 MB bz2 takes long enough to deserve its own
+ // phase on the progress bar.
+ progress(DownloadProgress {
+ id: id.to_owned(),
+ received,
+ total,
+ unpacking: true,
+ done: false,
+ });
+
+ let _ = std::fs::remove_dir_all(&staging);
+ std::fs::create_dir_all(&staging)
+ .map_err(|error| format!("could not create {}: {error}", staging.display()))?;
+
+ let file = std::fs::File::open(&archive)
+ .map_err(|error| format!("could not reopen the download: {error}"))?;
+ let tar = bzip2::read::BzDecoder::new(std::io::BufReader::new(file));
+ let mut entries = tar::Archive::new(tar);
+ for entry in entries
+ .entries()
+ .map_err(|error| format!("could not read the archive: {error}"))?
+ {
+ if cancel.load(Ordering::Relaxed) {
+ return Err("download cancelled".to_owned());
+ }
+ // `unpack_in` refuses paths that escape the staging folder,
+ // so a hostile archive can drop files nowhere else.
+ entry
+ .and_then(|mut entry| entry.unpack_in(&staging))
+ .map_err(|error| format!("could not unpack the archive: {error}"))?;
+ }
+
+ let unpacked = staging.join(id);
+ if onnx_file(&unpacked).is_none() {
+ return Err("the archive did not contain the expected model".to_owned());
+ }
+ // A leftover folder from an older torn unpack would fail the
+ // rename; it holds no model (checked above), so it goes.
+ let _ = std::fs::remove_dir_all(&destination);
+ std::fs::rename(&unpacked, &destination)
+ .map_err(|error| format!("could not finish {}: {error}", destination.display()))?;
+
+ progress(DownloadProgress {
+ id: id.to_owned(),
+ received,
+ total,
+ unpacking: false,
+ done: true,
+ });
+ Ok(())
+ })();
+
+ // The archive and staging folder are dead weight whether the unpack
+ // finished or failed; only `<id>/` matters now.
+ let _ = std::fs::remove_file(&archive);
+ let _ = std::fs::remove_dir_all(&staging);
+ result
+ }
+
+ /// Asks the running download to stop. Idle is a harmless no-op.
+ pub fn cancel_download(&self) {
+ self.downloads.cancel();
+ }
+
+ /// Removes a downloaded model folder, unless the engine is speaking from it.
+ pub fn delete_model(&self, dirs: &AppDirs, id: &str) -> Result<(), String> {
+ let dir = model_dir(dirs, id)?;
+ let mut engine = self
+ .engine
+ .try_lock()
+ .map_err(|_| "speech is being generated - wait for it or cancel it first".to_owned())?;
+ if engine.as_ref().is_some_and(|cached| cached.model_id == id) {
+ *engine = None;
+ }
+ std::fs::remove_dir_all(&dir)
+ .map_err(|error| format!("could not remove {}: {error}", dir.display()))
+ }
+
+ /// Synthesizes one narration clip into `<project>/audio/`. Blocks for
+ /// the whole synthesis: run it on its own thread. `progress` is called
+ /// with a 0..1 fraction as sentences complete.
+ pub fn speak(
+ &self,
+ dirs: &AppDirs,
+ request: &SpeakRequest,
+ progress: impl FnMut(f32) + Send + 'static,
+ ) -> Result<SpeakResult, String> {
+ let job = self.gate.begin("speech generation")?;
+ let cancel = job.cancel_handle();
+
+ let text = request.text.trim();
+ if text.is_empty() {
+ return Err("nothing to say: the text is empty".to_owned());
+ }
+ if !VOICES.iter().any(|(id, _)| *id == request.voice) {
+ return Err(format!("unknown voice {}", request.voice));
+ }
+ let speed = if request.speed.is_finite() {
+ request.speed.clamp(0.5, 2.0)
+ } else {
+ 1.0
+ };
+
+ let dir = model_dir(dirs, &request.model_id)?;
+ if onnx_file(&dir).is_none() {
+ return Err(format!(
+ "model {} is not downloaded - see Settings > Speech",
+ request.model_id
+ ));
+ }
+
+ // The WAV goes in the project so it travels (and dies) with it - and
+ // in `audio/`, not `cache/`, because a clip on the timeline points at
+ // it: cache is for things that can be regenerated.
+ let root = Path::new(&request.project);
+ if !projects::is_project(root) {
+ return Err(format!("{} is not a project folder", request.project));
+ }
+ let out_dir = root.join("audio");
+ std::fs::create_dir_all(&out_dir)
+ .map_err(|error| format!("could not create {}: {error}", out_dir.display()))?;
+
+ let mut engine = self
+ .engine
+ .lock()
+ .map_err(|_| "speech state poisoned".to_owned())?;
+ if engine.as_ref().map(|cached| cached.model_id.as_str()) != Some(request.model_id.as_str())
+ {
+ // Load before overwriting: a failed load keeps the old engine.
+ let tts = load_engine(&dir)?;
+ *engine = Some(CachedEngine {
+ model_id: request.model_id.clone(),
+ tts,
+ });
+ }
+ let tts = &engine.as_ref().expect("engine cached above").tts;
+
+ let progress_cancel = Arc::clone(&cancel);
+ let mut progress = progress;
+ let mut last_fraction = -1.0f32;
+ let audio = tts
+ .generate_with_config(
+ text,
+ &GenerationConfig {
+ sid: request.voice,
+ speed,
+ ..Default::default()
+ },
+ Some(move |_samples: &[f32], fraction: f32| -> bool {
+ if progress_cancel.load(Ordering::Relaxed) {
+ return false;
+ }
+ // The engine reports once per sentence; only meaningful
+ // movement is worth a redraw.
+ if fraction - last_fraction >= 0.01 {
+ last_fraction = fraction;
+ progress(fraction);
+ }
+ true
+ }),
+ )
+ .ok_or_else(|| "speech generation failed".to_owned())?;
+
+ // A cancelled run still returns the samples made so far; the user
+ // asked for none of them.
+ if cancel.load(Ordering::Relaxed) {
+ return Err("speech generation cancelled".to_owned());
+ }
+ if audio.samples().is_empty() {
+ return Err("the engine produced no audio for this text".to_owned());
+ }
+
+ // Wall-clock millis plus process id: unique enough for files created
+ // by one human clicking a button, and stable for the media bin name.
+ let stamp = std::time::SystemTime::now()
+ .duration_since(std::time::UNIX_EPOCH)
+ .map(|elapsed| elapsed.as_millis())
+ .unwrap_or(0);
+ let voice_name = VOICES
+ .iter()
+ .find(|(id, _)| *id == request.voice)
+ .map(|(_, name)| *name)
+ .unwrap_or("voice");
+ let file = out_dir.join(format!("speech-{voice_name}-{stamp}.wav"));
+
+ if !audio.save(&file.to_string_lossy()) {
+ return Err(format!("could not write {}", file.display()));
+ }
+
+ let duration = audio.samples().len() as f64 / f64::from(audio.sample_rate().max(1));
+ Ok(SpeakResult {
+ path: file.to_string_lossy().into_owned(),
+ duration,
+ })
+ }
+
+ /// Asks the running synthesis to stop at the next sentence boundary.
+ pub fn cancel(&self) {
+ self.gate.cancel();
+ }
+
+ /// Whether a synthesis is running.
+ pub fn is_busy(&self) -> bool {
+ self.gate.is_busy()
+ }
+}
+
+/// Builds the engine for one model folder.
+///
+/// The lexicon list mirrors the official sherpa-onnx invocation for this
+/// bundle: US English and Chinese, joined by commas, each only if present.
+fn load_engine(dir: &Path) -> Result<OfflineTts, String> {
+ let network = onnx_file(dir)
+ .ok_or_else(|| "the model folder has no .onnx network - re-download it".to_owned())?;
+ let existing = |name: &str| {
+ let path = dir.join(name);
+ path.is_file().then(|| path.to_string_lossy().into_owned())
+ };
+ let lexicon: Vec<String> = ["lexicon-us-en.txt", "lexicon-zh.txt"]
+ .iter()
+ .filter_map(|name| existing(name))
+ .collect();
+
+ let threads = std::thread::available_parallelism()
+ .map(|count| count.get().min(8))
+ .unwrap_or(4) as i32;
+
+ let config = OfflineTtsConfig {
+ model: OfflineTtsModelConfig {
+ kokoro: OfflineTtsKokoroModelConfig {
+ model: Some(network.to_string_lossy().into_owned()),
+ voices: existing("voices.bin"),
+ tokens: existing("tokens.txt"),
+ data_dir: {
+ let data = dir.join("espeak-ng-data");
+ data.is_dir().then(|| data.to_string_lossy().into_owned())
+ },
+ dict_dir: {
+ let dict = dir.join("dict");
+ dict.is_dir().then(|| dict.to_string_lossy().into_owned())
+ },
+ lexicon: (!lexicon.is_empty()).then(|| lexicon.join(",")),
+ ..Default::default()
+ },
+ num_threads: threads,
+ ..Default::default()
+ },
+ ..Default::default()
+ };
+
+ OfflineTts::create(&config)
+ .ok_or_else(|| "the speech engine failed to load - try re-downloading the model".to_owned())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn every_known_model_has_a_url_and_size() {
+ for model in KNOWN_MODELS {
+ assert!(model_url(model.id).ends_with(&format!("{}.tar.bz2", model.id)));
+ assert!(model.approx_bytes > 0);
+ }
+ }
+
+ #[test]
+ fn voice_ids_are_unique_and_names_well_formed() {
+ let mut seen = std::collections::HashSet::new();
+ for (id, name) in VOICES {
+ assert!(seen.insert(*id), "duplicate voice id {id}");
+ // The caller decodes accent and gender from the prefix; a name
+ // that breaks the pattern would render as gibberish.
+ let (prefix, rest) = name.split_once('_').expect("prefix_name shape");
+ assert!(
+ matches!(prefix, "af" | "am" | "bf" | "bm" | "zf" | "zm"),
+ "{name}"
+ );
+ assert!(!rest.is_empty());
+ }
+ }
+
+ #[test]
+ fn finds_the_onnx_network_by_extension() {
+ let scratch = std::env::temp_dir().join(format!("concat-tts-test-{}", std::process::id()));
+ let _ = std::fs::remove_dir_all(&scratch);
+ std::fs::create_dir_all(&scratch).expect("scratch dir");
+ assert!(onnx_file(&scratch).is_none());
+ std::fs::write(scratch.join("model.int8.onnx"), b"x").expect("writes");
+ assert!(onnx_file(&scratch).is_some());
+ let _ = std::fs::remove_dir_all(&scratch);
+ }
+}
diff --git a/engine/crates/concat/Cargo.toml b/engine/crates/concat/Cargo.toml
index 0da2da27..244ecd8b 100644
--- a/engine/crates/concat/Cargo.toml
+++ b/engine/crates/concat/Cargo.toml
@@ -38,6 +38,10 @@ slint = { version = "1.17", default-features = false, features = [
"unstable-winit-030",
] }
+# The engine. What Settings > About reports about it comes from here; the
+# rest of the wiring is the work in progress.
+concat-media.workspace = true
+
# The renderer is picked by a feature here rather than by editing the
# dependency, so both variants build from one tree and can be compared:
#
diff --git a/engine/crates/concat/src/main.rs b/engine/crates/concat/src/main.rs
index bff11353..7e713f7d 100644
--- a/engine/crates/concat/src/main.rs
+++ b/engine/crates/concat/src/main.rs
@@ -1980,7 +1980,7 @@ impl Studio {
format!("{count} installed · {on_disk:.0} MB on disk").into()
},
version: "0.1.0".into(),
- engine: "concat-engine · ffmpeg 7.1".into(),
+ engine: format!("concat-engine · FFmpeg {}", concat_media::linked_version()).into(),
});
sync(&models.transcribers, Studio::model_rows(&self.transcribers));
sync(&models.voices, Studio::model_rows(&self.voices));
@@ -2922,7 +2922,10 @@ fn system_facts() -> Vec<(&'static str, String)> {
}
.into(),
),
- ("Engine", "concat-engine · ffmpeg 7.1".into()),
+ (
+ "Engine",
+ format!("concat-engine · FFmpeg {}", concat_media::linked_version()),
+ ),
("Operating system", os_description()),
(
"Processor",
diff --git a/flake.nix b/flake.nix
index 9bb1e02b..ccceefb5 100644
--- a/flake.nix
+++ b/flake.nix
@@ -1,5 +1,10 @@
{
- # Nix support (#7). Linux only on purpose: Tauri under Nix on macOS means
+ # Nix support (#7). Packages the deprecated Tauri app in desktop/, which no
+ # longer builds against the engine (the engine links FFmpeg now and the
+ # host's Rust moved into engine/crates). To be repointed at the Slint
+ # window, engine/crates/concat, once that is packaged.
+ #
+ # Linux only on purpose: Tauri under Nix on macOS means
# Apple SDK juggling, and the people asking for a flake run Linux.
#
# nix build .#concat the app, with ffmpeg and whisper-cli wired in