Skip to content

armgit / PS5LM

Public Imported from GitHub
0

llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8

ci: validate the existing PS5 app with a pinned public native build (#36)

Nicholas Tindleecbcb8cc54fa
diff --git a/.github/workflows/native-build.yml b/.github/workflows/native-build.yml
new file mode 100644
index 0000000..45a7c93
--- /dev/null
+++ b/.github/workflows/native-build.yml
@@ -0,0 +1,75 @@
+name: Native app validation (not PS5 hardware)
+on:
+ pull_request:
+ push:
+ branches: [main, 'ci/**']
+permissions:
+ contents: read
+concurrency:
+ group: native-app-${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: true
+jobs:
+ host-checks:
+ name: Existing model planner and script syntax
+ runs-on: ubuntu-24.04
+ timeout-minutes: 5
+ steps:
+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
+ with:
+ persist-credentials: false
+ - name: Shell and Python syntax (no submodules needed)
+ run: |
+ for script in scripts/*.sh ps5/app/ps5cc ps5/app/ps5cxx; do bash -n "$script"; done
+ PYTHONPYCACHEPREFIX=build/pycache python3 -m py_compile scripts/*.py
+ - name: Build-script job limit and compiler-failure regressions
+ run: PYTHONPYCACHEPREFIX=build/pycache python3 -m unittest discover -s tests -p 'test_build_app.py' -v
+ - name: Existing synthetic GGUF model-planner regression test
+ run: |
+ mkdir -p build
+ c++ -std=c++17 -O1 ps5/app/model_plan_test.cpp ps5/app/model_plan.cpp -o build/model_plan_test
+ build/model_plan_test
+
+ ps5-title:
+ name: Full PS5 title cross-build and static package checks
+ runs-on: ubuntu-24.04
+ timeout-minutes: 45
+ container:
+ # Official Debian image; trixie supplies compatible LLVM19/SPIRV host tools.
+ image: debian:trixie-20261020-slim@sha256:62bd423aeee912639ca8ee681c2b762cb3e25137e0a92b20be455898a9f034ad
+ options: --cpus 2
+ defaults:
+ run:
+ shell: bash
+ env:
+ DEBIAN_FRONTEND: noninteractive
+ BUILD_JOBS: '2'
+ LLVM_CONFIG: /usr/bin/llvm-config-19
+ steps:
+ - name: Public cross-build prerequisites
+ run: |
+ apt-get update
+ apt-get install -y --no-install-recommends \
+ build-essential git ca-certificates wget curl unzip rsync cmake ninja-build \
+ clang-19 llvm-19-dev lld-19 libclang-rt-19-dev libclang-19-dev libclang-cpp19-dev \
+ libllvmspirvlib-19-dev libclc-19-dev spirv-tools spirv-headers glslang-tools \
+ meson pkgconf flex bison python3 python3-mako python3-yaml python3-packaging python3-setuptools \
+ zlib1g-dev libzstd-dev libffi-dev libedit-dev libxml2-dev libvulkan-dev glslc ffmpeg xxd
+ echo /usr/lib/llvm-19/bin >> "$GITHUB_PATH"
+ - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
+ with:
+ persist-credentials: false
+ - name: Pinned llama.cpp and UI sources
+ run: |
+ # Container root differs from the runner owner. Trust only this checkout,
+ # in the disposable job container; never use safe.directory='*'.
+ git config --global --add safe.directory "$GITHUB_WORKSPACE"
+ git submodule update --init --depth 1
+ - name: Build patched RADV and title, inspect ELF and FSELF
+ run: scripts/test-native-build.sh
+ - name: State the full-build validation boundary
+ if: always()
+ run: |
+ echo '## Native PS5 title cross-build' >> "$GITHUB_STEP_SUMMARY"
+ echo 'Builds the pinned public SDK/runtime, patched RADV, UI and existing app title; checks ELF/FSELF structure.' >> "$GITHUB_STEP_SUMMARY"
+ echo 'Build tools and checksum-verified public SDK/driver dependencies download; no model or proprietary firmware is needed.' >> "$GITHUB_STEP_SUMMARY"
+ echo 'This is static cross-build validation, not console boot, Vulkan execution, PS5 emulation or model inference.' >> "$GITHUB_STEP_SUMMARY"
diff --git a/docs/BUILDING.md b/docs/BUILDING.md
index a18ef33..bb4673f 100644
--- a/docs/BUILDING.md
+++ b/docs/BUILDING.md
@@ -27,6 +27,10 @@ works too.
## Build
+For the automated, pinned-source Debian build and its validation limits, see
+[CI validation](CI.md). `scripts/test-native-build.sh` builds and statically checks
+the existing app without a console, model download, or proprietary firmware.
+
On Arch, the host packages (or run them in an `archlinux` container with the repository mounted):
```sh
diff --git a/docs/CI.md b/docs/CI.md
new file mode 100644
index 0000000..3e12f55
--- /dev/null
+++ b/docs/CI.md
@@ -0,0 +1,84 @@
+# CI validation
+
+[`native-build.yml`](../.github/workflows/native-build.yml) checks the existing app
+on pull requests and pushes to `main` or `ci/**`. It uses ordinary Ubuntu hosted
+runners, a read-only token, no secrets, and no console connection. Superseded runs
+are cancelled. There are two independent jobs:
+
+- **Host checks** (5-minute limit): shell/Python syntax, the existing synthetic
+ GGUF `model_plan_test.cpp`, and regressions for dashboard compiler job limits
+ and propagation of failed background compilers. No submodule or model download
+ is needed for these checks.
+- **Full PS5 title cross-build** (45-minute limit, two compiler jobs): bootstrap
+ the public SDK, reproduce clean-room `libc.prx`, build patched RADV and the
+ existing llama-server/native dashboard, package the title folder, and inspect
+ its ELF/FSELF structure and integrity. It does not execute the resulting title.
+
+The second job downloads build tools, source dependencies and checksum-verified
+public SDK archives. It does **not** download a trained model or proprietary
+console firmware. Only the optional browser UI is omitted: its upstream fallback
+fetches a floating `latest` release. The native console dashboard is built.
+Normal `scripts/build-app.sh` builds keep the prebuilt browser UI enabled;
+`PS5LM_PREBUILT_UI=OFF` explicitly opts out and clears legacy CMake aliases.
+
+## Reproduce the checks
+
+Host checks need Bash, Python 3 and a C++17 compiler:
+
+```sh
+for script in scripts/*.sh ps5/app/ps5cc ps5/app/ps5cxx; do bash -n "$script"; done
+PYTHONPYCACHEPREFIX=build/pycache python3 -m py_compile scripts/*.py
+PYTHONPYCACHEPREFIX=build/pycache python3 -m unittest discover -s tests -p 'test_build_app.py' -v
+mkdir -p build
+c++ -std=c++17 -O1 ps5/app/model_plan_test.cpp ps5/app/model_plan.cpp -o build/model_plan_test
+build/model_plan_test
+```
+
+For the full build, use the digest-pinned Debian trixie container and exact apt
+package list in the workflow, or an x86-64 Linux host with equivalent tools:
+
+```sh
+git submodule update --init --depth 1
+export PATH=/usr/lib/llvm-19/bin:$PATH
+LLVM_CONFIG=/usr/bin/llvm-config-19 BUILD_JOBS=2 scripts/test-native-build.sh
+```
+
+The script keeps dependencies in `.deps/native-ci-src` (override with
+`PS5LM_NATIVE_SOURCES`) and validates existing dependency revisions before reuse.
+It uses upstream's checked SDK setup and `make libc`, which rebuilds the runtime
+twice and compares the recorded raw/signed hashes. That target also produces the
+same host packaging tool required by the app, without building an unrelated demo
+title. It applies this repository's three Mesa patches with fixed commit identity
+and dates. The removed OpenGL release is fetched from the mirror already
+documented in [BUILDING.md](BUILDING.md), checked against the original archive
+hash, then checked by the pinned UI fetcher's file manifest.
+
+| Input | Pin |
+|---|---|
+| Debian container | `trixie-20261020-slim@sha256:62bd423aeee912639ca8ee681c2b762cb3e25137e0a92b20be455898a9f034ad` |
+| PS5_Vulkan | `f410e67a0241b3e8edcdbcd2a9b285dfc4a412ed` |
+| PS5_PayloadSDK fork | `b83202be73e930050e00de0f1b4d9ec46c0391bf` |
+| Mesa base | `1ba9874f478657ccf6da72ce55f0260164cc0bd0` |
+| Mesa after this repository's patches | `9e0a7c63bc6d4b6bcb60831163bb3fe9f46e91ea` |
+| llama.cpp / UI | The checkout's gitlink revisions (verified before building) |
+| OpenGL SDK archive SHA-256 | `f93643c04c843d56143b00951df1f8042ea7706ae9f19e4e9abf158e1ead77c5` |
+
+`build/native-ci/result.json` records the source pins and SHA-256 hashes of the
+packaged files after successful inspections. The title is in
+`build/app-ps5/title/PPSA99575`. A failed check exits nonzero; there is no automatic
+deployment, release publication or binary execution. CI publishes logs, not a
+release package.
+
+## What a green result means
+
+It proves the checked source can build/link/package with these public tools and
+pass static ELF/FSELF checks, including the native UI and Vulkan driver build.
+It does **not** prove the title boots on a PS5, the GPU executes inference, models
+fit memory, or networking/performance works on hardware. A generic QEMU x86-64
+guest cannot supply the PS5 loader, system services or AGC GPU interface; it would
+not close that gap. Hardware smoke tests remain separate.
+
+Source pins and archive checksums make the build inputs explicit, but apt package
+updates and build paths/compiler versions can change title bytes. Only the
+upstream clean-room runtime has a fixed byte-reproduction check; the title hashes
+identify a particular build, not a promised reproducible release binary.
diff --git a/scripts/build-app.sh b/scripts/build-app.sh
index 495f410..70c66a7 100755
--- a/scripts/build-app.sh
+++ b/scripts/build-app.sh
@@ -9,6 +9,8 @@
# tools/build-radv.sh release and `make app` (for the host tool and libc.prx).
#
# scripts/build-app.sh configure, build and package
+# PS5LM_PREBUILT_UI=OFF ... omit the optional downloaded browser UI
+# CMAKE_BUILD_PARALLEL_LEVEL also bounds the native dashboard compilers.
set -euo pipefail
root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
@@ -76,7 +78,9 @@ cmake -S "$llama" -B "$out/llama" -G Ninja \
-DVulkan_GLSLC_EXECUTABLE="$(command -v glslc)" -DSPIRV-Headers_DIR="$headers/cmake" \
-DMATH_LIBRARY=SceLibcInternal \
-DLLAMA_OPENSSL=OFF -DLLAMA_BUILD_COMMON=ON -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF \
- -DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_SERVER=ON > "$out.configure.log" 2>&1 ||
+ -DLLAMA_BUILD_TOOLS=ON -DLLAMA_BUILD_SERVER=ON \
+ -ULLAMA_BUILD_WEBUI -ULLAMA_USE_PREBUILT_WEBUI \
+ -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI="${PS5LM_PREBUILT_UI:-ON}" > "$out.configure.log" 2>&1 ||
{ tail -30 "$out.configure.log" >&2; exit 1; }
cmake --build "$out/llama" --target llama-server-impl
@@ -134,6 +138,9 @@ mkdir -p "$obj/kit"
mapfile -t kit_sources < <(find "$kit/src/gfx" "$kit/src/ui" "$kit/src/core" "$kit/src/audio" "$kit/src/third_party" \
"$kit/src/platform/ps5" -type f \( -name '*.cpp' -o -name '*.c' \) ! -name display_egl.cpp | sort)
kit_objects=()
+kit_pending=()
+jobs_limit=${CMAKE_BUILD_PARALLEL_LEVEL:-$(nproc)}
+[[ $jobs_limit =~ ^[1-9][0-9]*$ ]] || { echo "build-app: invalid job limit" >&2; exit 2; }
for src in "${kit_sources[@]}"; do
o="$obj/kit/$(echo "${src#"$kit/src/"}" | tr '/' '_').o"
kit_objects+=("$o")
@@ -143,9 +150,14 @@ for src in "${kit_sources[@]}"; do
else
cxx "${kitflags[@]}" -w -c "$src" -o "$o" &
fi
- (( $(jobs -r | wc -l) >= $(nproc) )) && wait -n
+ # Retain each PID: bare wait can hide an already-finished compiler failure.
+ kit_pending+=("$!")
+ if (( ${#kit_pending[@]} >= jobs_limit )); then
+ wait "${kit_pending[0]}"
+ kit_pending=("${kit_pending[@]:1}")
+ fi
done
-wait
+for pid in "${kit_pending[@]}"; do wait "$pid"; done
rm -f "$obj/libps5lm_kit.a"
"$sdk/bin/llvm-ar" rcs "$obj/libps5lm_kit.a" "${kit_objects[@]}"
for src in display dashboard stats; do
diff --git a/scripts/test-native-build.sh b/scripts/test-native-build.sh
new file mode 100755
index 0000000..93a6335
--- /dev/null
+++ b/scripts/test-native-build.sh
@@ -0,0 +1,148 @@
+#!/usr/bin/env bash
+# Full native cross-build and static package validation, NOT console execution.
+set -euo pipefail
+root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$root"
+output="$root/build/native-ci"
+mkdir -p "$output/tools"
+# Clear stale success evidence even if a preflight check below fails.
+rm -f "$output/result.json"
+export PS5LM_NATIVE_SOURCES="${PS5LM_NATIVE_SOURCES:-$root/.deps/native-ci-src}"
+export BUILD_JOBS="${BUILD_JOBS:-2}"
+[[ $BUILD_JOBS =~ ^[1-9][0-9]*$ ]] || { echo 'BUILD_JOBS must be a positive integer' >&2; exit 2; }
+export CMAKE_BUILD_PARALLEL_LEVEL="$BUILD_JOBS"
+# SDK archive wrappers resolve this as a path, not a PATH command name.
+llvm_config=$(command -v "${LLVM_CONFIG:-llvm-config}") || {
+ echo "title-check: missing LLVM_CONFIG tool: ${LLVM_CONFIG:-llvm-config}" >&2; exit 2;
+}
+export LLVM_CONFIG="$llvm_config"
+for tool in git clang clang++ make wget unzip sha256sum timeout python3 cmake ninja meson rsync glslangValidator glslc pkg-config curl ffmpeg; do
+ command -v "$tool" >/dev/null || { echo "title-check: missing $tool" >&2; exit 2; }
+done
+ui="$root/third_party/ps5-homebrew-ui"
+ui_revision=$(git ls-tree HEAD third_party/ps5-homebrew-ui | awk '{print $3}')
+[[ -f "$ui/tools/fetch-opengl-sdk.sh" && $(git -C "$ui" rev-parse HEAD) == "$ui_revision" ]] || {
+ echo 'title-check: initialize the pinned UI submodule first' >&2; exit 2;
+}
+llama="$root/third_party/llama.cpp"
+llama_revision=$(git ls-tree HEAD third_party/llama.cpp | awk '{print $3}')
+[[ -f "$llama/CMakeLists.txt" && $(git -C "$llama" rev-parse HEAD) == "$llama_revision" ]] || {
+ echo 'title-check: initialize the pinned llama.cpp submodule first' >&2; exit 2;
+}
+vk="$PS5LM_NATIVE_SOURCES/PS5_Vulkan"
+sdk_fork="$PS5LM_NATIVE_SOURCES/PS5_PayloadSDK"
+vk_revision=f410e67a0241b3e8edcdbcd2a9b285dfc4a412ed
+sdk_revision=b83202be73e930050e00de0f1b4d9ec46c0391bf
+fetch_pinned() {
+ local url=$1 directory=$2 revision=$3
+ if [[ ! -d "$directory/.git" ]]; then
+ [[ ! -e "$directory" ]] || { echo "native-check: refusing existing non-repo $directory" >&2; exit 2; }
+ mkdir -p "$directory"
+ git -C "$directory" init -q
+ timeout 120 git -C "$directory" fetch --depth 1 "$url" "$revision"
+ git -C "$directory" checkout -q --detach FETCH_HEAD
+ fi
+ [[ $(git -C "$directory" rev-parse HEAD) == "$revision" ]] || {
+ echo "native-check: unexpected revision in $directory" >&2; exit 2;
+ }
+ git -C "$directory" diff --quiet HEAD -- || {
+ echo "native-check: tracked dependency source is modified: $directory" >&2; exit 2;
+ }
+}
+fetch_pinned https://github.com/mihawk-99/PS5_Vulkan.git "$vk" "$vk_revision"
+fetch_pinned https://github.com/mihawk-99/PS5_PayloadSDK.git "$sdk_fork" "$sdk_revision"
+
+# Upstream scripts verify the v0.42 SDK release and zlib archive SHA-256s.
+# These are public homebrew sources/tools, not proprietary firmware dumps.
+export PS5_PAYLOAD_SDK_FORK="$sdk_fork"
+timeout 300 bash "$vk/tools/setup-native-dependencies.sh"
+export PS5_VULKAN_ROOT="$vk"
+export PS5_PAYLOAD_SDK="$vk/.deps/native/ps5-payload-sdk"
+[[ $(cat "$PS5_PAYLOAD_SDK/.ps5-sdk-revision") == "$sdk_revision" ]]
+# Rebuild twice and verify upstream's recorded clean-room runtime checksums.
+# This also builds the host packaging tool without linking an unrelated demo.
+timeout 180 make -C "$vk" libc
+
+mesa="$PS5LM_NATIVE_SOURCES/PS5_Mesa"
+mesa_base=1ba9874f478657ccf6da72ce55f0260164cc0bd0
+mesa_patched=9e0a7c63bc6d4b6bcb60831163bb3fe9f46e91ea
+if [[ ! -d "$mesa/.git" ]]; then
+ [[ ! -e "$mesa" ]] || { echo "title-check: refusing existing non-repo $mesa" >&2; exit 2; }
+ mkdir -p "$mesa"
+ git -C "$mesa" init -q
+ timeout 180 git -C "$mesa" fetch --depth 1 https://github.com/mihawk-99/PS5_Mesa.git "$mesa_base"
+ git -C "$mesa" checkout -q --detach FETCH_HEAD
+fi
+git -C "$mesa" diff --quiet HEAD -- || { echo 'title-check: modified Mesa sources' >&2; exit 2; }
+if [[ $(git -C "$mesa" rev-parse HEAD) == "$mesa_base" ]]; then
+ git -C "$mesa" -c user.name='PS5LM CI' -c user.email='[email protected]' \
+ -c commit.gpgsign=false am --committer-date-is-author-date "$root"/patches/mesa/*.patch
+fi
+[[ $(git -C "$mesa" rev-parse HEAD) == "$mesa_patched" ]] || {
+ echo 'title-check: unexpected patched Mesa revision' >&2; exit 2;
+}
+for patch in "$root"/patches/mesa/*.patch; do
+ git -C "$mesa" apply --reverse --check "$patch"
+done
+export PS5_MESA_FORK="$mesa" PS5_MESA_REVISION="$mesa_patched"
+export PS5_VULKAN_ROOT="$vk" PS5_PAYLOAD_SDK="$vk/.deps/native/ps5-payload-sdk"
+# Upstream build-radv invokes Ninja without a job limit; bound both host and target stages.
+ninja_real="$(command -v ninja)"
+printf '#!/bin/sh\nexec "%s" -j "%s" "$@"\n' "$ninja_real" "$BUILD_JOBS" > "$output/tools/ninja"
+chmod +x "$output/tools/ninja"
+export NINJA="$output/tools/ninja"
+timeout 1800 bash "$vk/tools/build-radv.sh" release
+
+# make libc already built this exact host tool, without the unrelated demo title.
+mkdir -p "$vk/build/host"
+cp "$vk/build/runtime-shim/ps5-native-tool" "$vk/build/host/ps5-native-tool"
+# The original OpenGL release was removed. Use the mirror documented in BUILDING.md,
+# with the original archive hash; the pinned UI fetcher also verifies every SDK file.
+glcache="$ui/.deps/ps5-opengl"
+glarchive="$glcache/ps5-opengl-sdk-1.0.0.tar.gz"
+if [[ ! -f "$glcache/ps5-opengl-sdk-1.0.0/sdk/manifest.sha256" ]]; then
+ mkdir -p "$glcache"
+ curl -fL --retry 2 --max-time 300 \
+ https://github.com/MyLuxy/overdrive-deps/releases/download/ps5-opengl-v1.0.0/ps5-opengl-sdk-1.0.0.tar.gz \
+ -o "$glarchive.part"
+ echo "f93643c04c843d56143b00951df1f8042ea7706ae9f19e4e9abf158e1ead77c5 $glarchive.part" | sha256sum --check --status
+ mv "$glarchive.part" "$glarchive"
+fi
+glsdk=$(bash "$ui/tools/fetch-opengl-sdk.sh")
+export PS5_OPENGL_SDK="$glsdk"
+# Cross-build validation omits llama.cpp's optional browser UI (which otherwise
+# falls back to a floating Hugging Face "latest" download). Keep the console UI.
+[[ ! -e "$root/third_party/llama.cpp/tools/ui/dist/index.html" ]] || {
+ echo 'title-check: remove custom source-tree browser assets or use a clean checkout' >&2; exit 2;
+}
+rm -rf -- "$root/build/app-ps5/llama/tools/ui/dist"
+export PS5LM_PREBUILT_UI=OFF
+timeout 1800 bash scripts/build-app.sh
+bash "$vk/tools/inspect.sh" "$root/build/app-ps5/eboot.elf"
+title_id=$(python3 -c 'import json; print(json.load(open("ps5/app/sce_sys/param.json"))["titleId"])')
+app="$root/build/app-ps5/title/$title_id"
+bash "$vk/tools/inspect.sh" "$app/eboot.bin"
+"$vk/build/host/ps5-native-tool" self --inspect --file "$app/eboot.bin"
+python3 - "$output" "$app" "$mesa_base" "$mesa_patched" "$ui_revision" "$llama_revision" "$vk_revision" "$sdk_revision" <<'PY'
+import hashlib
+import json
+from pathlib import Path
+import sys
+
+output, app = map(Path, sys.argv[1:3])
+files = ('eboot.bin', 'sce_module/libc.prx', 'sce_sys/param.json', 'sce_sys/icon0.png', 'sce_sys/pic0.dds', 'sce_sys/pic1.dds')
+result = {
+ 'status': 'passed',
+ 'scope': 'Complete native title cross-build and static ELF/FSELF validation; NOT console/Vulkan execution',
+ 'mesa_base_revision': sys.argv[3],
+ 'mesa_patched_revision': sys.argv[4],
+ 'ui_revision': sys.argv[5],
+ 'llama_revision': sys.argv[6],
+ 'ps5_vulkan_revision': sys.argv[7],
+ 'ps5_payload_sdk_revision': sys.argv[8],
+ 'optional_browser_ui': 'omitted; native console dashboard is built',
+ 'files_sha256': {name: hashlib.sha256((app / name).read_bytes()).hexdigest() for name in files},
+}
+(output / 'result.json').write_text(json.dumps(result, indent=2) + '\n')
+print(json.dumps(result, indent=2))
+PY
diff --git a/tests/test_build_app.py b/tests/test_build_app.py
new file mode 100644
index 0000000..7900d11
--- /dev/null
+++ b/tests/test_build_app.py
@@ -0,0 +1,203 @@
+"""Exercise build-app.sh's real UI compiler queue without a PS5 SDK.
+
+The extracted fragment is deliberately bounded by stable surrounding commands;
+if the build script is reorganized, update this harness rather than testing a
+copy of the scheduling algorithm.
+"""
+
+import json
+import os
+from pathlib import Path
+import shlex
+import subprocess
+import sys
+import tempfile
+import time
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+# One locked record makes overlapping fake compiler processes observable without
+# timing assertions. Both the C and C++ paths use this executable.
+COMPILER = r'''
+import fcntl
+import json
+import os
+from pathlib import Path
+import sys
+import time
+
+source = Path(sys.argv[sys.argv.index('-c') + 1])
+output = Path(sys.argv[sys.argv.index('-o') + 1])
+state_path = Path(os.environ['QUEUE_STATE'])
+def record(starting):
+ with state_path.open('r+') as stream:
+ fcntl.flock(stream, fcntl.LOCK_EX)
+ state = json.load(stream)
+ state['active'] += 1 if starting else -1
+ state['peak'] = max(state['peak'], state['active'])
+ if starting:
+ state['started'].append(source.name)
+ stream.seek(0)
+ json.dump(state, stream)
+ stream.truncate()
+
+record(True)
+failed = source.name == os.environ.get('FAIL_SOURCE')
+try:
+ # A failure can finish before the parent reaches wait; successful compilers
+ # stay alive briefly so the concurrency bound is meaningfully exercised.
+ if not failed:
+ time.sleep(0.08)
+ output.write_text('fake object\n')
+finally:
+ record(False)
+sys.exit(17 if failed else 0)
+'''
+
+
+class BuildAppQueueTests(unittest.TestCase):
+ @classmethod
+ def setUpClass(cls):
+ script = (ROOT / 'scripts/build-app.sh').read_text()
+ start = script.index('kit_objects=()\n')
+ end = script.index('rm -f "$obj/libps5lm_kit.a"', start)
+ cls.queue = script[start:end]
+
+ def run_queue(self, jobs='2', fail=None, cached=(), count=5, host_cpus=2):
+ with tempfile.TemporaryDirectory(prefix='ps5lm-build-queue-') as directory:
+ root = Path(directory)
+ kit = root / 'kit'
+ obj = root / 'obj'
+ (kit / 'src').mkdir(parents=True)
+ (obj / 'kit').mkdir(parents=True)
+ (root / 'ps5/app').mkdir(parents=True)
+ compiler = root / 'ps5/app/ps5cc'
+ compiler.write_text(f'#!{sys.executable}\n' + COMPILER)
+ compiler.chmod(0o755)
+ state_path = root / 'state.json'
+ state_path.write_text(json.dumps({'active': 0, 'peak': 0, 'started': []}))
+ sources = []
+ for index in range(count):
+ extension = 'c' if index % 2 else 'cpp'
+ source = kit / 'src' / f'{index}.{extension}'
+ source.write_text('// fake input\n')
+ sources.append(source)
+ if index in cached:
+ output = obj / 'kit' / (source.name + '.o')
+ output.write_text('cached object\n')
+ # Avoid depending on filesystem timestamp granularity.
+ os.utime(output, (time.time() + 10, time.time() + 10))
+
+ setup = '\n'.join([
+ 'set -euo pipefail',
+ f'root={shlex.quote(str(root))}',
+ f'kit={shlex.quote(str(kit))}',
+ f'obj={shlex.quote(str(obj))}',
+ 'kitflags=()',
+ 'kit_sources=(' + ' '.join(shlex.quote(str(p)) for p in sources) + ')',
+ 'cxx() { "$root/ps5/app/ps5cc" "$@"; }',
+ # Keep the default deterministic, independent of runner size.
+ f'nproc() {{ printf "{host_cpus}\\n"; }}',
+ ])
+ env = os.environ.copy()
+ env.pop('CMAKE_BUILD_PARALLEL_LEVEL', None)
+ env.pop('FAIL_SOURCE', None)
+ if jobs is not None:
+ env['CMAKE_BUILD_PARALLEL_LEVEL'] = jobs
+ if fail is not None:
+ env['FAIL_SOURCE'] = sources[fail].name
+ env['QUEUE_STATE'] = str(state_path)
+ completed = subprocess.run(
+ ['bash', '-c', setup + '\n' + self.queue],
+ env=env, capture_output=True, text=True, timeout=10,
+ )
+ state = json.loads(state_path.read_text())
+ objects = {p.name: p.read_text() for p in (obj / 'kit').glob('*.o')}
+ return completed, state, objects
+
+ def test_success_respects_job_limit(self):
+ for jobs in ('1', '2', '3'):
+ with self.subTest(jobs=jobs):
+ result, state, objects = self.run_queue(jobs=jobs)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(len(state['started']), 5)
+ self.assertEqual(state['active'], 0)
+ self.assertGreaterEqual(state['peak'], 1)
+ self.assertLessEqual(state['peak'], int(jobs))
+ self.assertEqual(len(objects), 5)
+
+ def test_fast_compiler_failures_are_not_lost(self):
+ # The old nproc/bare-wait loop loses failures on a many-core host;
+ # CMAKE_BUILD_PARALLEL_LEVEL must take precedence over that host size.
+ for position in (0, 2, 4):
+ with self.subTest(position=position):
+ result, state, _ = self.run_queue(fail=position, host_cpus=64)
+ self.assertEqual(result.returncode, 17, result.stderr)
+ self.assertEqual(state['active'], 0)
+ self.assertLessEqual(state['peak'], 2)
+
+ def test_cached_objects_skip_compilation(self):
+ result, state, objects = self.run_queue(cached=(0, 2, 4))
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(sorted(state['started']), ['1.c', '3.c'])
+ self.assertEqual(state['active'], 0)
+ for index in (0, 2, 4):
+ self.assertEqual(objects[f'{index}.cpp.o'], 'cached object\n')
+
+ def test_all_cached_or_empty_sources_need_no_wait(self):
+ for count, cached in ((0, ()), (5, range(5))):
+ with self.subTest(count=count):
+ result, state, _ = self.run_queue(count=count, cached=cached)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(state['started'], [])
+ self.assertEqual(state['active'], 0)
+
+ def test_invalid_job_limits_fail_before_compiling(self):
+ for jobs in ('0', '-1', '01', 'two'):
+ with self.subTest(jobs=jobs):
+ result, state, _ = self.run_queue(jobs=jobs)
+ self.assertEqual(result.returncode, 2, result.stderr)
+ self.assertIn('invalid job limit', result.stderr)
+ self.assertEqual(state['started'], [])
+
+ def test_default_uses_nproc(self):
+ result, state, _ = self.run_queue(jobs=None)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertLessEqual(state['peak'], 2)
+ self.assertEqual(len(state['started']), 5)
+
+
+class NativeBuildPreflightTests(unittest.TestCase):
+ def test_early_failure_removes_previous_success_report(self):
+ cases = (
+ ({'BUILD_JOBS': '0'}, 'BUILD_JOBS must be a positive integer'),
+ ({'BUILD_JOBS': '2', 'LLVM_CONFIG': 'ps5lm-nonexistent-llvm-config-for-test'},
+ 'LLVM_CONFIG'),
+ )
+ for overrides, message in cases:
+ with self.subTest(overrides=overrides):
+ with tempfile.TemporaryDirectory(prefix='ps5lm-native-preflight-') as directory:
+ root = Path(directory)
+ (root / 'scripts').mkdir()
+ script = root / 'scripts/test-native-build.sh'
+ script.write_text((ROOT / 'scripts/test-native-build.sh').read_text())
+ report = root / 'build/native-ci/result.json'
+ report.parent.mkdir(parents=True)
+ report.write_text('{"status": "passed"}\n')
+ env = os.environ.copy()
+ env.update(overrides)
+ result = subprocess.run(
+ ['bash', str(script)], env=env,
+ capture_output=True, text=True, timeout=5,
+ )
+ self.assertEqual(result.returncode, 2, result.stderr)
+ self.assertIn(message, result.stderr)
+ self.assertFalse(report.exists())
+ # These failures must occur before any dependency bootstrap.
+ self.assertFalse((root / '.deps').exists())
+
+
+if __name__ == '__main__':
+ unittest.main()