llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8
Docs: what other PS5 homebrew taught us
Mert Cobanov4173291e0554
diff --git a/docs/CONSOLE.md b/docs/CONSOLE.mdindex 6f8431f..9ee5f75 100644--- a/docs/CONSOLE.md+++ b/docs/CONSOLE.md@@ -22,12 +22,17 @@ The Relapse kernel stage sometimes hangs and the console then shuts down, or stiThe flexible pool is shared with the system: holding about 6 GiB froze the home screen until the process ended. Plan on 4 to 5 GiB for model, context and buffers in a payload. Bigger models (Qwen 3.8 27B) need a process with direct memory, that is a native app.+Why a payload has no direct memory: elfldr starts every payload as a new process with budget 0 (`sys_budget_set(0)` before it executes SceSpZeroConf, `elfldr.c` in ps5-payload-dev/elfldr), the budget of a system app. `0x80020023` is `0x80020000 + EAGAIN`: the pool is empty, the call is not refused. etaHEN's jailbreak request changes a process's rights (uid, sandbox, authid), not its budget, and kstuff patches no budgets; every homebrew project using more than 6 GiB is a native app.+## CPU, for one payload process- AVX, AVX2, FMA, F16C, BMI2; no AVX-512. 16 logical CPUs online.- **Affinity mask `0xea00`: only CPUs 9, 11, 13, 14 and 15**, shared with system services. Sony priority 700 (normal) for the main thread and new threads.- 1, 2 and 4 threads busy for a second each: fine.- Busy-waiting on all five (llama.cpp's defaults: one thread per physical core, `--poll 50`) starved the system: services stopped answering and the console shut down twice. `patches/0001-ps5-safe-cpu-defaults.patch` makes the default thread count the allowed CPUs minus two and turns polling off.+- `--poll 0` only ends the spinning between graphs. ggml's barrier spins at every operation inside a graph whatever the poll value, so during generation every worker runs flat out: the thread count is the real load setting.+- Only CPUs 14 and 15 are the two halves of one core (SMT siblings are 2k and 2k+1, as PS5SX2 and PS5CEMU-HAR lay them out). 9, 11 and 13 share their cores with threads outside the payload.+- llama.cpp's `-C`, `--cpu-strict` and `--prio` did nothing here: ggml took the PS5 for an unsupported platform. `patches/0002-ps5-thread-affinity-and-priority.patch` maps them to `scePthreadSetaffinity` and `scePthreadSetprio`, only ever lowers priority, and leaves the main thread unpinned. Not measured yet (#7).## llama.cpp@@ -62,7 +67,7 @@ Locking the weights (`-lm mlock`) fixes it: Qwen3.5 2B went from 0.9 to **8.5 to- The kernel caps `SO_RCVBUF` at 64 KB, whatever is asked for. With the Hugging Face CDN ~150 ms away, one connection tops out at 0.9 MB/s.- 16 connections fetching 8 MiB pieces in parallel: **about 12.5 MB/s** (Qwen3.5 4B, 2.6 GB, in 217 s).-At exit, Sony's libc prints `[SceLibc] A heap error is detected` (SceLibcInternalHeap) after all work is done. Not investigated yet.+At exit, Sony's libc prints `[SceLibc] A heap error is detected` (SceLibcInternalHeap) after all work is done. After `main`, the SDK's start code unloads the modules the payload loaded and calls Sony's `exit()`, which runs the C++ static destructors while other threads may still run; the tools now leave through `_Exit` instead (`ps5/compat/exit.cpp`). Not yet checked on the console (#3).## shsrv quirksdiff --git a/docs/PORTING.md b/docs/PORTING.mdindex 0ad26e1..4ab1a22 100644--- a/docs/PORTING.md+++ b/docs/PORTING.md@@ -6,7 +6,7 @@ What it took to build llama.cpp for the PS5, and why. Upstream is pinned at `b11The SDK compiles with clang's own PS5 target, `x86_64-sie-ps5`, which defines `__PROSPERO__`, `__SCE__` and `__FreeBSD__ 9`. Its CMake toolchain (`prospero-cmake`) presents the console as FreeBSD on x86_64. Programs link dynamically against stubs of Sony's libraries (`libSceLibcInternal`, `libkernel_web`, `libSceNet`, ...) and run as payloads under elfldr.-ggml and libllama build for it as they are. One source patch so far, for safety on the console rather than for building (below).+ggml and libllama build for it as they are. Two source patches so far, for the console's CPUs rather than for building (below).## Build fixes (all in `scripts/build-llama.sh`)@@ -14,11 +14,18 @@ ggml and libllama build for it as they are. One source patch so far, for safety2. **There is no `libm`.** ggml links `m` unless `MATH_LIBRARY` is set; the math functions live in Sony's libc, so `-DMATH_LIBRARY=SceLibcInternal`.3. **`accept4` is declared but not exported.** The SDK's `sys/socket.h` declares it (FreeBSD 10), libkernel exports only `accept`, and cpp-httplib's server loop calls `accept4(..., SOCK_CLOEXEC)`. `ps5/compat/compat.c` implements it on `accept` and `fcntl`, and goes last on every link line through `CMAKE_C_STANDARD_LIBRARIES` and `CMAKE_CXX_STANDARD_LIBRARIES`.-Build options: AVX, AVX2, FMA, F16C and BMI2 on, AVX-512 off (Zen 2), no OpenMP (ggml's own thread pool), static libraries, no OpenSSL.+Build options: AVX, AVX2, FMA, F16C and BMI2 on, AVX-512 off (Zen 2), no OpenMP (ggml's own thread pool), static libraries, no OpenSSL. mbedTLS is built with `MBEDTLS_THREADING_C` and `MBEDTLS_THREADING_PTHREAD` (`scripts/build-mbedtls.sh`): ps5lm downloads over 16 connections at once, TLS 1.3 handshakes share PSA's global key store, and that store is only thread-safe with both (mbedTLS `docs/architecture/psa-thread-safety`; pacbrew's PS5 package sets them too).++## Leaving a tool (`ps5/compat/exit.cpp`)++After `main` returns, the SDK's start code (`crt/crt.c`) unloads the Sony modules the payload loaded and calls Sony's `exit()`, which runs every C++ static destructor registered with `__cxa_atexit` while other threads may still run. The SDK moved to this in 2024 (commit f19dd3e); before, payloads ended with the `_exit` syscall. `[SceLibc] A heap error is detected` comes at exit, after the tool's work is done, which points at this phase. `ps5/CMakeLists.txt` links `compat/exit.cpp` into the tools with `--wrap=main`: the start code's call lands in `__wrap_main`, which runs the tool's `main`, drains llama.cpp's log thread and stdio, and leaves through `_Exit`. ps5lm never returns from `main` and leaves the same way when asked to quit.++ProsperoAI's libc replacements (`pthread_once`, `strtof`, `fseek`, an mmap allocator for large blocks) are not copied: ProsperoAI is a native app on a clean-room `libc.prx` whose stubs they patch over, while a payload binds to Sony's own libc, and a second allocator would bring exactly the mixed frees the heap check catches.## Source patches (`patches/`, applied by `scripts/build-llama.sh`)1. **`0001-ps5-safe-cpu-defaults.patch`.** A payload may only use 5 of the 16 logical CPUs, and system services share them. llama.cpp's defaults (one thread per physical core, busy-waiting at `--poll 50`) took all five and the console shut down. On `__PROSPERO__` the default thread count becomes the CPUs in the affinity mask (`scePthreadGetaffinity`) minus two, and polling defaults to 0. `-t` and `--poll` still override.+2. **`0002-ps5-thread-affinity-and-priority.patch`.** The SDK's compiler defines `__FreeBSD__` but not `__linux__`, so ggml's thread pinning and priority fell into its "unsupported platforms" branch and `-C`, `--cpu-strict` and `--prio` did nothing. On `__PROSPERO__` they go through `scePthreadSetaffinity` and `scePthreadSetprio`. Priority is only lowered (`--prio -1` gives 720; the default is 700, and 256 runs first), the way PS5SX2 and PS5CEMU-HAR only raise threads that mostly sleep. The calling thread is never pinned: threads it starts later inherit its mask, which left a PS5SX2 helper thread stuck on one CPU (PS5SX2 commit f9abb4a). `set_process_priority` refuses to raise anything, since what `setpriority` does to a payload is unknown.## The build@@ -32,4 +39,4 @@ Build options: AVX, AVX2, FMA, F16C and BMI2 on, AVX-512 off (Zen 2), no OpenMP## On the console-Runs: see [CONSOLE.md](CONSOLE.md). mmap of the model works; one block of 6 GiB is the ceiling; `std::thread::hardware_concurrency()` reports 16 while only 5 CPUs are allowed, hence the patch. Open: the `[SceLibc] A heap error is detected` message at exit.+Runs: see [CONSOLE.md](CONSOLE.md). mmap of the model works; one block of 6 GiB is the ceiling; `std::thread::hardware_concurrency()` reports 16 while only 5 CPUs are allowed, hence the patch. Open: whether leaving through `_Exit` ends the `[SceLibc] A heap error is detected` message (#3).diff --git a/docs/RESEARCH.md b/docs/RESEARCH.mdindex d8b1482..b1f87e1 100644--- a/docs/RESEARCH.md+++ b/docs/RESEARCH.md@@ -24,15 +24,37 @@ On top of it: kstuff (fake-signed executables), a HEN (etaHEN or OnionHEN) and [**[PS5SX2](https://github.com/Swordpdf/PS5SX2)**, PCSX2 on the PS5 (September 2026). The best public notes on the console's memory as a native app sees it:- `mmap` in the app sandbox fails; flexible memory (`sceKernelMapFlexibleMemory`) is only ~190 to 448 MiB-- large data goes in direct memory; the Vulkan driver offers a 12 GiB device heap with `PS5VK_WIDE_MEMORY`+- large data goes in direct memory; its own fork of the ps5vk driver offers a 12 GiB device heap with `PS5VK_WIDE_MEMORY` (a PS5SX2 switch, not one of PS5_Vulkan's)- JIT and other rights come from the HEN: the app writes `{"PID":n}` to `/download0/etahen_jailbreak`**[ProsperoAI](https://github.com/blackbearreloaded/ProsperoAI)** (GPL-3.0), LLM chat on the PS5 GPU (September 2026). Runs Qwen3.5 9B Q4_0 (6.37 GiB) at about 30 tok/s decode and Mistral 7B Q4_0 at 25 to 29 tok/s, with a 4,096 token context. It writes AGC kernels by hand for each model (`src/backends/qwen35`, `src/backends/mistral`) from per-model recipes and supports Q4_0 only, so a new model or quant means new kernels. It vendors llama.cpp's CPU libraries built for the PS5. Proof that a 6+ GiB model and fast GPU decode work on this console.-**[PS5_Vulkan](https://github.com/mihawk-99/PS5_Vulkan)** (Mihawk-99), a Vulkan driver for the PS5's GPU built on Mesa 26.2's RADV. Shaders go SPIR-V → NIR → ACO → AGC (the PSBC compiler from the ps5-opengl SDK). Reports Vulkan 1.4 and 1,569,390 passing CTS cases with no failures; no async compute queues yet. Runs vkQuake at 120 fps in 4K, RetroArch's Vulkan cores and the Eden emulator.+**[PS5_Vulkan](https://github.com/mihawk-99/PS5_Vulkan)** (Mihawk-99, GPL-3.0) builds two Vulkan drivers for the PS5's GPU. The one going forward is RADV, Mesa 26.2's AMD driver, from the author's fork [PS5_Mesa](https://github.com/mihawk-99/PS5_Mesa) with a PS5 winsys over the console's AGC functions. It reports Vulkan 1.4 and compiles shaders with ACO on the console when a pipeline is created. ps5vk, the first driver (Vulkan 1.1), is the one built on the PSBC compiler from the ps5-opengl SDK. vkQuake, RetroArch's Vulkan cores and ProsperoEden run on RADV. For ggml-vulkan it has what is required (Vulkan 1.2 or later, 16-bit storage, timeline semaphores, subgroup operations, fp16, `maxStorageBufferRange` of 4 GiB) but no cooperative matrix and no accelerated integer dot product, so matrix products take the fp16 shaders. A submit that makes no progress for 10 s ends in `VK_ERROR_DEVICE_LOST` (`radv_ps5_winsys.c`). Its measurements are from firmware 6.02 to 12.70, mostly on a PS5 Pro; 13.60 is untested.**[ps5-native-app-boilerplate](https://github.com/blackbearreloaded/ps5-native-app-boilerplate)** (GPL-3.0): the build of a native app (`eboot.bin` as an fSELF, `sce_sys/param.json`, `sce_module/libc.prx`) for ShadowMountPlus.+## Lessons from other homebrew++Read from the code, fix commits and closed issues of ps5-payload-dev (sdk, elfldr, websrv, ftpsrv, SDL, pacbrew-repo), Payload Manager, PS5 Game Compressor, etaHEN, kstuff, PS5SX2, PS5CEMU-HAR, ProsperoAI, ps5-opengl and PS5_Vulkan, in October 2026. What PS5LM took from them, and where it came from:++| Lesson | Source | In PS5LM |+|---|---|---|+| After rest mode a listening socket can go stale without an error; rebuild the server on `SIGCONT`, on a new address, and when it stops answering | Payload Manager #61, fixed in cd9ad1f; websrv e5a6369; ftpsrv 4f7408a | ps5lm's watchdog |+| Set only `SO_REUSEADDR`; cpp-httplib's default `SO_REUSEPORT` lets a second copy bind the same port | ftpsrv, websrv, Payload Manager servers | ps5lm's single-instance check |+| Name the process with `thr_set_name`; Payload Manager finds payloads by that name | ftpsrv, websrv, shsrv, Payload Manager | ps5lm |+| Never let the PS5 browser cache a homebrew page | Payload Manager #40 and 3efc4f4 | `Cache-Control: no-store` |+| A wrong console clock shows up as TLS failures | Payload Manager #23, #45, #54 | the download error says so |+| `statfs` on `/data` reports more free space than Settings | PS5 Game Compressor #43 | 512 MiB kept spare |+| Sync data before recording it done or renaming it | PS5 Game Compressor (`fsync` before `rename`) | download piece record |+| Reset the auto power-down timer during long work | PS5 Game Compressor `gc_power_guard.c` | during downloads |+| mbedTLS needs `THREADING_C` and `THREADING_PTHREAD` for concurrent TLS 1.3 | mbedTLS docs; pacbrew's mbedtls package | `build-mbedtls.sh` |+| Pin with `scePthreadSetaffinity` inside the given mask, never widen it, and keep the SMT sibling free; raise only threads that mostly sleep | PS5SX2 (`GSRenderer.cpp`), PS5CEMU-HAR (`port/ps5/threads.cpp`) | `patches/0002` |+| A pinned thread's children inherit its mask | PS5SX2 f9abb4a | main thread left unpinned |+| `std::thread`'s default stack is small; a 64 KB frame overflowed it | PS5SX2 30814bd | `spawn()` keeps 4 MiB stacks |+| Do not run exit-time destructors behind live threads | SDK `crt.c` and f19dd3e; ProsperoAI and the boilerplate never return from `main` | `ps5/compat/exit.cpp` |++For the native app (#4), the projects agree on one recipe for big memory: `sceKernelAllocateDirectMemory(0, sceKernelGetDirectMemorySize(), len, 2 MiB, 12, &phys)`, then `sceKernelMapDirectMemory(&addr, len, 0x33, 0, phys, 2 MiB)` (CPU and GPU read/write), freed with `sceKernelMunmap` and `sceKernelReleaseDirectMemory` (PS5SX2 `pcsx2/Memory.cpp`, PS5_Vulkan `driver/ps5vk_direct_memory.c`, ProsperoAI `src/ps5_agc_backend.cpp`, PS5CEMU-HAR `port/cemu/MemMapperPS5.cpp`). Without an address hint the kernel places mappings first fit in the window the GPU driver needs later, and `vkCreateDevice` then fails (PS5SX2 0ac429f): hint CPU buffers elsewhere, as PS5CEMU-HAR (0x10_0000_0000) and PS5_Vulkan (0x40_0000_0000) do. An app's flexible heap is only 448 MiB configured, so big `std::vector`s and the KV cache belong in direct memory too. ProsperoAI, a native app, maps one block of about 7.38 GiB and adds `amm` and `kernel` page table sizes to its `param.json`. The etaHEN jailbreak request (`{"PID":n}` written to `/download0/etahen_jailbreak.tmp`, then renamed to `/download0/etahen_jailbreak`) needs the title ID on etaHEN's allowlist and gives rights, not memory.+## Qwen 3.8- **Qwen3.8-27B** (14 August 2026, Apache 2.0): dense, hybrid attention (Gated DeltaNet and gated attention), vision encoder, 262K context, a native MTP block. GGUFs from [unsloth](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) use the `qwen35` architecture and need llama.cpp from its release week or later.diff --git a/docs/ROADMAP.md b/docs/ROADMAP.mdindex c407307..9fe787a 100644--- a/docs/ROADMAP.md+++ b/docs/ROADMAP.md@@ -65,9 +65,9 @@ An Ollama-like way to get models and chat, without a PC in the loop. One payload## Phase 2: fit and speed on the CPU-- [ ] Weights in direct memory if malloc's ceiling is too low: a ggml CPU buffer type on `sceKernelAllocateDirectMemory`, as a patch in `patches/`+- [ ] Weights in direct memory if malloc's ceiling is too low: a ggml CPU buffer type on `sceKernelAllocateDirectMemory`, as a patch in `patches/`. Needs the native app: a payload's budget has no direct memory ([CONSOLE.md](CONSOLE.md)); the recipe is in [RESEARCH.md](RESEARCH.md)- [ ] Loading: mmap or `--no-mmap`, from `/data` or a USB drive, whichever Phase 0 shows is faster-- [ ] Threads: the best `-t` for the cores a payload is allowed, with pinning+- [ ] Threads: the best `-t` for the cores a payload is allowed, with pinning (`patches/0002` makes `-C` and `--prio` work, #7)- [ ] Qwen 3.8's native MTP block for speculative decoding- [ ] Context budget: 8k to 32k tokens@@ -75,13 +75,14 @@ An Ollama-like way to get models and chat, without a PC in the loop. One payload## Phase 3: the GPU through Vulkan-The route: ggml's Vulkan backend on [Mihawk-99's PS5_Vulkan](https://github.com/mihawk-99/PS5_Vulkan), a RADV based driver that programs the PS5's GPU (AGC) directly and passes the Vulkan CTS. llama.cpp keeps its own kernels, so every model and quant comes along.+The route: ggml's Vulkan backend on [Mihawk-99's PS5_Vulkan](https://github.com/mihawk-99/PS5_Vulkan), whose RADV driver (Mesa 26.2 from [PS5_Mesa](https://github.com/mihawk-99/PS5_Mesa), on a PS5 winsys over the console's AGC functions) reports Vulkan 1.4 and compiles shaders with ACO on the console. llama.cpp keeps its own kernels, so every model and quant comes along. RADV has everything ggml-vulkan requires but no cooperative matrix and no accelerated integer dot product, so matrix products take the fp16 shaders ([RESEARCH.md](RESEARCH.md)).-- [ ] Build the driver on Linux (an OrbStack Ubuntu machine is enough)-- [ ] App shell: `eboot.bin` with `sce_sys/param.json`, from [ps5-native-app-boilerplate](https://github.com/blackbearreloaded/ps5-native-app-boilerplate), mounted by ShadowMountPlus, jailbroken at start through the HEN request PS5SX2 uses-- [ ] ggml-vulkan cross-compiled, its shaders compiled to SPIR-V on the host+- [ ] Build RADV: `tools/setup-native-dependencies.sh`, then `tools/build-radv.sh release` in PS5_Vulkan (CachyOS or Arch is the tested host; an Arch machine in OrbStack avoids an untested Ubuntu). Output: `libvulkan_radeon.ps5.a`+- [ ] App shell: `eboot.bin` with `sce_sys/param.json`, from [ps5-native-app-boilerplate](https://github.com/blackbearreloaded/ps5-native-app-boilerplate), mounted by ShadowMountPlus, jailbroken at start through the HEN request PS5SX2 uses (the title ID must be on etaHEN's allowlist)+- [ ] ggml-vulkan cross-compiled, its shaders compiled to SPIR-V on the host, linked as `tools/radv-link.sh` does (`--whole-archive`, `--defsym` for the `vk*` entry points ggml calls directly)- [ ] `test-backend-ops` on the console: every op the `qwen35` graph uses matches the CPU-- [ ] Weights in device memory (the driver's 12 GiB heap, `PS5VK_WIDE_MEMORY`)+- [ ] Prompt processing in small batches: a submit that makes no progress for 10 s loses the device, and Gated DeltaNet loops over every token of a batch in one dispatch+- [ ] Weights in device memory. RADV's heaps come out of one direct memory pool of about 12 GiB, shared with the CPU side, so UD-IQ2_XXS (6.77 GiB) before UD-Q2_K_XL- [ ] Qwen3.8-27B decoding on the GPU, target 15 tok/s or more**Done when** the GPU decodes at least five times faster than the CPU.