llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8
Experimental fleet: one model, several PS5 workers
Status: host controller, and a PS5 worker that is part of the app and starts
only when you opt in; CPU RPC inference and the title's cross-build are tested
in CI. On hardware, one PS5 worker has served one model together with a PC's
CPU, a Mac's GPU (Metal) and an RTX 3090 (CUDA and Vulkan) (2026-11-01 and
11-02, below); more consoles at once are not validated by the maintainers. This is not a performance claim. Up to v0.10.0 the worker was a
separate build, ps5lm-app-fleet-v0.10.0.zip; from v0.11.0 every PS5LM has it.
What it is for: a model that does not fit in one console's memory runs across
several, each holding a run of its layers. Every token passes through every
worker over the network, so it is not faster than one console; it makes larger
models possible. Workers are PS5s; a PC can join too, with its CPU or, with a
GPU host build (PS5LM_FLEET_GPU), its GPU.
Quick start
- On each PS5: PS5LM v0.11.0 or later (for v0.10.0, its
ps5lm-app-fleet-v0.10.0.zipfrom the releases), then put/data/PS5LM/fleet-worker.txton it over FTP: the console's own IP and a port below 40000, then the acknowledgement line (Configure workers). Start PS5LM: it shows the worker screen instead of the dashboard. - On a Linux or macOS PC on the same network:
git submodule update --init --depth 1 third_party/llama.cpp, thenscripts/build-fleet-host.sh(Host). To add the PC's GPU as a worker, build withPS5LM_FLEET_GPU=metal(orcuda,vulkan) and startbuild/fleet-host/bin/ggml-rpc-server -H <PC IP> -p 30053 -d MTL0(CUDA0,Vulkan0), then list it as one more node. - Nodes: copy
fleet/nodes.example.jsonand list each console'sIP:portwith its memory budget. - Plan, then serve:
python3 -m fleet plan --model model.gguf --nodes nodes.jsonshows which layers go where;python3 -m fleet serve ... --ack-insecure-rpcstarts one llama-server on127.0.0.1:8081of the PC (Plan and serve). - Back to normal: delete
fleet-worker.txtand restart PS5LM.
Dense llama and qwen2 GGUFs only; Qwen 3.5/3.8 (qwen35) is refused. RPC
has no authentication: keep workers on a network you trust. On macOS, the app
that runs the controller (Terminal, iTerm, a multiplexer) needs Local Network
permission, or connections to the consoles fail with "No route to host".
Architecture
GGUF file / public HTTPS URL
|
Linux/macOS controller: bounded download + SHA256 + GGUF inspection
| contiguous layer/KV capacity plan
one llama-server (API bound to 127.0.0.1)
/ \
llama.cpp RPC llama.cpp RPC
PS5 A: Vulkan layers 0..k PS5 B: Vulkan layers k+1..L + output
This is model parallelism, not load balancing. A request needs every selected worker. The host holds the complete GGUF on disk, loads input/shared tensors locally, and upstream llama.cpp transfers the assigned tensors to each RPC backend. There is no separate shard-file format, full-model replica per worker, custom activation protocol, or independent model server on each console. RPC also manages the distributed computation and per-layer KV cache. Transfers may return through the controller; this does not promise direct PS5-to-PS5 pipelines.
The pinned dependency already supplies this machinery. We keep its wire format
(RPC 8.0.0) and use --split-mode layer, an explicit RPC-only device list,
and --tensor-split. Midpoint boundaries reproduce the pinned loader's L+1
logical slots (the extra slot is output), avoiding floating-point boundary drift.
Host accelerators are not silently added. The planner minimises the maximum
estimated memory utilisation for the given node order, not latency or tokens/s.
Deliberately narrow model admission
- 2–15 selected workers, exactly one device per endpoint, at least one repeating layer per worker. The pinned CLI rejects 16 tensor-split entries.
- Complete, single-file, little-endian GGUF v3, dense
llamaorqwen2layouts, at most 512 layers, known tensor encodings and validated shapes. - One sequence, FP16 K/V cache, fixed context no greater than training context. KV is rounded up to 256 tokens conservatively. Flash attention, automatic fit, lazy loading and opportunistic host-operation offload are disabled.
- Unknown architectures, hybrid/recurrent state, MoE, split GGUF, missing tensors, unsupported quantizations and replicated RoPE weight tensors fail closed.
- Budgets include exact inspected weight spans, tied-output duplication, estimated per-layer KV and operator-selected reserves. Reserves must cover allocations, dequantization, activation/compute buffers and backend overhead. A successful plan is not an allocation guarantee or reservation. Live memory can change.
Why not the README's Qwen 3.8 27B yet? docs/MODELS.md identifies that model's
architecture as qwen35. PS5LM's normal single-console planner has broader
architecture support than this first fleet estimator. The fleet does not yet
model Qwen35's hybrid/recurrent state and placement, so it rejects that model
rather than extrapolating a dense-transformer KV formula. Next step is a
Qwen35-specific inspector/placement test against the pinned loader, followed by
small real RPC inference and two-console validation. This is not a claim that
upstream RPC itself cannot run it.
Security boundary — read before enabling workers
Upstream explicitly calls RPC experimental and insecure. It has no authentication or encryption and is not an ordinary safe inference API. Anyone who can reach it must be treated as trusted with the worker process and memory. IP validation is not authentication.
- Use an isolated, trusted wired LAN/VLAN with access limited to the controller. No guest/shared LAN, port forwarding, public interface, public proxy or tunnel.
- The worker requires an explicit numeric private IPv4 bind, high port and a
written acknowledgement. No wildcard
0.0.0.0or hostname is accepted. GGML_RPC_NO_COMM=1is forced by this integration.patches/fleet/0002makes the server rejectCOMM_INITbefore opening the extra wildcard peer listener in upstream tensor-parallel code. This fleet uses layer splitting, not those collectives.patches/fleet/0001initializes BSD socket lengths and suppresses SIGPIPE portably. Neither patch makes arbitrary RPC requests safe. Keep the network isolated.- Workers have no disk tensor cache; normal PS5LM API, model auto-load and downloads are disabled in worker mode. There is one controller per worker.
- The host API is hard-bound to loopback; the CLI starts a foreground process, cleans it up on Ctrl+C/SIGTERM, and does not deploy consoles, modify firewall rules, create credentials, enroll nodes or persist a background service.
Revisions and build
Developed on PS5LM c4595784ce75c1d58777b11ef812d3e087e74296, with llama.cpp
970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc and UI
bf44d80384d5b8f8f4f213cf98d748887211328d. Build host and workers from the same
checkout and patch set. The host manifest checks the local binary checksum and
llama revision. A remote handshake checks protocol compatibility, not a remote
Git revision or source attestation; the operator must deploy matching builds.
Host (CPU + RPC)
Python 3.10+, a C++17 compiler, CMake, Ninja and Git are required. NumPy and PyYAML are needed only to generate the tiny test fixture, not for the controller.
git submodule update --init --depth 1 third_party/llama.cpp
scripts/build-fleet-host.sh
# build/fleet-host/bin/{llama-server,ggml-rpc-server,...}
# build/fleet-host/fleet-build.json (revision, patches, binary SHA256)
The host script applies the repository's numbered llama.cpp patches and
patches/fleet, disables RDMA, and builds the RPC tools plus the upstream
multi-server regression. It is CPU only unless PS5LM_FLEET_GPU is metal,
cuda or vulkan; then ggml-rpc-server -d MTL0 (CUDA0, Vulkan0) serves
that GPU as a worker. The controller always names its RPC devices explicitly,
so a GPU in the host build is used only through a worker you start and list. It does not fetch model weights or start a listener. Set CMAKE,
NINJA or CMAKE_BUILD_PARALLEL_LEVEL for local tool locations/resources.
PS5 worker — part of the app
Follow BUILDING.md to prepare PS5_Vulkan, its pinned PS5_Mesa and
PS5_PayloadSDK forks, patched RADV, native packaging tool, libc.prx and the
checksum-pinned ps5-opengl SDK. Then:
git submodule update --init --depth 1
scripts/build-app.sh
Every build has the worker: llama.cpp with GGML_RPC=ON (GGML_RPC_RDMA=OFF)
and patches/fleet on top of the numbered patches, and the title with
fleet_worker.cpp and the worker's startup mode in main.cpp. It stays idle,
with no listener, unless /data/PS5LM/fleet-worker.txt exists. Up to v0.10.0
this was a separate PS5LM_FLEET=1 build and zip.
CI builds the title on every pull request (native-build.yml,
scripts/test-native-build.sh, with the structural ELF/FSELF checks,
CI.md). That proves the worker links, not that it boots, that its
Vulkan results are right, or that a
real multi-console run works. QEMU is not a substitute: its
x86 system models
and Linux/BSD user modes
provide no PS5 loader, AGC or direct memory.
On a console (2026-11-01): the fleet build on one PS5 (worker on
port 8082) and ggml-rpc-server on a Linux PC's CPU, with
python3 -m fleet serve on that PC, served qwen2.5-0.5b-instruct-q4_k_m.gguf
with blocks 0–22 on the PS5 and block 23 plus the output on the PC, and
answered a chat request correctly. This run found the 40000 port limit below;
the worker writes its own and ggml's messages to /data/PS5LM/app.log.
The same day, with the worker in the normal app (this build) and a Mac's GPU:
ggml-rpc-server -d MTL0 from a PS5LM_FLEET_GPU=metal host build on an Apple
M-series Mac, the controller on the Linux PC. Blocks 0–22 on the PS5, block 23
and the output on the Mac's GPU; it answered correctly at 13 tokens/s (7.4 with
the PC's CPU in that place). One small model, so not a performance figure.
On 2026-11-02, the same PS5 worker with an RTX 3090 on the Linux PC, from
PS5LM_FLEET_GPU=cuda and vulkan host builds (ggml-rpc-server -d CUDA0,
-d Vulkan0), the controller on the same PC: blocks 0–19 on the PS5, blocks
20–23 and the output on the 3090 (--tensor-split 39,11). Both answered
correctly, at 28 to 30 tokens/s on a longer reply. The controller's binary must
run on the PC it is started on: a build/fleet-host made inside a newer Linux
container stops at GLIBC_2.43 not found on an older host.
Configure workers (only after building and validating the title)
On each console, explicitly create /data/PS5LM/fleet-worker.txt containing
that console's own private IP, port 1024–39999, and the exact acknowledgement:
192.168.50.21:30052
I_UNDERSTAND_RPC_IS_UNAUTHENTICATED
The port must be below 40000: a PS5 title gets "Permission denied" for 40000
and up (measured on a console), so upstream's default 50052 cannot work there.
Workers on a PC or Mac (ggml-rpc-server) can keep 50052.
Restart PS5LM. It displays an experimental worker screen, not “Ready”: upstream has no ready callback, so the host must successfully handshake. A malformed, unreadable, non-regular or oversized config fails closed without starting the normal model server. No config keeps ordinary PS5LM behaviour. Remove this file and restart to return to single-console serving.
Restart the worker title between different model runs and after errors. PS5LM documents retained Vulkan memory after in-process unload; this increment does not promise safe hot reload. Stop through the normal title-close / existing quit-file mechanism. A worker failure fails the whole model; no failover/resume is implemented.
Plan and serve from the controller
Copy fleet/nodes.example.json, replace endpoints, and set conservative host and
worker memory/reserve values from measurements. The example's 10 GiB worker
budget with 2 GiB reserve is an operator example, not measured fleet capacity.
memory_mib is the total allowed budget before subtracting reserve_mib.
Do not run the normal local PS5 model alongside a worker.
# Offline capacity estimate from an already downloaded model:
python3 -m fleet plan --model /models/model.gguf \
--nodes my-nodes.json --context 4096 --plan-out plan.json
# The model argument also accepts a public HTTPS URL:
python3 -m fleet plan --model 'https://example.org/model.gguf' \
--sha256 <expected-full-file-sha256> --nodes my-nodes.json --context 4096
# Explicitly opt in after isolating/trusting the network:
python3 -m fleet serve --model /models/model.gguf \
--nodes my-nodes.json --context 4096 --ack-insecure-rpc --plan-out run-plan.json
plan never connects to workers and reports operator_budgets_only. A URL is
fully downloaded before inspection; the downloader validates public HTTPS and
redirect destinations, pins DNS to the TLS connection, bounds bytes/time/disk,
streams to a partial file and hashes the complete file. No gated model token,
remote code, pickle or arbitrary extraction is supported. Prefer an expected
--sha256; without it the digest records identity but not publisher authenticity.
Default download limit is 32 GiB, with --max-download-gib available up to 128.
Model URLs are not automatically downloaded in CI.
serve validates the local binary manifest, performs the upstream RPC handshake,
requires exactly the requested device order/topology, caps every operator budget
by reported live free memory, and repeats preflight after acquisition. It then
recomputes the plan and launches a single llama-server. It does not execute
an old JSON plan as trusted instructions. Wait for llama-server readiness:
curl --fail http://127.0.0.1:8081/health
curl --fail http://127.0.0.1:8081/completion \
-H 'Content-Type: application/json' \
-d '{"prompt":"The capital of France is", "n_predict":16, "temperature":0}'
Review server logs for actual device/layer allocation. Model-load failure, unsupported operations, worker disconnect or OOM is not converted into a successful fleet deployment. A snapshot/printed plan does not mean inference has started. Input and unsupported-operation work can still occur on the host; backend behaviour and memory must be checked on real consoles.
Tests / CI
scripts/test-fleet.sh # offline Python, C++ config/runtime, existing planner
python3 -m pip install -r tests/requirements-fleet.txt # fixture only; use a venv
scripts/test-fleet.sh --rpc # build + real localhost RPC regression/inference
The smoke creates a 231 KB, deterministic, untrained four-layer GGUF. It runs
a local CPU reference, two CPU RPC processes, and the actual python -m fleet serve CLI. It checks token IDs, assigned layers and per-worker computation.
This proves a real distributed execution path on host CPUs, not useful language
quality, PS5 GPU operation, large-model scaling or network speed. Tests use
loopback only and clean up their child processes.
.github/workflows/fleet-host.yml runs these checks on pull requests and pushes
on main or fleet/**, using ordinary Ubuntu hosted runners, read-only
repository permissions and per-ref cancellation. The host job has two parallel
compiler processes and a 25-minute timeout; the separate native-object/runtime
job has a 15-minute timeout. Full title CI uses a digest-pinned official Debian
container on an ordinary Ubuntu runner, a two-CPU quota and a 45-minute timeout.
Actions are pinned to commits; fixture dependencies are version-pinned.
Jobs check out the pinned submodules they need and install public build tools.
The host job downloads no trained model, PS5 SDK or driver. The native job does
fetch the pinned public homebrew SDK and source dependencies, but no proprietary
firmware, trained model, OpenGL SDK or RADV driver. The full-title job also
builds patched RADV, downloads the checksum-verified OpenGL SDK and public
ps5-exporter payload for packaging, and builds presentation assets. No secrets,
pull_request_target, private runners or console are required.
The inspected upstream revision had no checked-in workflow to reuse. This new workflow separates host inference, native-object compilation and full title cross-building. All three remain separate from the hardware gate. See the validation report for local evidence and the PR checks for hosted results.