llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8
Chat with a model in the PS5's own browser
Mert Cobanov65667a23077f
diff --git a/README.md b/README.mdindex 09fc1df..55eca3c 100644--- a/README.md+++ b/README.md@@ -4,7 +4,7 @@The first target is **Qwen 3.8** (27B, hybrid Gated DeltaNet attention), the newest open Qwen. No one has run it on a PS5 yet.-> **Status:** llama.cpp runs on a PS5. Qwen3.5 0.8B generates 14 tokens/s on two of the console's CPU cores (firmware 13.60, PS5 Slim). Next: the chat UI in the PS5's browser, then bigger models. See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CONSOLE.md](docs/CONSOLE.md).+> **Status:** llama.cpp runs on a PS5. Qwen3.5 0.8B generates 14 tokens/s on two of the console's CPU cores (firmware 13.60, PS5 Slim). There is a chat page in the PS5's own browser (`scripts/ps5-chat.sh`). Next: bigger models. See [docs/ROADMAP.md](docs/ROADMAP.md) and [docs/CONSOLE.md](docs/CONSOLE.md).## Why llama.cppdiff --git a/docs/CONSOLE.md b/docs/CONSOLE.mdindex 4c4109e..aeae680 100644--- a/docs/CONSOLE.md+++ b/docs/CONSOLE.md@@ -41,6 +41,15 @@ First run, Qwen3.5 0.8B Q4_K_M, `-t 2 --poll 0 -lm none`, 512 context:`llama-server` loads Granite 4.2 3B (2.1 GiB) with mmap, so mapping a model file works in a payload.+Chat through `llama-server` and `ps5/webui` in the PS5's browser (browser open, `-c 4096 -np 1 -lm none`, 3 threads from the safe defaults):++| Model | Prompt | Generation |+|---|---|---|+| Qwen3.5 0.8B Q4_K_M, thinking off | 67 tok/s | 13.1 tok/s |+| Granite 4.2 3B Q4_K_M | 0.6 tok/s | 0.2 tok/s |++Granite is 4 times bigger but 65 times slower: something in that architecture's CPU path, to look into. llama.cpp's own web UI renders blank in the PS5 browser; `ps5/webui` (plain ES5) works.+At exit, Sony's libc prints `[SceLibc] A heap error is detected` (SceLibcInternalHeap) after all work is done. Not investigated yet.## shsrv quirksdiff --git a/docs/ROADMAP.md b/docs/ROADMAP.mdindex cd3f408..2c36e93 100644--- a/docs/ROADMAP.md+++ b/docs/ROADMAP.md@@ -43,7 +43,7 @@ Decoding is bound by memory bandwidth: tokens per second ≈ effective bandwidth- [x] `llama-cli`, `llama-server` and `llama-bench` link ([PORTING.md](PORTING.md) lists the three build fixes)- [x] Smoke test: Qwen3.5 0.8B generates text on the PS5 CPU, 14.2 tok/s on two threads- [x] Safe CPU defaults for the PS5 (`patches/0001`): threads = allowed CPUs minus two, no busy-waiting-- [ ] `llama-server` on the console: chat from a laptop or phone with llama.cpp's built-in web UI (port 8081, since websrv takes 8080)+- [x] `llama-server` on the console with a chat page in the PS5's own browser (`scripts/ps5-chat.sh`, `ps5/webui`); phones and laptops use the same page on port 8081- [ ] `llama-bench` numbers for the small model- [ ] **First light: Qwen3.8-27B generates text on the PS5.** UD-Q2_K_XL if Phase 0 says it fits, UD-IQ2_XXS if not. A video of it, posted with the repo.diff --git a/ps5/webui/index.html b/ps5/webui/index.htmlnew file mode 100644index 0000000..1ffe403--- /dev/null+++ b/ps5/webui/index.html@@ -0,0 +1,174 @@+<!doctype html>+<html lang="en">+<head>+<meta charset="utf-8">+<meta name="viewport" content="width=device-width, initial-scale=1">+<title>PS5LM</title>+<!--+ The chat page llama-server serves on the PS5 (llama-server --path www).+ llama.cpp's own web UI renders blank in the PS5 browser, so this page sticks+ to plain ES5 and XMLHttpRequest, with no external files. Sized for a TV.+ SPDX-License-Identifier: GPL-3.0-or-later+-->+<style>+ :root {+ --bg: #0b0d12;+ --panel: #151923;+ --text: #e8ebf2;+ --muted: #8a93a6;+ --accent: #3d7bfd;+ --user: #1f2a44;+ --border: #262c3a;+ }+ html, body { margin: 0; height: 100%; background: var(--bg); color: var(--text);+ font-family: -apple-system, "Segoe UI", Helvetica, Arial, sans-serif; }+ body { display: flex; flex-direction: column; }+ header { display: flex; align-items: baseline; gap: 16px; padding: 18px 32px;+ border-bottom: 1px solid var(--border); }+ header h1 { margin: 0; font-size: 30px; letter-spacing: 0.5px; }+ header span { color: var(--muted); font-size: 20px; }+ #log { flex: 1; overflow-y: auto; padding: 24px 32px; }+ .msg { max-width: 1100px; margin: 0 0 18px; padding: 16px 22px; border-radius: 14px;+ font-size: 26px; line-height: 1.45; white-space: pre-wrap; word-wrap: break-word; }+ .user { background: var(--user); margin-left: auto; }+ .bot { background: var(--panel); border: 1px solid var(--border); }+ .think { color: var(--muted); font-style: italic; font-size: 22px; }+ .meta { color: var(--muted); font-size: 18px; margin-top: 8px; }+ form { display: flex; gap: 14px; padding: 18px 32px 26px; border-top: 1px solid var(--border); }+ #input { flex: 1; font-size: 26px; padding: 16px 18px; border-radius: 12px;+ border: 1px solid var(--border); background: var(--panel); color: var(--text); }+ button { font-size: 24px; padding: 0 30px; border-radius: 12px; border: 0;+ background: var(--accent); color: #fff; }+ button.stop { background: #a83a3a; }+</style>+</head>+<body>+<header><h1>PS5LM</h1><span id="model">llama.cpp on PlayStation 5</span></header>+<div id="log"></div>+<form id="form">+ <input id="input" type="text" autocomplete="off" placeholder="Type a message">+ <button id="send" type="submit">Send</button>+</form>+<script>+(function () {+ var log = document.getElementById('log');+ var form = document.getElementById('form');+ var input = document.getElementById('input');+ var send = document.getElementById('send');+ var SYSTEM = 'You are a helpful assistant running on a PlayStation 5. ' ++ 'Answer in one to three short sentences, in the language the user writes in.';+ var messages = [];+ var xhr = null;++ function bubble(cls, text) {+ var div = document.createElement('div');+ div.className = 'msg ' + cls;+ div.textContent = text || '';+ log.appendChild(div);+ log.scrollTop = log.scrollHeight;+ return div;+ }++ function busy(on) {+ send.textContent = on ? 'Stop' : 'Send';+ send.className = on ? 'stop' : '';+ }++ // The model name, for the header.+ var info = new XMLHttpRequest();+ info.open('GET', '/v1/models');+ info.onload = function () {+ try {+ var id = JSON.parse(info.responseText).data[0].id;+ document.getElementById('model').textContent = id.split('/').pop() + ' on PlayStation 5';+ } catch (e) {}+ };+ info.send();++ function ask(text) {+ messages.push({ role: 'user', content: text });+ // Keep the request small: the console's context is 4096 tokens.+ if (messages.length > 12) messages = messages.slice(messages.length - 12);+ bubble('user', text);+ var out = bubble('bot', '');+ var think = null;+ var answer = '';+ var reasoning = '';+ var seen = 0;+ var pending = '';+ var started = Date.now();++ xhr = new XMLHttpRequest();+ xhr.open('POST', '/v1/chat/completions');+ xhr.setRequestHeader('Content-Type', 'application/json');++ // Server-sent events arrive in pieces; parse whole "data:" lines as they come.+ xhr.onprogress = function () {+ pending += xhr.responseText.substring(seen);+ seen = xhr.responseText.length;+ var lines = pending.split('\n');+ pending = lines.pop();+ for (var i = 0; i < lines.length; i++) {+ var line = lines[i].replace(/^\s+|\s+$/g, '');+ if (line.indexOf('data:') !== 0) continue;+ var data = line.substring(5).replace(/^\s+/, '');+ if (data === '[DONE]') continue;+ var chunk;+ try { chunk = JSON.parse(data); } catch (e) { continue; }+ var delta = chunk.choices && chunk.choices[0] && chunk.choices[0].delta;+ if (delta && delta.reasoning_content) {+ if (!think) { think = document.createElement('div'); think.className = 'think'; out.appendChild(think); }+ reasoning += delta.reasoning_content;+ think.textContent = reasoning;+ }+ if (delta && delta.content) {+ answer += delta.content;+ if (!out.lastChild || out.lastChild.nodeType !== 3) out.appendChild(document.createTextNode(''));+ out.lastChild.nodeValue = answer;+ }+ if (chunk.timings) {+ var meta = document.createElement('div');+ meta.className = 'meta';+ meta.textContent = chunk.timings.predicted_n + ' tokens, ' ++ chunk.timings.predicted_per_second.toFixed(1) + ' tok/s, ' ++ ((Date.now() - started) / 1000).toFixed(1) + ' s';+ out.appendChild(meta);+ }+ log.scrollTop = log.scrollHeight;+ }+ };++ xhr.onloadend = function () {+ xhr.onprogress();+ if (xhr.status && xhr.status !== 200 && !answer) out.textContent = 'Error ' + xhr.status + ': ' + xhr.responseText;+ if (answer) messages.push({ role: 'assistant', content: answer });+ xhr = null;+ busy(false);+ input.focus();+ };++ // Small models on four CPU cores: short answers, no thinking phase.+ xhr.send(JSON.stringify({+ messages: [{ role: 'system', content: SYSTEM }].concat(messages),+ stream: true,+ max_tokens: 256,+ chat_template_kwargs: { enable_thinking: false }+ }));+ busy(true);+ }++ form.onsubmit = function (e) {+ e.preventDefault();+ if (xhr) { xhr.abort(); return; }+ var text = input.value.replace(/^\s+|\s+$/g, '');+ if (!text) return;+ input.value = '';+ ask(text);+ };++ bubble('bot', 'Hi! I am a language model running on this PlayStation 5, on its own CPU. Ask me anything.');+ input.focus();+})();+</script>+</body>+</html>diff --git a/scripts/ps5-chat.sh b/scripts/ps5-chat.shnew file mode 100755index 0000000..028a7a7--- /dev/null+++ b/scripts/ps5-chat.sh@@ -0,0 +1,53 @@+#!/usr/bin/env bash+# Starts llama-server on the console and opens its chat UI in the PS5's own+# browser. Needs ftpsrv (2121) and shsrv (2323) running on the console, and the+# model already in /data/PS5LM/models.+#+# PS5_HOST=192.168.8.52 scripts/ps5-chat.sh [model.gguf]+set -euo pipefail++root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"+host=${PS5_HOST:?set PS5_HOST to the console IP}+model=${1:-granite3b.gguf}+port=${PS5LM_PORT:-8081}+logdir="$root/build/console"+mkdir -p "$logdir"++# Upload the current build and the chat page. shsrv needs absolute paths and+# short command lines (keep model file names short).+curl -sS --disable-epsv --ftp-create-dirs -T "$root/build/llama-ps5/bin/llama-server" \+ "ftp://$host:2121/data/PS5LM/bin/llama-server"+curl -sS --disable-epsv --ftp-create-dirs -T "$root/ps5/webui/index.html" \+ "ftp://$host:2121/data/PS5LM/www/index.html"++# Keep the shell session that runs the server open in the background; its output+# goes to build/console/server.log. The context is capped: the model's own+# default (131K for Granite) would not fit in a payload's ~6 GiB. The model is+# read into memory (-lm none): mapped from the file, its pages get evicted once+# the browser opens, and generation drops to a fraction of a token a second.+# --path www serves ps5/webui instead of llama.cpp's UI, which the PS5 browser+# renders blank.+nohup bash -c "+ { sleep 1+ printf 'chmod 755 /data/PS5LM/bin/llama-server\n'; sleep 1+ printf 'cd /data/PS5LM\n'; sleep 1+ printf '/data/PS5LM/bin/llama-server -m models/$model -c 4096 -np 1 -lm none --path www --host 0.0.0.0 --port $port\n'+ sleep 86400+ } | nc $host 2323+" > "$logdir/server.log" 2>&1 &+echo "ps5-chat: llama-server starting (log: $logdir/server.log)"++for _ in $(seq 1 90); do+ sleep 2+ code=$(curl -s -m 3 -o /dev/null -w "%{http_code}" "http://$host:$port/health" || true)+ [ "$code" = "200" ] && break+done+if [ "$code" != "200" ]; then+ echo "ps5-chat: server not healthy (last status $code), see $logdir/server.log" >&2+ exit 1+fi+echo "ps5-chat: server ready"++{ sleep 1; printf 'browse http://127.0.0.1:%s/\n' "$port"; sleep 3; printf 'exit\n'; sleep 1; } \+ | nc -w 5 "$host" 2323 > /dev/null 2>&1 || true+echo "ps5-chat: chat UI opened in the PS5 browser; from another device: http://$host:$port/"