llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8
llama.cpp: b11827, one patch fewer to carry (#39)
Mert Cobanov8a2da9164a68
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 650b8c4..93b1f68 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -50,6 +50,12 @@ v0.1.x are the upstream payload releases by cobanov.`ps5lm-app-fleet` zip any more: every PS5LM can be a fleet worker, and stays the normal appunless `/data/PS5LM/fleet-worker.txt` asks for one, as before. `PS5LM_FLEET` is gone from`scripts/build-app.sh`, and CI builds one title again.+- llama.cpp moves to b11827 (970b51ce7, 2026-10-22), 219 commits on, among them ggml-vulkan's+ fixes for GPU-visible host memory (#30468) and concurrent first device init (#29766), its strided+ f16 KV copy on AMD (#27703), and upstream's own `tojson` floats as Python writes them (#30362,+ #30483), so `patches/0006-jinja-tojson-floats-as-python.patch` is gone; `patches/0005` is+ re-based on the new `server-http.cpp`, and `patches/0009` (introspection) on the new+ `server-context.cpp`. It builds and links for the console as before.- The planner counts the compute buffers' growth with the prompt batch (`-ub`, about 38 bytes atoken per unit of feed-forward width, measured with ggml-vulkan on the console's shader paths),and the batch for language models is one constant, `kPromptUbatch`, still 64 untildiff --git a/docs/FLEET.md b/docs/FLEET.mdindex d38dde5..953a771 100644--- a/docs/FLEET.md+++ b/docs/FLEET.md@@ -118,7 +118,7 @@ memory. IP validation is not authentication.## Revisions and buildDeveloped on PS5LM `c4595784ce75c1d58777b11ef812d3e087e74296`, with llama.cpp-`04f7226a2c12c7f382c545a6db5a034266aa53cb` and UI+`970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc` and UI`bf44d80384d5b8f8f4f213cf98d748887211328d`. Build host and workers from the samecheckout and patch set. The host manifest checks the local binary checksum andllama revision. A remote handshake checks **protocol compatibility**, not a remotediff --git a/docs/INTROSPECTION.md b/docs/INTROSPECTION.mdindex 0145ed6..e84bf0d 100644--- a/docs/INTROSPECTION.md+++ b/docs/INTROSPECTION.md@@ -18,7 +18,7 @@ The field also works on `/completion` itself, so an existing client only has toadd one key.Implementation: `patches/0009-introspection-logits-and-activations.patch` over-llama.cpp `04f7226`. Logits come from the context's own output buffer+llama.cpp `970b51ce7`. Logits come from the context's own output buffer(`llama_get_logits_ith`), so no work is repeated; activations are copied out ofthe compute graph while it runs, through the graph-eval callback llama.cppalready exposes (`llama_context_params.cb_eval`). Nothing is captured unless adiff --git a/docs/MODELS.md b/docs/MODELS.mdindex f84cc74..ab0a17d 100644--- a/docs/MODELS.md+++ b/docs/MODELS.md@@ -1,6 +1,6 @@# Models-Models worth running on a PS5, from what is popular on Ollama in October 2026, with the GGUF sizes on Hugging Face. Every architecture below loads in the pinned llama.cpp (`b11327`).+Models worth running on a PS5, from what is popular on Ollama in October 2026, with the GGUF sizes on Hugging Face. Every architecture below loads in the pinned llama.cpp (`b11827`).The console has 16 GB shared by everything. How much one PS5LM process can hold is what `probes/memprobe` measures. Until then: ProsperoAI already keeps a 6.37 GiB model resident on this console, so up to about 6 GiB is safe, and anything above 8 GiB waits for the probe.diff --git a/docs/PORTING.md b/docs/PORTING.mdindex 4ab1a22..862b023 100644--- a/docs/PORTING.md+++ b/docs/PORTING.md@@ -1,6 +1,6 @@# Porting notes-What it took to build llama.cpp for the PS5, and why. Upstream is pinned at `b11327` (2026-10-01).+What it took to build llama.cpp for the PS5, and why. Upstream is pinned at `b11827` (2026-10-22).## The targetdiff --git a/fleet/controller.py b/fleet/controller.pyindex 16958a7..64505b3 100644--- a/fleet/controller.py+++ b/fleet/controller.py@@ -19,7 +19,7 @@ import sysfrom .models import ModelError, fetch_model, inspect_filefrom .planner import FleetError, MIB, inventory, plan-LLAMA_REVISION = '04f7226a2c12c7f382c545a6db5a034266aa53cb'+LLAMA_REVISION = '970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc'ROOT = Path(__file__).resolve().parents[1]DEFAULT_BINARY = ROOT / 'build/fleet-host/bin/llama-server'diff --git a/patches/0005-listener-survives-network-drops.patch b/patches/0005-listener-survives-network-drops.patchindex e5decc4..2730849 100644--- a/patches/0005-listener-survives-network-drops.patch+++ b/patches/0005-listener-survives-network-drops.patch@@ -1,5 +1,5 @@diff --git a/tools/server/server-http.cpp b/tools/server/server-http.cpp-index 33cbb156c..7ac78f9ad 100644+index cf147c7e7..a5f26d018 100644--- a/tools/server/server-http.cpp+++ b/tools/server/server-http.cpp@@ -10,6 +10,7 @@@@ -8,9 +8,9 @@ index 33cbb156c..7ac78f9ad 100644#include <string>+#include <atomic>#include <thread>+ #include <mutex>- //-@@ -23,6 +24,7 @@ public:+@@ -24,6 +25,7 @@ public:std::vector<std::thread> threads; // one thread per listenerstd::unique_ptr<httplib::ThreadPool> pool; // single pool shared among all listenersint n_threads_http = 0;@@ -18,7 +18,7 @@ index 33cbb156c..7ac78f9ad 100644};class server_http_task_queue : public httplib::TaskQueue {-@@ -528,9 +530,23 @@ bool server_http_context::start() {+@@ -529,9 +531,23 @@ bool server_http_context::start() {pimpl->pool = std::make_unique<httplib::ThreadPool>(pimpl->n_threads_http, pimpl->n_threads_http + 1024);for (size_t i = 0; i < pimpl->servers.size(); ++i) {const auto & srv = pimpl->servers[i];@@ -45,7 +45,7 @@ index 33cbb156c..7ac78f9ad 100644}});srv->wait_until_ready();-@@ -546,6 +562,7 @@ bool server_http_context::start() {+@@ -547,6 +563,7 @@ bool server_http_context::start() {}void server_http_context::stop() const {diff --git a/patches/0006-jinja-tojson-floats-as-python.patch b/patches/0006-jinja-tojson-floats-as-python.patchdeleted file mode 100644index d2c138d..0000000--- a/patches/0006-jinja-tojson-floats-as-python.patch+++ /dev/null@@ -1,69 +0,0 @@-diff --git a/common/jinja/value.cpp b/common/jinja/value.cpp-index 6edd8b9..d467942 100644---- a/common/jinja/value.cpp-+++ b/common/jinja/value.cpp-@@ -5,6 +5,8 @@- // for converting from JSON to jinja values- #include "json.h"--+#include <charconv>-+#include <cmath>- #include <sstream>- #include <string>- #include <cctype>-@@ -1519,6 +1521,46 @@ void global_from_json(context & ctx, const common_json & json_obj, bool mark_inp-- // recursively convert value to JSON string- // TODO: avoid circular references-+// A float as Python's json.dumps writes it (repr): the shortest digits that read back the same-+// value, fixed notation for exponents -4 to 15 with ".0" on whole numbers, else 1e+16 style.-+// ostream's default (6 significant digits) wrote 0.0 as 0 and 1234567.89 as 1.23457e+06.-+static std::string float_to_json(double v) {-+ if (std::isnan(v)) {-+ return "NaN";-+ }-+ if (std::isinf(v)) {-+ return v > 0 ? "Infinity" : "-Infinity";-+ }-+ char buf[64];-+ const auto res = std::to_chars(buf, buf + sizeof(buf), v, std::chars_format::scientific);-+ const std::string sci(buf, res.ptr); // [-]d[.ddd]e(+|-)dd-+ const size_t e_pos = sci.find('e');-+ const bool neg = sci[0] == '-';-+ std::string digits;-+ for (size_t i = neg ? 1 : 0; i < e_pos; i++) {-+ if (sci[i] != '.') {-+ digits += sci[i];-+ }-+ }-+ const int exp = std::stoi(sci.substr(e_pos + 1));-+ std::string out = neg ? "-" : "";-+ if (exp >= -4 && exp < 16) {-+ if (exp < 0) {-+ out += "0." + std::string(-exp - 1, '0') + digits;-+ } else {-+ if (digits.size() <= (size_t) exp + 1) {-+ digits.append(exp + 1 - digits.size(), '0');-+ }-+ const std::string frac = digits.substr(exp + 1);-+ out += digits.substr(0, exp + 1) + "." + (frac.empty() ? "0" : frac);-+ }-+ } else {-+ out += digits.substr(0, 1) + (digits.size() > 1 ? "." + digits.substr(1) : "");-+ out += (exp < 0 ? "e-" : "e+") + std::string(std::abs(exp) < 10 ? "0" : "") + std::to_string(std::abs(exp));-+ }-+ return out;-+}-+- static void value_to_json_internal(std::ostringstream & oss, const value & val, int curr_lvl, int indent, const std::string_view item_sep, const std::string_view key_sep) {- auto indent_str = [indent, curr_lvl]() -> std::string {- return (indent > 0) ? std::string(curr_lvl * indent, ' ') : "";-@@ -1534,7 +1576,7 @@ static void value_to_json_internal(std::ostringstream & oss, const value & val,- } else if (is_val<value_int>(val)) {- oss << val->as_int();- } else if (is_val<value_float>(val)) {-- oss << val->as_float();-+ oss << float_to_json(val->as_float());- } else if (is_val<value_string>(val)) {- oss << "\"";- for (char c : val->as_string().str()) {diff --git a/patches/0009-introspection-logits-and-activations.patch b/patches/0009-introspection-logits-and-activations.patchindex 6ffa8de..44dc433 100644--- a/patches/0009-introspection-logits-and-activations.patch+++ b/patches/0009-introspection-logits-and-activations.patch@@ -1,5 +1,5 @@diff --git a/tools/server/CMakeLists.txt b/tools/server/CMakeLists.txt-index 70b8d7d..7da0f33 100644+index e0b9d0ec1..7cef7a7ee 100644--- a/tools/server/CMakeLists.txt+++ b/tools/server/CMakeLists.txt@@ -25,6 +25,8 @@ add_library(${TARGET} STATIC@@ -12,10 +12,10 @@ index 70b8d7d..7da0f33 100644if (BUILD_SHARED_LIBS)diff --git a/tools/server/server-context.cpp b/tools/server/server-context.cpp-index 60c5633..c785782 100644+index a0acf1655..6dba147c4 100644--- a/tools/server/server-context.cpp+++ b/tools/server/server-context.cpp-@@ -292,6 +292,13 @@ struct server_slot {+@@ -325,6 +325,13 @@ struct server_slot {std::vector<completion_token_output> generated_token_probs;@@ -29,7 +29,7 @@ index 60c5633..c785782 100644bool has_next_token = true;bool has_new_line = false;bool truncated = false;-@@ -397,6 +404,13 @@ struct server_slot {+@@ -435,6 +442,13 @@ struct server_slot {generated_token_probs.clear();json_schema = json();@@ -43,7 +43,7 @@ index 60c5633..c785782 100644task_prev = std::move(task);task.reset();-@@ -953,6 +967,9 @@ public:+@@ -1023,6 +1037,9 @@ public:server_decision_context decision;@@ -53,7 +53,7 @@ index 60c5633..c785782 100644server_queue queue_tasks;server_response queue_results;-@@ -1019,6 +1036,14 @@ private:+@@ -1089,6 +1106,14 @@ private:int slots_debug = 0; // env: LLAMA_SERVER_SLOTS_DEBUGint slots_n_diff = 0; // env: LLAMA_SERVER_SLOTS_N_DIFF@@ -68,7 +68,7 @@ index 60c5633..c785782 100644int n_empty_consecutive = 0;std::unique_ptr<server_prompt_cache> prompt_cache;-@@ -1213,11 +1238,33 @@ private:+@@ -1284,11 +1309,33 @@ private:params_base.load_progress_callback_user_data = &load_progress_text;}@@ -103,7 +103,7 @@ index 60c5633..c785782 100644if (model_tgt == nullptr) {SRV_ERR("failed to load model, '%s'\n", params_base.model.path.c_str());return false;-@@ -1230,6 +1277,13 @@ private:+@@ -1301,6 +1348,13 @@ private:vocab = llama_model_get_vocab(model_tgt);@@ -117,7 +117,7 @@ index 60c5633..c785782 100644try {decision.init(model_tgt);} catch (const std::exception & e) {-@@ -2290,6 +2344,27 @@ private:+@@ -2411,6 +2465,27 @@ private:res->generation_params = slot.task->params; // copy the parameters@@ -145,7 +145,7 @@ index 60c5633..c785782 100644queue_results.send(std::move(res));}-@@ -3231,13 +3306,35 @@ private:+@@ -3513,13 +3588,35 @@ private:int32_t off_next = 0;int32_t n_batch = llama_n_batch(ctx_tgt);for (int32_t off = 0; off < batch.size(); off = off_next) {@@ -183,7 +183,7 @@ index 60c5633..c785782 100644#ifdef DEBUG_TIMINGSllama_synchronize(ctx_tgt);#endif-@@ -3978,13 +4075,21 @@ private:+@@ -4353,13 +4450,21 @@ private:break;}@@ -206,7 +206,7 @@ index 60c5633..c785782 100644/* is_prompt = */ true);if (!slot.task->decision.order.empty()) {batch.set_decision_order(batch.size() - 1, slot.task->decision.order[slot.prompt.n_tokens()]);-@@ -4286,6 +4391,159 @@ private:+@@ -4661,6 +4766,159 @@ private:return true;}@@ -366,7 +366,7 @@ index 60c5633..c785782 100644void post_decode(int32_t n_batch_tokens, int32_t off) {// for checking if a given batch index is inside the current sub-batchauto is_inside_view = [&](int32_t idx) {-@@ -4842,6 +5100,34 @@ std::unique_ptr<server_res_generator> server_routes::handle_completions_impl(+@@ -5230,6 +5488,34 @@ std::unique_ptr<server_res_generator> server_routes::handle_completions_impl(task.apply_chat_session(chat_session);@@ -401,7 +401,7 @@ index 60c5633..c785782 100644task.id_slot = json_value(data, "id_slot", -1);sse_ping_interval = task.params.sse_ping_interval;-@@ -5690,6 +5976,49 @@ void server_routes::init_routes() {+@@ -6283,6 +6569,49 @@ void server_routes::init_routes() {return handle_embeddings_impl(req, TASK_RESPONSE_TYPE_OAI_EMBD);};@@ -449,15 +449,15 @@ index 60c5633..c785782 100644+ };+this->post_rerank = [this](const server_http_req & req) {- auto res = create_response();- if (!params.embedding || params.pooling_type != LLAMA_POOLING_TYPE_RANK) {+ return handle_rerank_impl(req);+ };diff --git a/tools/server/server-context.h b/tools/server/server-context.h-index ed23f7d..d2ae01d 100644+index 4a8e327fc..3d2637a32 100644--- a/tools/server/server-context.h+++ b/tools/server/server-context.h-@@ -154,6 +154,8 @@ struct server_routes {- server_http_context::handler_t post_embeddings_oai;+@@ -159,6 +159,8 @@ struct server_routes {server_http_context::handler_t post_rerank;+ server_http_context::handler_t post_tts;server_http_context::handler_t post_systemone;+ // PS5LM fork: raw logits and per-layer activations+ server_http_context::handler_t post_introspect;@@ -466,7 +466,7 @@ index ed23f7d..d2ae01d 100644diff --git a/tools/server/server-introspect.cpp b/tools/server/server-introspect.cppnew file mode 100644-index 0000000..af9c2ee+index 000000000..af9c2ee80--- /dev/null+++ b/tools/server/server-introspect.cpp@@ -0,0 +1,624 @@@@ -1096,7 +1096,7 @@ index 0000000..af9c2ee+}diff --git a/tools/server/server-introspect.h b/tools/server/server-introspect.hnew file mode 100644-index 0000000..a80f7e1+index 000000000..a80f7e1d7--- /dev/null+++ b/tools/server/server-introspect.h@@ -0,0 +1,205 @@@@ -1306,7 +1306,7 @@ index 0000000..a80f7e1+ }+};diff --git a/tools/server/server-schema.cpp b/tools/server/server-schema.cpp-index 54e126a..30efbbb 100644+index 1e2dce888..a3609b179 100644--- a/tools/server/server-schema.cpp+++ b/tools/server/server-schema.cpp@@ -187,6 +187,17 @@ std::vector<std::unique_ptr<field>> make_llama_cmpl_schema(const common_params &@@ -1328,7 +1328,7 @@ index 54e126a..30efbbb 100644->set_desc("Return probabilities of top n_probs tokens after applying the sampling chain"));diff --git a/tools/server/server-task.cpp b/tools/server/server-task.cpp-index 3f30da2..542d559 100644+index d7ac2a1dc..180984684 100644--- a/tools/server/server-task.cpp+++ b/tools/server/server-task.cpp@@ -351,6 +351,10 @@ json server_task_result_cmpl_final::to_json_non_oaicompat() {@@ -1343,18 +1343,18 @@ index 3f30da2..542d559 100644}diff --git a/tools/server/server-task.h b/tools/server/server-task.h-index 50926ec..93b29ec 100644+index 810cefbfe..97d31fd9c 100644--- a/tools/server/server-task.h+++ b/tools/server/server-task.h-@@ -10,6 +10,7 @@-+@@ -11,6 +11,7 @@// TODO: prevent including the whole server-common.h as we only use server_tokens#include "server-common.h"+ #include "mtmd-helper.h"+#include "server-introspect.h" // PS5LM fork: introspection request/response typesenum server_task_type {-@@ -99,6 +100,12 @@ struct task_params {+@@ -111,6 +112,12 @@ struct task_params {// Embeddingsint32_t embd_normalize = 2; // (-1=none, 0=max absolute int16, 1=taxicab, 2=Euclidean/L2, >2=p-norm)@@ -1367,7 +1367,7 @@ index 50926ec..93b29ec 100644json format_logit_bias(const std::vector<llama_logit_bias> & logit_bias) const;json to_json(bool only_metrics = false) const;};-@@ -384,6 +391,9 @@ struct server_task_result_cmpl_final : server_task_result {+@@ -407,6 +414,9 @@ struct server_task_result_cmpl_final : server_task_result {task_params generation_params;@@ -1378,10 +1378,10 @@ index 50926ec..93b29ec 100644bool verbose = false;task_response_type res_type = TASK_RESPONSE_TYPE_NONE;diff --git a/tools/server/server.cpp b/tools/server/server.cpp-index 871e827..12dd9bd 100644+index ce44405b8..ba39d0e63 100644--- a/tools/server/server.cpp+++ b/tools/server/server.cpp-@@ -241,6 +241,8 @@ int llama_server(common_params & params, int argc, char ** argv, server_child &+@@ -258,6 +258,8 @@ int llama_server(common_params & params, int argc, char ** argv, server_child &routes.post_systemone = models_routes->proxy_post;routes.post_tokenize = models_routes->proxy_post;routes.post_detokenize = models_routes->proxy_post;@@ -1390,7 +1390,7 @@ index 871e827..12dd9bd 100644routes.post_apply_template = models_routes->proxy_post;routes.post_chat_completions_tok = models_routes->proxy_post;routes.post_responses_tok_oai = models_routes->proxy_post;-@@ -289,6 +291,9 @@ int llama_server(common_params & params, int argc, char ** argv, server_child &+@@ -308,6 +310,9 @@ int llama_server(common_params & params, int argc, char ** argv, server_child &ctx_http.post("/v1/systemone", ex_wrapper(routes.post_systemone));ctx_http.post("/tokenize", ex_wrapper(routes.post_tokenize));ctx_http.post("/detokenize", ex_wrapper(routes.post_detokenize));diff --git a/tests/fleet_rpc_no_comm_test.py b/tests/fleet_rpc_no_comm_test.pyindex 9645ef6..75e07e4 100644--- a/tests/fleet_rpc_no_comm_test.py+++ b/tests/fleet_rpc_no_comm_test.py@@ -67,7 +67,8 @@ def run(binary):time.sleep(0.05)sock.settimeout(1)hello = request(sock, 14, bytes(24), 28)- assert hello[:3] == bytes([8, 0, 0]), hello[:3]+ # Only the major version breaks compatibility; upstream bumps the patch per new ggml op.+ assert hello[0] == 8, hello[:3]for rank in (0, 1):body = struct.pack('<IIII64s', 0, rank, 2, comm_port, b'127.0.0.1')assert request(sock, 20, body, 1) == b'\0', f'rank {rank} was not refused'diff --git a/third_party/llama.cpp b/third_party/llama.cppindex 04f7226..970b51c 160000--- a/third_party/llama.cpp+++ b/third_party/llama.cpp@@ -1 +1 @@-Subproject commit 04f7226a2c12c7f382c545a6db5a034266aa53cb+Subproject commit 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc