llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8
Model market: plan a model from the first bytes of its GGUF
Abhiram9a4d7aa05d39
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f21c826..ec4cde9 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -25,6 +25,9 @@ v0.1.x are the upstream payload releases by cobanov.- Images: a vision model's `mmproj` file beside it is loaded with it(`--mmproj`), and the planner counts its memory.+- The planner can plan from the first bytes of a GGUF (a range request), the+ base of the model market's fit column.+### Changed- One MEMORY tile replaces MEMORY and GPU MEMORY (the GPU and CPU share onepool); CONTEXT takes in the session counts.diff --git a/docs/plans/model-market.md b/docs/plans/model-market.mdindex ba1b0c1..2b49a2c 100644--- a/docs/plans/model-market.md+++ b/docs/plans/model-market.md@@ -31,6 +31,19 @@ download folder (USB by default if it is faster) depends on this.On the console: search finds a model, the page shows its fit, the downloadfinishes with a verified checksum, and the model loads from the library.+## Progress++- Done: the planner plans from the first bytes of a GGUF+ (`read_model_header`), so a 16 MiB range request is enough to show a+ model's fit before downloading it; checked against whole-file plans for+ Llama 3.2 3B, Gemma 4 E4B, Qwen 3.8 27B and Granite 3.3 2B.+- Open, needs the console: HTTPS from the title. The app is built without+ TLS (`LLAMA_OPENSSL=OFF`). Two ways: the system's `libSceHttp2` and+ `libSceSsl` (the SDK has their stubs; whether they resolve in a title is+ unknown), or a TLS library built into the app (mbedTLS) with a CA bundle.+ Try the system library first: no certificates to ship.+- Open, needs the console: the write-speed measurement above.+## Risks- TLS in a title: certificate checks need a CA bundle.diff --git a/ps5/app/model_plan.cpp b/ps5/app/model_plan.cppindex 0c87dba..37c21f5 100644--- a/ps5/app/model_plan.cpp+++ b/ps5/app/model_plan.cpp@@ -129,13 +129,16 @@ std::string find_mmproj(const std::string & model_path) {return best;}-ModelInfo read_model_info(const std::string & path) {+namespace {++// Reads a GGUF header from `f` (and closes it). `file_bytes` is the whole+// file's size when `f` holds only its start; 0 to measure `f`.+ModelInfo parse_model_info(FILE * f, const std::string & path, uint64_t file_bytes) {ModelInfo m;m.path = path;m.file_name = path.substr(path.find_last_of('/') + 1);m.name = m.file_name;- FILE * f = std::fopen(path.c_str(), "rb");if (!f) {m.error = "cannot open";return m;@@ -178,7 +181,7 @@ ModelInfo read_model_info(const std::string & path) {}}std::fseek(f, 0, SEEK_END);- m.file_bytes = (uint64_t) std::ftell(f);+ m.file_bytes = file_bytes ? file_bytes : (uint64_t) std::ftell(f);std::fclose(f);if (r.bad) {m.error = "metadata could not be read";@@ -219,6 +222,13 @@ ModelInfo read_model_info(const std::string & path) {if (!m.ok) {m.error = "no layer count for architecture '" + m.arch + "'";}+ return m;+}++} // namespace++ModelInfo read_model_info(const std::string & path) {+ ModelInfo m = parse_model_info(std::fopen(path.c_str(), "rb"), path, 0);m.mmproj = find_mmproj(path);if (!m.mmproj.empty()) {struct stat st = {};@@ -227,6 +237,12 @@ ModelInfo read_model_info(const std::string & path) {return m;}+ModelInfo read_model_header(const std::string & name, const std::string & head, uint64_t file_bytes) {+ // fmemopen does not copy: `head` outlives the parse.+ FILE * f = head.empty() ? nullptr : fmemopen(const_cast<char *>(head.data()), head.size(), "rb");+ return parse_model_info(f, name, file_bytes);+}+double kv_bytes_per_token(const ModelInfo & m, const std::string & type) {const double per_elem = type == "f16" ? 2.0 : type == "q8_0" ? 34.0 / 32 : 18.0 / 32; // q4_0return (double) m.n_kv_layers() * m.n_head_kv * (m.key_length + m.value_length) * per_elem;diff --git a/ps5/app/model_plan.hpp b/ps5/app/model_plan.hppindex 15a1b51..befbcbc 100644--- a/ps5/app/model_plan.hpp+++ b/ps5/app/model_plan.hpp@@ -39,6 +39,11 @@ struct ModelInfo {};// Reads the metadata of a GGUF file (not its tensors).+// The same from the first bytes of a GGUF file (a range request to a model+// host), with the whole file's size: enough to plan before downloading.+// Fetch enough to hold the metadata (16 MiB covers large vocabularies).+ModelInfo read_model_header(const std::string & name, const std::string & head, uint64_t file_bytes);+// The vision projector ("mmproj") beside a model, or empty.std::string find_mmproj(const std::string & model_path);