Skip to content

armgit / PS5LM

Public Imported from GitHub
0

llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8

Model market: plan a model from the first bytes of its GGUF

Abhiram9a4d7aa05d39
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f21c826..ec4cde9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,6 +25,9 @@ v0.1.x are the upstream payload releases by cobanov.
- Images: a vision model's `mmproj` file beside it is loaded with it
(`--mmproj`), and the planner counts its memory.
+- The planner can plan from the first bytes of a GGUF (a range request), the
+ base of the model market's fit column.
+
### Changed
- One MEMORY tile replaces MEMORY and GPU MEMORY (the GPU and CPU share one
pool); CONTEXT takes in the session counts.
diff --git a/docs/plans/model-market.md b/docs/plans/model-market.md
index ba1b0c1..2b49a2c 100644
--- a/docs/plans/model-market.md
+++ b/docs/plans/model-market.md
@@ -31,6 +31,19 @@ download folder (USB by default if it is faster) depends on this.
On the console: search finds a model, the page shows its fit, the download
finishes with a verified checksum, and the model loads from the library.
+## Progress
+
+- Done: the planner plans from the first bytes of a GGUF
+ (`read_model_header`), so a 16 MiB range request is enough to show a
+ model's fit before downloading it; checked against whole-file plans for
+ Llama 3.2 3B, Gemma 4 E4B, Qwen 3.8 27B and Granite 3.3 2B.
+- Open, needs the console: HTTPS from the title. The app is built without
+ TLS (`LLAMA_OPENSSL=OFF`). Two ways: the system's `libSceHttp2` and
+ `libSceSsl` (the SDK has their stubs; whether they resolve in a title is
+ unknown), or a TLS library built into the app (mbedTLS) with a CA bundle.
+ Try the system library first: no certificates to ship.
+- Open, needs the console: the write-speed measurement above.
+
## Risks
- TLS in a title: certificate checks need a CA bundle.
diff --git a/ps5/app/model_plan.cpp b/ps5/app/model_plan.cpp
index 0c87dba..37c21f5 100644
--- a/ps5/app/model_plan.cpp
+++ b/ps5/app/model_plan.cpp
@@ -129,13 +129,16 @@ std::string find_mmproj(const std::string & model_path) {
return best;
}
-ModelInfo read_model_info(const std::string & path) {
+namespace {
+
+// Reads a GGUF header from `f` (and closes it). `file_bytes` is the whole
+// file's size when `f` holds only its start; 0 to measure `f`.
+ModelInfo parse_model_info(FILE * f, const std::string & path, uint64_t file_bytes) {
ModelInfo m;
m.path = path;
m.file_name = path.substr(path.find_last_of('/') + 1);
m.name = m.file_name;
- FILE * f = std::fopen(path.c_str(), "rb");
if (!f) {
m.error = "cannot open";
return m;
@@ -178,7 +181,7 @@ ModelInfo read_model_info(const std::string & path) {
}
}
std::fseek(f, 0, SEEK_END);
- m.file_bytes = (uint64_t) std::ftell(f);
+ m.file_bytes = file_bytes ? file_bytes : (uint64_t) std::ftell(f);
std::fclose(f);
if (r.bad) {
m.error = "metadata could not be read";
@@ -219,6 +222,13 @@ ModelInfo read_model_info(const std::string & path) {
if (!m.ok) {
m.error = "no layer count for architecture '" + m.arch + "'";
}
+ return m;
+}
+
+} // namespace
+
+ModelInfo read_model_info(const std::string & path) {
+ ModelInfo m = parse_model_info(std::fopen(path.c_str(), "rb"), path, 0);
m.mmproj = find_mmproj(path);
if (!m.mmproj.empty()) {
struct stat st = {};
@@ -227,6 +237,12 @@ ModelInfo read_model_info(const std::string & path) {
return m;
}
+ModelInfo read_model_header(const std::string & name, const std::string & head, uint64_t file_bytes) {
+ // fmemopen does not copy: `head` outlives the parse.
+ FILE * f = head.empty() ? nullptr : fmemopen(const_cast<char *>(head.data()), head.size(), "rb");
+ return parse_model_info(f, name, file_bytes);
+}
+
double kv_bytes_per_token(const ModelInfo & m, const std::string & type) {
const double per_elem = type == "f16" ? 2.0 : type == "q8_0" ? 34.0 / 32 : 18.0 / 32; // q4_0
return (double) m.n_kv_layers() * m.n_head_kv * (m.key_length + m.value_length) * per_elem;
diff --git a/ps5/app/model_plan.hpp b/ps5/app/model_plan.hpp
index 15a1b51..befbcbc 100644
--- a/ps5/app/model_plan.hpp
+++ b/ps5/app/model_plan.hpp
@@ -39,6 +39,11 @@ struct ModelInfo {
};
// Reads the metadata of a GGUF file (not its tensors).
+// The same from the first bytes of a GGUF file (a range request to a model
+// host), with the whole file's size: enough to plan before downloading.
+// Fetch enough to hold the metadata (16 MiB covers large vocabularies).
+ModelInfo read_model_header(const std::string & name, const std::string & head, uint64_t file_bytes);
+
// The vision projector ("mmproj") beside a model, or empty.
std::string find_mmproj(const std::string & model_path);