Distribute and run LLMs with a single file.
C++ cross-platformggufllama-cpplocal-ailocal-inferencelocal-llmopen-source-aisingle-file-executablespeech-to-text
Bring new server up to 2421 embedding/sec
Justine Tunney070aa13acfca
diff --git a/llama.cpp/main/main.cpp b/llama.cpp/main/main.cppindex cf60335e..cad9c765 100644--- a/llama.cpp/main/main.cpp+++ b/llama.cpp/main/main.cpp@@ -147,6 +147,10 @@ static std::string chat_add_and_format(struct llama_model * model, std::vector<lint main(int argc, char ** argv) {+ mallopt(M_GRANULARITY, 2 * 1024 * 1024);+ mallopt(M_MMAP_THRESHOLD, 16 * 1024 * 1024);+ mallopt(M_TRIM_THRESHOLD, 128 * 1024 * 1024);+if (llamafile_has(argv, "--version")) {puts("llamafile v" LLAMAFILE_VERSION_STRING);return 0;diff --git a/llamafile/server/doc/technical_details.md b/llamafile/server/doc/technical_details.mdindex 9885a520..84f06327 100644--- a/llamafile/server/doc/technical_details.md+++ b/llamafile/server/doc/technical_details.md@@ -9,7 +9,7 @@ technical aspects of LLaMAfiler.This server can serve embeddings an order of a magnitude faster than thellama.cpp example server. We measured this on Threadripper PRO 7995WX w/all-MiniLM-L6-v2 Q6\_K. Using a prompt with 50 tokens this server served-2,015 `/embedding` requests per second, whereas+2,427 `/embedding` requests per second, whereasllama.cpp/build/bin/server built with glibc served 771 per second.The following wrk script was used:@@ -34,16 +34,16 @@ The results for llamafiler were:```Thread Stats Avg Stdev Max +/- Stdev- Latency 15.81ms 2.73ms 30.86ms 71.49%- Req/Sec 63.08 7.41 90.00 84.66%+ Latency 13.12ms 2.95ms 32.75ms 87.75%+ Req/Sec 75.85 8.86 101.00 76.56%Latency Distribution- 50% 15.65ms- 75% 17.46ms- 90% 19.18ms- 99% 23.26ms- 4441 requests in 2.20s, 23.04MB read-Requests/sec: 2015.82-Transfer/sec: 10.46MB+ 50% 12.66ms+ 75% 13.54ms+ 90% 15.29ms+ 99% 25.43ms+ 2427 requests in 1.00s, 12.58MB read+Requests/sec: 2420.96+Transfer/sec: 12.55MB```The results for llama.cpp in CPU mode were:diff --git a/llamafile/server/main.cpp b/llamafile/server/main.cppindex 8293058b..04f60443 100644--- a/llamafile/server/main.cpp+++ b/llamafile/server/main.cpp@@ -43,6 +43,10 @@ main(int argc, char* argv[])exit(0);}+ mallopt(M_GRANULARITY, 2 * 1024 * 1024);+ mallopt(M_MMAP_THRESHOLD, 16 * 1024 * 1024);+ mallopt(M_TRIM_THRESHOLD, 128 * 1024 * 1024);+// get configLoadZipArgs(&argc, &argv);llamafile_get_flags(argc, argv);