Distribute and run LLMs with a single file.
C++ cross-platformggufllama-cpplocal-ailocal-inferencelocal-llmopen-source-aisingle-file-executablespeech-to-text
Avoid bind() conflicts on port 8080 w/ server
Justine Tunneyc1c8683d3782
diff --git a/llama.cpp/README.llamafile b/llama.cpp/README.llamafileindex 4efc2136..e5ee79f5 100644--- a/llama.cpp/README.llamafile+++ b/llama.cpp/README.llamafile@@ -33,6 +33,7 @@ LOCAL MODIFICATIONS- Introduce pledge() SECCOMP sandboxing to improve security- Introduce ggml_interrupt() to gracefully handle signals- Remove log callback pointer API from Metal GPU module+ - Avoid bind() conflicts on port 8080 w/ server- Allow --grammar to be used on --image prompts- Use runtime dispatching for matmul quants- Remove operating system #ifdef statementsdiff --git a/llama.cpp/server/server.cpp b/llama.cpp/server/server.cppindex 172e4a26..1a3d66ec 100644--- a/llama.cpp/server/server.cpp+++ b/llama.cpp/server/server.cpp@@ -2994,10 +2994,15 @@ int main(int argc, char **argv)svr.set_read_timeout (sparams.read_timeout);svr.set_write_timeout(sparams.write_timeout);- if (!svr.bind_to_port(sparams.hostname, sparams.port))- {- fprintf(stderr, "\ncouldn't bind to server socket: hostname=%s port=%d\n\n", sparams.hostname.c_str(), sparams.port);- return 1;+ for (int i = 0;; ++i) {+ if (svr.bind_to_port(sparams.hostname, sparams.port)) {+ break;+ } else if (i < 10) {+ ++sparams.port;+ } else {+ fprintf(stderr, "\ncouldn't bind to server socket: hostname=%s port=%d\n\n", sparams.hostname.c_str(), sparams.port);+ return 1;+ }}// Set the base directory for serving static files