Distribute and run LLMs with a single file.
C++ cross-platformggufllama-cpplocal-ailocal-inferencelocal-llmopen-source-aisingle-file-executablespeech-to-text
updates to readme
Stephen Hooda89c21dcabc9
diff --git a/README.md b/README.mdindex 04c53985..cac21462 100644--- a/README.md+++ b/README.md@@ -1,11 +1,13 @@# llamafile-llamafile lets you distribute LLMs as a single file.+llamafile lets you distribute and run LLMs with a single file.Our goal is to make the "build once anywhere, run anywhere" dream come-true for AI developers. We're doing that by combining llama.cpp with-Cosmopolitan Libc into one framework that lets you build apps for LLMs-as a single-file artifact that runs locally on most PCs and servers.+true for AI developers. We're doing that by combining [llama.cpp]+(https://github.com/ggerganov/llama.cpp) with [Cosmopolitan Libc]+(https://github.com/jart/cosmopolitan) into one framework that lets+you build apps for LLMs as a single-file artifact that runs locally on+most PCs and servers.First, your llamafiles can run on multiple CPU microarchitectures. Weadded runtime dispatching to llama.cpp that lets new Intel systems use@@ -17,11 +19,12 @@ launches the appropriate one. Our file format is compatible with WIN32and most UNIX shells. It's also able to be easily converted (by eitheryou or your users) to the platform-native format, whenever required.-Thirdly, your llamafiles can run on six OSes. You'll only need to build-your code once, using a Linux-style toolchain. The GCC-based compiler we-provide is itself an Actually Portable Executable, so you can build your-software for all six OSes from the comfort of whichever one you prefer-most for development.+Thirdly, your llamafiles can run on six OSes (macOS, Windows, Linux,+FreeBSD, OpenBSD, and NetBSD). You'll only need to build your code once,+using a Linux-style toolchain. The GCC-based compiler we provide is+itself an Actually Portable Executable, so you can build your software+for all six OSes from the comfort of whichever one you prefer most for+development.Lastly, the weights for your LLM can be embedded within your llamafile.We added support for PKZIP to the GGML library. This lets uncompressed@@ -41,7 +44,7 @@ chmod +x llamafile./llamafile```-On MacOS Arm64 you need to have xcode installed for llamafile to be able+On macOS with Apple Silicon you need to have Xcode installed for llamafile to be ableto bootstrap itself.On Windows, you may need to rename `llamafile` to `llamafile.exe` in@@ -62,7 +65,7 @@ sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/### GPU Support-On Apple Arm64, everything should just work if xcode is installed.+On Apple Silicon, everything should just work if Xcode is installed.On Linux, Nvidia cuBLAS GPU support will be compiled on the fly if (1)you have the `cc` compiler installed, (2) you pass the `--n-gpu-layers@@ -102,8 +105,9 @@ make -j8```Here's an example of how to generate code for a libc function using the-llama.cpp command line interface using WizardCoder weights from Hugging-Face.+llama.cpp command line interface, utilizing [WizardCoder-Python-13B](https://huggingface.co/TheBloke/WizardCoder-Python-13B-V1.0-GGUF)+(license: [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/))+weights.```shmake -j8 o//llama.cpp/main/main@@ -114,8 +118,13 @@ o//llama.cpp/main/main \-p $'```c\nvoid *memcpy_sse2(char *dst, const char *src, size_t size) {\n'```-Here's an example of how to run the HTTP server in such a way that the-weights are embedded inside the executable.+Here's an example of how to run llama.cpp's built-in HTTP server in such+a way that the weights are embedded inside the executable. This example+uses [LLaVA v1.5-7B]() (license: [LLaMA](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md),+[OpenAI](https://openai.com/policies/terms-of-use),+[ShareGPT](https://chrome.google.com/webstore/detail/sharegpt-share-your-chatg/daiacboceoaocpibfodeljbdfacokfjb)),+a multimodal LLM that works with llama.cpp's recently-added support for+image inputs.```shmake -j8@@ -141,7 +150,7 @@ If you want to be able to just say:./server```-And have it run the web server without having to specify arguments (for+...and have it run the web server without having to specify arguments (forthe paths you already know are in there), then you can add a special`.args` to the zip archive, which specifies the default arguments. Inthis case, we're going to try our luck with the normal `zip` command,@@ -172,9 +181,13 @@ mv server.com server./server```-Congratulations. You've just made your own LLM server that's easy to+Congratulations. You've just made your own LLM executable that's easy toshare with your friends.+(Note that the examples provided above are not endorsements or+recommendations of specific models, licenses, or data sets on the part+of Mozilla.)+## zipalign documentation```@@ -251,7 +264,7 @@ shell script into memory. It's possible to avoid this process by runningthe[`assimilate`](https://github.com/jart/cosmopolitan/blob/master/tool/build/assimilate.c)program that comes included with the `cosmocc` compiler. What the-`assimilate` program does, is it turns the shell script executable into+`assimilate` program does is turn the shell script executable intothe host platform's native executable format. This guarantees a fallbackpath exists for traditonal release processes when it's needed.@@ -261,7 +274,7 @@ Cosmopolitan Libc uses static linking, since that's the only way to getthe same executable to run on six OSes. This presents a challenge forllama.cpp, because it's not possible to statically link GPU support. Theway we solve that is by checking if a compiler is installed on the host-system. For Apple, that would be xcode, and for other platforms, that+system. For Apple, that would be Xcode, and for other platforms, thatwould be `nvcc`. llama.cpp has a single file implementation of each GPUmodule, named `ggml-metal.m` (Objective C) and `ggml-cuda.cu` (NvidiaC). llamafile embeds those source files within the zip archive and asks@@ -294,3 +307,10 @@ just trampoline signal handlers and code morph the TLS instructions toavoid these tradeoffs entirely. See[cosmopolitan/dlopen.c](https://github.com/jart/cosmopolitan/blob/master/libc/dlopen/dlopen.c)for further details.++## Licensing++While the llamafile project is Apache 2.0-licensed, our changes+to llama.cpp are licensed under MIT (just like the llama.cpp project+itself) so as to remain compatible and upstreamable in the future,+should that be desired.