Distribute and run LLMs with a single file.
C++ cross-platformggufllama-cpplocal-ailocal-inferencelocal-llmopen-source-aisingle-file-executablespeech-to-text
Release llamafile v0.4
Justine Tunney188f7fcc33a7
diff --git a/README.md b/README.mdindex 7e855cfb..3371173a 100644--- a/README.md+++ b/README.md@@ -143,7 +143,7 @@ enable you to work around Windows' 4GB executable file size limit.For Windows users, here's an example for the Mistral LLM:```sh-curl -o llamafile.exe https://github.com/Mozilla-Ocho/llamafile/releases/download/0.3/llamafile-server-0.3+curl -o llamafile.exe https://github.com/Mozilla-Ocho/llamafile/releases/download/0.4/llamafile-server-0.4curl -o mistral.gguf https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q4_K_M.gguf.\llamafile.exe -m mistral.gguf```@@ -151,7 +151,7 @@ curl -o mistral.gguf https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGHere's the same example, but for macOS, Linux, and BSD users:```sh-curl -L https://github.com/Mozilla-Ocho/llamafile/releases/download/0.3/llamafile-server-0.3 >llamafile+curl -L https://github.com/Mozilla-Ocho/llamafile/releases/download/0.4/llamafile-server-0.4 >llamafilecurl -L https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q4_K_M.gguf >mistral.ggufchmod +x llamafile./llamafile -m mistral.gguf@@ -227,21 +227,14 @@ weights fit into memory.On Apple Silicon, everything should just work if Xcode is installed.+On Windows, GPU should just work so long as (1) you're using our release+binaries, and (2) you pass the `-ngl 35` flag. You also need an NVIDIA+graphics card that supports CUDA. There's no support yet for AMD GPUs.+On Linux, Nvidia cuBLAS GPU support will be compiled on the fly if (1)-you have the `cc` compiler installed, (2) you pass the `--n-gpu-layers-35` flag (or whatever value is appropriate) to enable GPU, and (3) the-CUDA developer toolkit is installed on your machine and the `nvcc`-compiler is on your path.--On Windows, install [CUDA](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html) (you only need CUDA and the compiler, so use-the network installer and deselect other options). You must edit-Windows Environment Variables to add to PATH:-C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\bin-Invoke x64 Native Tools Command Prompt for VS 2022 (this is required to-compile the dependencies for CUDA at first start) and run llamafile.-For the first invocation, it will build a DLL with native GPU support.-When you invoke it with a model, verify GPU is used by looking for:-"total VRAM used" to be a non-zero number (usually the size of the model).+you have the `cc` compiler installed, (2) you pass the `-ngl 35` flag to+enable GPU, and (3) the CUDA developer toolkit is installed on your+machine and the `nvcc` compiler is on your path.In the event that GPU support couldn't be compiled and dynamicallylinked on the fly for any reason, llamafile will fall back to CPUdiff --git a/llamafile/version.h b/llamafile/version.hindex 7ffcae19..6083fae6 100644--- a/llamafile/version.h+++ b/llamafile/version.h@@ -1,7 +1,7 @@#pragma once#define LLAMAFILE_MAJOR 0-#define LLAMAFILE_MINOR 3+#define LLAMAFILE_MINOR 4#define LLAMAFILE_PATCH 0#define LLAMAFILE_VERSION (100000000 * LLAMAFILE_MAJOR + 1000000 * LLAMAFILE_MINOR + LLAMAFILE_PATCH)