Skip to content

Distribute and run LLMs with a single file.

Put finishing touches on zipalign tool

Justine Tunney76d47c087ad8
diff --git a/README.md b/README.md
index 26bf91b4..33316204 100644
--- a/README.md
+++ b/README.md
@@ -244,7 +244,7 @@ In the event that GPU support couldn't be compiled and dynamically
linked on the fly for any reason, llamafile will fall back to CPU
inference.
-## Source instructions
+## Source installation
Here's how to build llamafile from source. First, you need the cosmocc
toolchain, which is a fat portable binary version of GCC. Here's how you
@@ -259,18 +259,18 @@ cd ..
export PATH="$PWD/cosmocc/bin:$PATH"
```
-You can now build the llamafile repository by running make:
+You can now build and install the llamafile repository by running make:
```sh
make -j8
+sudo make install
```
Here's an example of how to generate code for a libc function using the
llama.cpp command line interface, utilizing WizardCoder-Python-13B weights:
```sh
-make -j8 o//llama.cpp/main/main
-o//llama.cpp/main/main \
+llamafile \
-m ~/weights/wizardcoder-python-13b-v1.0.Q8_0.gguf \
--temp 0 \
-r $'```\n' \
@@ -280,28 +280,19 @@ o//llama.cpp/main/main \
Here's a similar example that instead utilizes Mistral-7B-Instruct weights:
```sh
-make -j8 o//llama.cpp/main/main
-o//llama.cpp/main/main \
+llamafile \
-m ~/weights/mistral-7b-instruct-v0.1.Q4_K_M.gguf \
--temp 0.7 \
-r $'\n' \
-p $'### Instruction: Write a story about llamas\n### Response:\n'
```
-Here's an example of how to run llama.cpp's built-in HTTP server in such
-a way that the weights are embedded inside the executable. This example
-uses LLaVA v1.5-7B, a multimodal LLM that works with llama.cpp's
+Here's an example of how to run llama.cpp's built-in HTTP server. This
+example uses LLaVA v1.5-7B, a multimodal LLM that works with llama.cpp's
recently-added support for image inputs.
```sh
-make -j8
-
-o//llamafile/zipalign -j0 \
- o//llama.cpp/server/server \
- ~/weights/llava-v1.5-7b-Q8_0.gguf \
- ~/weights/llava-v1.5-7b-mmproj-Q8_0.gguf
-
-o//llama.cpp/server/server \
+llamafile-server \
-m llava-v1.5-7b-Q8_0.gguf \
--mmproj llava-v1.5-7b-mmproj-Q8_0.gguf \
--host 0.0.0.0
@@ -313,18 +304,15 @@ images to it.
If you want to be able to just say:
-```
-./server
+```sh
+./llava-server.llamafile
```
-...and have it run the web server without having to specify arguments (for
-the paths you already know are in there), then you can add a special
-`.args` to the zip archive, which specifies the default arguments. In
-this case, we're going to try our luck with the normal `zip` command,
-which requires we temporarily rename the file. First, let's create the
-arguments file:
+...and have it run the web server without having to specify arguments,
+then you can embed both the weights and a special `.args` inside, which
+specifies the default arguments. First, let's create a `.args` file:
-```
+```sh
cat <<EOF >.args
-m
llava-v1.5-7b-Q8_0.gguf
@@ -338,46 +326,42 @@ EOF
As we can see above, there's one argument per line. The `...` argument
optionally specifies where any additional CLI arguments passed by the
-user are to be inserted. Next, we'll add the argument file to the
-executable:
+user are to be inserted. Next, we'll add both the weights and the
+argument file to the executable:
-```
-mv o//llama.cpp/server/server server.com
-o//llamafile/zipalign -j0 server.com .args
-mv server.com server
-./server
-```
-
-Congratulations. You've just made your own LLM executable that's easy to
-share with your friends.
+```sh
+cp /usr/local/bin/llamafile-server llava-server.llamafile
-## zipalign documentation
+zipalign -j0 \
+ llava-server.llamafile \
+ ~/weights/llava-v1.5-7b-Q8_0.gguf \
+ ~/weights/llava-v1.5-7b-mmproj-Q8_0.gguf \
+ .args
+./llava-server.llamafile
```
-SYNOPSIS
- o//llamafile/zipalign ZIP FILE...
-
-DESCRIPTION
+Congratulations. You've just made your own LLM executable that's easy to
+share with your friends.
- Adds aligned uncompressed files to PKZIP archive
+## Distribution
- This tool is designed to concatenate gigabytes of LLM weights to an
- executable. This command goes 10x faster than `zip -j0`. Unlike zip
- you are not required to use the .com file extension for it to work.
- But most importantly, this tool has a flag that lets you insert zip
- files that are aligned on a specific boundary. The result is things
- like GPUs that have specific memory alignment requirements will now
- be able to perform math directly on the zip file's mmap()'d weights
+One good way to share a llamafile with your friends is by posting it on
+Hugging Face. If you do that, then it's recommended that you mention in
+your Hugging Face commit message what git revision or released version
+of llamafile you used when building your llamafile. That way everyone
+online will be able verify the provenance of its executable content. If
+you've made changes to the llama.cpp or cosmopolitan source code, then
+the Apache 2.0 license requires you to explain what changed. One way you
+can do that is by embedding a notice in your llamafile using `zipalign`
+that describes the changes, and mention it in your Hugging Face commit.
-FLAGS
+## Documentation
- -h help
- -N nondeterministic mode
- -a INT alignment (default 65536)
- -j strip directory components
- -0 store uncompressed (currently default)
-```
+There's a man page for each of the llamafile programs installed when you
+run `sudo make install`. The command manuals are also typeset as PDF
+files that you can download from our GitHub releases page. Lastly, most
+commands will display that information when passing the `--help` flag.
## Technical details
@@ -399,7 +383,7 @@ the local file is aligned on a page size boundary. That way, assuming
the zip file is uncompressed, once it's mmap()'d into memory we can pass
pointers directly to GPUs like Apple Metal, which require that data be
page size aligned. Since no existing ZIP archiving tool has an alignment
-flag, we had to write about [400 lines of code](llamafile/zipalign.c) to
+flag, we had to write about [500 lines of code](llamafile/zipalign.c) to
insert the ZIP files ourselves. However, once there, every existing ZIP
program should be able to read them, provided they support ZIP64. This
makes the weights much more easily accessible than they otherwise would
diff --git a/llamafile/zipalign.1 b/llamafile/zipalign.1
index 0c0070b8..cd6bfc0a 100644
--- a/llamafile/zipalign.1
+++ b/llamafile/zipalign.1
@@ -19,21 +19,70 @@ you are not required to use the .com file extension for it to work.
But most importantly, this tool has a flag that lets you insert zip
files that are aligned on a specific boundary. The result is things
like GPUs that have specific memory alignment requirements will now
-be able to perform math directly on the zip file's mmap()'d weights
+be able to perform math directly on the zip file's mmap()'d weights.
+.Pp
+This tool always operates in an append-only manner. Unlike the InfoZIP
+.Xr zip 1
+command,
+.Nm
+does not reflow existing assets to shave away space. For example, if
+.Nm
+is used on an existing PKZIP archive to replace an existing asset, then
+the bytes for the old revision of the asset will still be there, along
+with any alignment gaps that currently exist in the file between assets.
+.Pp
+The same concept also applies to the central directory listing that's
+stored at the end of the file. When changes are made, the old central
+directory is left behind as junk data. Therefore it's important, when
+adding multiple files to an archive at once, that the files all be
+passed in arguments at once, rather than calling this command multiple
+times.
.Sh OPTIONS
The following options are available:
.Bl -tag -width indent
.It Fl h
Show help.
+.It Fl v
+Operate in verbose mode.
.It Fl N
-Run in nondeterministic mode. This will cause the date/time of inserted assets to reflect the current time.
+Run in nondeterministic mode. This will cause the date/time of inserted
+assets to reflect the file modified time.
.It Fl a Ar INT
-Byte alignment for inserted zip assets. This must be a two power. It defaults to 65536.
+Byte alignment for inserted zip assets. This must be a two power. It
+defaults to 65536 since that ensures your asset will be page-aligned on
+all conceivable platforms, both now and in the future.
.It Fl j
-Strip directory components. The filename of each input path will be used as the ZIP asset name.
+Strip directory components. The filename of each input filepath will be
+used as the zip asset name. This is otherwise known as the basename. An
+error will be raised if the same zip asset name ends up being specified
+multiple times.
.It Fl 0
-Store zip assets without compression.
+Store zip assets without compression. This is the default. This option
+must be chosen when adding weights to a llamafile, otherwise it won't be
+possible to map them into memory. Using
+.Fl 0
+goes orders of a magnitude faster than using
+.Fl 6
+compression.
+.It Fl 6
+Store zip assets with sweet spot compression. Any value between
+.Fl 0
+and
+.Fl 9
+is accepted as choices for compression level. Using
+.Fl 6
+will oftentimes go 10x faster than
+.Fl 9
+and only has a marginal increase of size. Note uncompression speeds are
+unaffected.
+.It Fl 9
+Store zip assets with the maximum compression. This takes a very long
+time to compress. Uncompression will go just as fast. This might be a
+good idea when publishing archives that'll be widely consumed via the
+Internet for a long time.
.El
.Sh SEE ALSO
.Xr unzip 1 ,
.Xr llamafile 1
+.Sh AUTHORS
+.An "Justine Alexandra Roberts Tunney" Aq [email protected]
diff --git a/llamafile/zipalign.c b/llamafile/zipalign.c
index b80caeb8..25dbe15e 100644
--- a/llamafile/zipalign.c
+++ b/llamafile/zipalign.c
@@ -15,6 +15,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
+#include "zip.h"
#include <time.h>
#include <fcntl.h>
#include <cosmo.h>
@@ -27,8 +28,8 @@
#include <getopt.h>
#include <stdbool.h>
#include <sys/uio.h>
+#include <sys/stat.h>
#include <third_party/zlib/zlib.h>
-#include "zip.h"
#define USAGE \
" ZIP FILE...\n\
@@ -36,24 +37,21 @@
DESCRIPTION\n\
\n\
Adds aligned uncompressed files to PKZIP archive\n\
-\n\
- This tool is designed to concatenate gigabytes of LLM weights to an\n\
- executable. This command goes 10x faster than `zip -j0`. Unlike zip\n\
- you are not required to use the .com file extension for it to work.\n\
- But most importantly, this tool has a flag that lets you insert zip\n\
- files that are aligned on a specific boundary. The result is things\n\
- like GPUs that have specific memory alignment requirements will now\n\
- be able to perform math directly on the zip file's mmap()'d weights\n\
\n\
FLAGS\n\
\n\
-h help\n\
+ -v verbose\n\
-N nondeterministic mode\n\
-a INT alignment (default 65536)\n\
-j strip directory components\n\
- -0 store uncompressed (currently default)\n\
+ -0 store uncompressed (default)\n\
+ -6 store with faster compression\n\
+ -9 store with maximum compression\n\
\n"
+#define CHUNK 2097152
+
#define Min(a, b) ((a) < (b) ? (a) : (b))
#define DOS_DATE(YEAR, MONTH_IDX1, DAY_IDX1) \
(((YEAR)-1980) << 9 | (MONTH_IDX1) << 5 | (DAY_IDX1))
@@ -62,11 +60,13 @@ FLAGS\n\
static const char *prog;
static int FLAG_junk;
+static int FLAG_level;
+static int FLAG_verbose;
static int FLAG_alignment = 65536;
static bool FLAG_nondeterministic;
static wontreturn void Die(const char *thing, const char *reason) {
- tinyprint(2, thing, ": ", reason, "\n", NULL);
+ tinyprint(2, thing, ": fatal error: ", reason, "\n", NULL);
exit(1);
}
@@ -105,9 +105,9 @@ static void GetDosLocalTime(int64_t utcunixts,
uint16_t *out_time,
uint16_t *out_date) {
struct tm tm;
- gmtime_r(&utcunixts, &tm);
+ localtime_r(&utcunixts, &tm);
*out_time = DOS_TIME(tm.tm_hour, tm.tm_min, tm.tm_sec);
- *out_date = DOS_DATE(tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday + 1);
+ *out_date = DOS_DATE(tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday);
}
int main(int argc, char *argv[]) {
@@ -118,9 +118,22 @@ int main(int argc, char *argv[]) {
// parse flags
int opt;
- while ((opt = getopt(argc, argv, "hj0Na:")) != -1) {
+ while ((opt = getopt(argc, argv, "0123456789hvjNa:")) != -1) {
switch (opt) {
case '0':
+ case '1':
+ case '2':
+ case '3':
+ case '4':
+ case '5':
+ case '6':
+ case '7':
+ case '8':
+ case '9':
+ FLAG_level = opt - '0';
+ break;
+ case 'v':
+ ++FLAG_verbose;
break;
case 'j':
FLAG_junk = true;
@@ -211,16 +224,6 @@ int main(int argc, char *argv[]) {
DieSys(zpath);
}
- // get time
- struct timespec now;
- uint16_t mtime, mdate;
- if (FLAG_nondeterministic) {
- now = timespec_real();
- } else {
- now = timespec_fromseconds(1700000000);
- }
- GetDosLocalTime(now.tv_sec, &mtime, &mdate);
-
// create array of zip entry names
char **names = Malloc(sizeof(char *) * argc);
for (int i = optind; i < argc; ++i) {
@@ -234,6 +237,15 @@ int main(int argc, char *argv[]) {
}
}
+ // verify there's no duplicate zip asset names
+ for (int i = optind; i < argc; ++i) {
+ for (int j = i + 1; j < argc; ++j) {
+ if (!strcmp(names[i], names[j])) {
+ Die(names[i], "zip asset name specified multiple times");
+ }
+ }
+ }
+
// delete central directory entries about to be replaced
int new_count = 0;
off_t new_index = 0;
@@ -273,35 +285,104 @@ int main(int argc, char *argv[]) {
// open input file
int fd;
- ssize_t size;
- char *name = names[i];
const char *path = argv[i];
if ((fd = open(path, O_RDONLY)) == -1) {
DieSys(path);
}
- if ((size = lseek(fd, 0, SEEK_END)) == -1) {
+
+ // get information about file
+ uint64_t size;
+ struct stat st;
+ if (fstat(fd, &st) == -1) {
DieSys(path);
}
+ size = st.st_size;
+ if (!S_ISREG(st.st_mode)) {
+ Die(path, "not a regular file");
+ }
+
+ // get time
+ int64_t ts;
+ uint16_t mtime, mdate;
+ if (FLAG_nondeterministic) {
+ ts = st.st_mtime;
+ } else {
+ ts = 1700000000;
+ }
+ GetDosLocalTime(ts, &mtime, &mdate);
// determine size and alignment of local file header
+ char *name = names[i];
size_t namlen = strlen(name);
size_t extlen = (2 + 2 + 8 + 8);
size_t hdrlen = kZipLfileHdrMinSize + namlen + extlen;
while ((zsize + hdrlen) & (FLAG_alignment - 1)) ++zsize;
+ // initialize zlib in raw deflate mode
+ z_stream zs;
+ int compression;
+ if (!FLAG_level) {
+ compression = kZipCompressionNone;
+ } else {
+ compression = kZipCompressionDeflate;
+ zs.zalloc = 0;
+ zs.zfree = 0;
+ zs.opaque = 0;
+ switch (deflateInit2(&zs, FLAG_level, Z_DEFLATED, -MAX_WBITS, DEF_MEM_LEVEL, Z_DEFAULT_STRATEGY)) {
+ case Z_OK:
+ break;
+ case Z_MEM_ERROR:
+ DieOom();
+ default:
+ unassert(!"deflateInit2() called with invalid parameters");
+ }
+ }
+
// copy file
ssize_t rc;
uint32_t crc = 0;
- _Alignas(4096) static uint8_t iobuf[2097152];
+ uint64_t compsize = 0;
+ _Alignas(4096) static uint8_t iobuf[CHUNK];
+ _Alignas(4096) static uint8_t cdbuf[CHUNK];
for (off_t i = 0; i < size; i += rc) {
- if ((rc = pread(fd, iobuf, Min(size, sizeof(iobuf)), i)) <= 0) {
+ // read chunk
+ if ((rc = pread(fd, iobuf, Min(size, CHUNK), i)) <= 0) {
DieSys(path);
}
crc = crc32(crc, iobuf, rc);
- if (pwrite(zfd, iobuf, rc, zsize + hdrlen + i) != rc) {
- DieSys(zpath);
+ if (!FLAG_level) {
+ // write uncompressed chunk to output
+ if (pwrite(zfd, iobuf, rc, zsize + hdrlen + compsize) != rc) {
+ DieSys(zpath);
+ }
+ compsize += rc;
+ } else {
+ // compress chunk and write to output
+ zs.avail_in = rc;
+ zs.next_in = iobuf;
+ do {
+ zs.next_out = cdbuf;
+ zs.avail_out = CHUNK;
+ int boop;
+ switch ((boop = deflate(&zs, rc != CHUNK ? Z_FINISH : Z_FULL_FLUSH))) {
+ case Z_MEM_ERROR:
+ DieOom();
+ case Z_STREAM_ERROR:
+ unassert(!"deflate() stream error");
+ default:
+ break;
+ }
+ ssize_t have = CHUNK - zs.avail_out;
+ if (pwrite(zfd, cdbuf, have, zsize + hdrlen + compsize) != have) {
+ DieSys(zpath);
+ }
+ compsize += have;
+ } while (!zs.avail_out);
}
}
+ if (FLAG_level) {
+ unassert(deflateEnd(&zs) == Z_OK);
+ }
// write local file header
uint8_t *lochdr = Malloc(hdrlen);
@@ -310,7 +391,7 @@ int main(int argc, char *argv[]) {
p = ZIP_WRITE32(p, kZipLfileHdrMagic);
p = ZIP_WRITE16(p, kZipEra2001);
p = ZIP_WRITE16(p, kZipGflagUtf8);
- p = ZIP_WRITE16(p, kZipCompressionNone);
+ p = ZIP_WRITE16(p, compression);
p = ZIP_WRITE16(p, mtime);
p = ZIP_WRITE16(p, mdate);
p = ZIP_WRITE32(p, crc);
@@ -323,7 +404,7 @@ int main(int argc, char *argv[]) {
p = ZIP_WRITE16(p, kZipExtraZip64);
p = ZIP_WRITE16(p, 8 + 8);
p = ZIP_WRITE64(p, size); // uncompressed size
- p = ZIP_WRITE64(p, size); // compressed size
+ p = ZIP_WRITE64(p, compsize); // compressed size
unassert(p == lochdr + hdrlen);
if (pwrite(zfd, lochdr, hdrlen, zsize) != hdrlen) {
@@ -343,7 +424,7 @@ int main(int argc, char *argv[]) {
p = ZIP_WRITE16(p, kZipOsUnix << 8 | kZipEra2001); // version made by
p = ZIP_WRITE16(p, kZipEra2001); // version needed to extract
p = ZIP_WRITE16(p, kZipGflagUtf8);
- p = ZIP_WRITE16(p, kZipCompressionNone);
+ p = ZIP_WRITE16(p, compression);
p = ZIP_WRITE16(p, mtime);
p = ZIP_WRITE16(p, mdate);
p = ZIP_WRITE32(p, crc);
@@ -354,23 +435,28 @@ int main(int argc, char *argv[]) {
p = ZIP_WRITE16(p, 0); // comment length
p = ZIP_WRITE16(p, 0); // disk number start
p = ZIP_WRITE16(p, kZipIattrBinary);
- p = ZIP_WRITE32(p, 0100644u << 16); // external file attributes
+ p = ZIP_WRITE32(p, st.st_mode << 16); // external file attributes
p = ZIP_WRITE32(p, 0xffffffffu); // lfile offset
p = mempcpy(p, name, namlen);
p = ZIP_WRITE16(p, kZipExtraZip64);
p = ZIP_WRITE16(p, 8 + 8 + 8);
- p = ZIP_WRITE64(p, size); // uncompressed size
- p = ZIP_WRITE64(p, size); // compressed size
+ p = ZIP_WRITE64(p, size); // uncompressed size
+ p = ZIP_WRITE64(p, compsize); // compressed size
p = ZIP_WRITE64(p, zsize); // lfile offset
unassert(p == cdirhdr + hdrlen);
// finish up
++cnt;
- zsize += hdrlen + size;
+ zsize += hdrlen + compsize;
if (close(fd)) {
DieSys(path);
}
+
+ // log asset creation
+ if (FLAG_verbose) {
+ tinyprint(2, path, " -> ", name, "\n", NULL);
+ }
}
// write out central directory