Skip to content

Distribute and run LLMs with a single file.

Synchronize with llama.cpp 2024-01-27

Justine Tunneydfd333589abd
diff --git a/README.md b/README.md
index 75246c1d..0afb006f 100644
--- a/README.md
+++ b/README.md
@@ -164,12 +164,16 @@ ChatCompletionMessage(content='There once was a programmer named Mike\nWho wrote
We also provide example llamafiles for other models, so you can easily
try out llamafile with different kinds of LLMs.
-| Model | Size | License | llamafile |
-| --- | --- | --- | --- |
-| LLaVA 1.5 | 3.97 GB | [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) | [llava-v1.5-7b-q4.llamafile](https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llava-v1.5-7b-q4.llamafile?download=true) |
-| Mistral-7B-Instruct | 5.15 GB | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | [mistral-7b-instruct-v0.2.Q5\_K\_M.llamafile](https://huggingface.co/jartine/Mistral-7B-Instruct-v0.2-llamafile/resolve/main/mistral-7b-instruct-v0.2.Q5_K_M.llamafile?download=true) |
-| Mixtral-8x7B-Instruct | 30.03 GB | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | [mixtral-8x7b-instruct-v0.1.Q5\_K\_M.llamafile](https://huggingface.co/jartine/Mixtral-8x7B-Instruct-v0.1-llamafile/resolve/main/mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile?download=true) |
-| WizardCoder-Python-13B | 7.33 GB | [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) | [wizardcoder-python-13b.llamafile](https://huggingface.co/jartine/wizardcoder-13b-python/resolve/main/wizardcoder-python-13b.llamafile?download=true) |
+| Model | Size | License | llamafile |
+| --- | --- | --- | --- |
+| LLaVA 1.5 | 3.97 GB | [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) | [llava-v1.5-7b-q4.llamafile](https://huggingface.co/jartine/llava-v1.5-7B-GGUF/resolve/main/llava-v1.5-7b-q4.llamafile?download=true) |
+| Mistral-7B-Instruct | 5.15 GB | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | [mistral-7b-instruct-v0.2.Q5\_K\_M.llamafile](https://huggingface.co/jartine/Mistral-7B-Instruct-v0.2-llamafile/resolve/main/mistral-7b-instruct-v0.2.Q5_K_M.llamafile?download=true) |
+| Mixtral-8x7B-Instruct | 30.03 GB | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | [mixtral-8x7b-instruct-v0.1.Q5\_K\_M.llamafile](https://huggingface.co/jartine/Mixtral-8x7B-Instruct-v0.1-llamafile/resolve/main/mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile?download=true) |
+| WizardCoder-Python-34B | 22.23 GB | [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) | [wizardcoder-python-34b-v1.0.Q5\_K\_M.llamafile](https://huggingface.co/jartine/WizardCoder-Python-34B-V1.0-llamafile/resolve/main/wizardcoder-python-34b-v1.0.Q5_K_M.llamafile?download=true) |
+| WizardCoder-Python-13B | 7.33 GB | [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) | [wizardcoder-python-13b.llamafile](https://huggingface.co/jartine/wizardcoder-13b-python/resolve/main/wizardcoder-python-13b.llamafile?download=true) |
+| TinyLlama-1.1B | 0.76 GB | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/) | [TinyLlama-1.1B-Chat-v1.0.Q5\_K\_M.llamafile](https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile?download=true) |
+| Rocket-3B | 1.89 GB | [cc-by-sa-4.0](https://creativecommons.org/licenses/by-sa/4.0/deed.en) | [rocket-3b.Q5\_K\_M.llamafile](https://huggingface.co/jartine/rocket-3B-llamafile/resolve/main/rocket-3b.Q5_K_M.llamafile?download=true) |
+| Phi-2 | 1.96 GB | [MIT](https://huggingface.co/microsoft/phi-2/resolve/main/LICENSE) | [phi-2.Q5\_K\_M.llamafile](https://huggingface.co/jartine/phi-2-llamafile/resolve/main/phi-2.Q5_K_M.llamafile?download=true) |
Here is an example for the Mistral command-line llamafile:
diff --git a/llama.cpp/README.llamafile b/llama.cpp/README.llamafile
index 98ca1794..d164f0a1 100644
--- a/llama.cpp/README.llamafile
+++ b/llama.cpp/README.llamafile
@@ -9,26 +9,22 @@ LICENSE
ORIGIN
https://github.com/ggerganov/llama.cpp/pull/4406/
- a1d6df129bcd3d42cda38c09217d8d4ec4ea3bdd
- 2024-01-26
+ 6db2b41a76ee78d5efdd5c3cddd5d7ad3f646855
+ 2024-01-27
LOCAL MODIFICATIONS
- Refactor ggml.c, llama.cpp, and llava to use llamafile_open() APIs
- Unify main, server, and llava-cli into single llamafile program
- Make cuBLAS / hipBLAS optional by introducing tinyBLAS library
- - Use Microsoft ABI on CUDA module and ggml-backend interfaces
- Add support to main() programs for Cosmo /zip/.args files
- Introduce pledge() SECCOMP sandboxing to improve security
- Call exit() rather than abort() when GGML_ASSERT() fails
- - Fix OpenAI server sampling w.r.t. temperature and seed
- - Remove log callback pointer API from Metal GPU module
+ - Make GPU logger callback API safer and less generic
- Write log to /dev/null when main.log fails to open
- Use _rand64() rather than time() as default seed
- Make main and llava-cli print timings on ctrl-c
- Avoid bind() conflicts on port 8080 w/ server
- - Allow --grammar to be used on --image prompts
- Use runtime dispatching for matmul quants
- Remove operating system #ifdef statements
- - Introduce --silent-prompt flag to main
- Remove stdout logging from LLaVA
diff --git a/llama.cpp/ggml.c b/llama.cpp/ggml.c
index f36b2539..e27b8bd3 100644
--- a/llama.cpp/ggml.c
+++ b/llama.cpp/ggml.c
@@ -7533,7 +7533,12 @@ static void ggml_compute_forward_add(
switch (src0->type) {
case GGML_TYPE_F32:
{
- ggml_compute_forward_add_f32(params, src0, src1, dst);
+ if (src1->type == GGML_TYPE_F32) {
+ ggml_compute_forward_add_f32(params, src0, src1, dst);
+ }
+ else {
+ GGML_ASSERT(false);
+ }
} break;
case GGML_TYPE_F16:
{
diff --git a/llama.cpp/llama.cpp b/llama.cpp/llama.cpp
index 1c72a52f..5b01de07 100644
--- a/llama.cpp/llama.cpp
+++ b/llama.cpp/llama.cpp
@@ -7692,7 +7692,7 @@ struct llama_grammar * llama_grammar_copy(const struct llama_grammar * grammar)
void llama_set_rng_seed(struct llama_context * ctx, uint32_t seed) {
if (seed == LLAMA_DEFAULT_SEED) {
- seed = time(NULL);
+ seed = _rand64();
}
ctx->rng.seed(seed);
}
@@ -9726,7 +9726,7 @@ struct llama_context * llama_new_context_with_model(
}
if (params.seed == LLAMA_DEFAULT_SEED) {
- params.seed = time(NULL);
+ params.seed = _rand64();
}
LLAMA_LOG_INFO("%s: n_ctx = %u\n", __func__, cparams.n_ctx);
diff --git a/llama.cpp/llava/clip.cpp b/llama.cpp/llava/clip.cpp
index 3b8d53e4..c8258cdd 100644
--- a/llama.cpp/llava/clip.cpp
+++ b/llama.cpp/llava/clip.cpp
@@ -96,6 +96,7 @@ static std::string format(const char * fmt, ...) {
enum projector_type {
PROJECTOR_TYPE_MLP,
+ PROJECTOR_TYPE_MLP_NORM,
PROJECTOR_TYPE_LDP,
PROJECTOR_TYPE_UNKNOWN,
};
@@ -302,10 +303,18 @@ struct clip_vision_model {
struct ggml_tensor * projection;
// LLaVA projection
- struct ggml_tensor * mm_0_w;
- struct ggml_tensor * mm_0_b;
- struct ggml_tensor * mm_2_w;
- struct ggml_tensor * mm_2_b;
+ struct ggml_tensor * mm_0_w = NULL;
+ struct ggml_tensor * mm_0_b = NULL;
+ struct ggml_tensor * mm_2_w = NULL;
+ struct ggml_tensor * mm_2_b = NULL;
+
+ // Yi type models with mlp+normalization projection
+ struct ggml_tensor * mm_1_w = NULL; // Yi type models have 0, 1, 3, 4
+ struct ggml_tensor * mm_1_b = NULL;
+ struct ggml_tensor * mm_3_w = NULL;
+ struct ggml_tensor * mm_3_b = NULL;
+ struct ggml_tensor * mm_4_w = NULL;
+ struct ggml_tensor * mm_4_b = NULL;
// MobileVLM projection
struct ggml_tensor * mm_model_mlp_1_w;
@@ -458,6 +467,7 @@ static ggml_cgraph * clip_image_build_graph(clip_ctx * ctx, const clip_image_f32
// pre-layernorm
{
embeddings = ggml_norm(ctx0, embeddings, eps);
+ ggml_set_name(embeddings, "pre_ln");
embeddings = ggml_add(ctx0, ggml_mul(ctx0, embeddings, model.pre_ln_w), model.pre_ln_b);
}
@@ -573,6 +583,27 @@ static ggml_cgraph * clip_image_build_graph(clip_ctx * ctx, const clip_image_f32
embeddings = ggml_mul_mat(ctx0, model.mm_2_w, embeddings);
embeddings = ggml_add(ctx0, embeddings, model.mm_2_b);
+
+ } else if (ctx->proj_type == PROJECTOR_TYPE_MLP_NORM) {
+ embeddings = ggml_mul_mat(ctx0, model.mm_0_w, embeddings);
+ embeddings = ggml_add(ctx0, embeddings, model.mm_0_b);
+ // ggml_tensor_printf(embeddings, "mm_0_w",0,true,false);
+ // First LayerNorm
+ embeddings = ggml_norm(ctx0, embeddings, eps);
+ embeddings = ggml_add(ctx0, ggml_mul(ctx0, embeddings, model.mm_1_w),
+ model.mm_1_b);
+
+ // GELU activation
+ embeddings = ggml_gelu(ctx0, embeddings);
+
+ // Second linear layer
+ embeddings = ggml_mul_mat(ctx0, model.mm_3_w, embeddings);
+ embeddings = ggml_add(ctx0, embeddings, model.mm_3_b);
+
+ // Second LayerNorm
+ embeddings = ggml_norm(ctx0, embeddings, eps);
+ embeddings = ggml_add(ctx0, ggml_mul(ctx0, embeddings, model.mm_4_w),
+ model.mm_4_b);
}
else if (ctx->proj_type == PROJECTOR_TYPE_LDP) {
// MobileVLM projector
@@ -811,6 +842,11 @@ struct clip_ctx * clip_model_load(const char * fname, const int verbosity = 1) {
else {
new_clip->proj_type = PROJECTOR_TYPE_MLP;
}
+ if (new_clip->proj_type == PROJECTOR_TYPE_MLP) {
+ if (gguf_find_tensor(ctx, format(TN_LLAVA_PROJ, 3, "weight").c_str()) != -1) {
+ new_clip->proj_type = PROJECTOR_TYPE_MLP_NORM;
+ }
+ }
}
if ((new_clip->backend = ggml_backend_metal_init())) {
@@ -952,11 +988,29 @@ struct clip_ctx * clip_model_load(const char * fname, const int verbosity = 1) {
vision_model.pre_ln_b = get_tensor(new_clip->ctx_data, format(TN_LN_PRE, "v", "bias"));
// LLaVA projection
- if (new_clip->proj_type == PROJECTOR_TYPE_MLP) {
+ if (new_clip->proj_type == PROJECTOR_TYPE_MLP || new_clip->proj_type == PROJECTOR_TYPE_MLP_NORM) {
vision_model.mm_0_w = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 0, "weight"));
vision_model.mm_0_b = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 0, "bias"));
- vision_model.mm_2_w = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 2, "weight"));
- vision_model.mm_2_b = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 2, "bias"));
+ try {
+ // Yi-type llava
+ vision_model.mm_1_w = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 1, "weight"));
+ vision_model.mm_1_b = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 1, "bias"));
+ } catch (std::runtime_error & e) { }
+ try {
+ // missing in Yi-type llava
+ vision_model.mm_2_w = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 2, "weight"));
+ vision_model.mm_2_b = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 2, "bias"));
+ } catch (std::runtime_error & e) { }
+ try {
+ // Yi-type llava
+ vision_model.mm_3_w = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 3, "weight"));
+ vision_model.mm_3_b = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 3, "bias"));
+ } catch (std::runtime_error & e) { }
+ try {
+ // Yi-type llava
+ vision_model.mm_4_w = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 4, "weight"));
+ vision_model.mm_4_b = get_tensor(new_clip->ctx_data, format(TN_LLAVA_PROJ, 4, "bias"));
+ } catch (std::runtime_error & e) { }
}
else if (new_clip->proj_type == PROJECTOR_TYPE_LDP) {
// MobileVLM projection
@@ -1271,7 +1325,6 @@ bool clip_model_quantize(const char * fname_inp, const char * fname_out, const i
".*weight",
};
- std::vector<uint8_t> read_data(512);
std::vector<uint8_t> work(512);
std::vector<float> conv_buf(512);
std::vector<int64_t> hist_all(1 << 4, 0);
@@ -1320,7 +1373,7 @@ bool clip_model_quantize(const char * fname_inp, const char * fname_out, const i
f32_data = (float *)conv_buf.data();
break;
default:
- fprintf(stderr, "Please use an input file in f32 or f16\n");
+ printf("Please use an input file in f32 or f16\n");
return false;
}
@@ -1387,7 +1440,7 @@ bool clip_model_quantize(const char * fname_inp, const char * fname_out, const i
fout.put(0);
}
- tinylogf("%s: n_dims = %d | quantize=%d | size = %f MB -> %f MB\n", name.c_str(), ggml_n_dims(cur), quantize,
+ printf("%s: n_dims = %d | quantize=%d | size = %f MB -> %f MB\n", name.c_str(), ggml_n_dims(cur), quantize,
orig_size / 1024.0 / 1024.0, new_size / 1024.0 / 1024.0);
}
@@ -1403,19 +1456,19 @@ bool clip_model_quantize(const char * fname_inp, const char * fname_out, const i
gguf_free(ctx_out);
{
- tinylogf("%s: original size = %8.2f MB\n", __func__, total_size_org / 1024.0 / 1024.0);
- tinylogf("%s: quantized size = %8.2f MB\n", __func__, total_size_new / 1024.0 / 1024.0);
+ printf("%s: original size = %8.2f MB\n", __func__, total_size_org / 1024.0 / 1024.0);
+ printf("%s: quantized size = %8.2f MB\n", __func__, total_size_new / 1024.0 / 1024.0);
int64_t sum_all = 0;
for (size_t i = 0; i < hist_all.size(); ++i) {
sum_all += hist_all[i];
}
- tinylogf("%s: hist: ", __func__);
+ printf("%s: hist: ", __func__);
for (size_t i = 0; i < hist_all.size(); ++i) {
- tinylogf("%5.3f ", hist_all[i] / (float)sum_all);
+ printf("%5.3f ", hist_all[i] / (float)sum_all);
}
- tinylogf("\n");
+ printf("\n");
}
return true;
@@ -1427,6 +1480,8 @@ int clip_n_mmproj_embd(const struct clip_ctx * ctx) {
}
else if (ctx->proj_type == PROJECTOR_TYPE_MLP) {
return ctx->vision_model.mm_2_b->ne[0];
+ } else if (ctx->proj_type == PROJECTOR_TYPE_MLP_NORM) {
+ return ctx->vision_model.mm_3_b->ne[0];
}
else {
std::string proj_type = PROJECTOR_TYPE_NAMES[ctx->proj_type];
diff --git a/llama.cpp/llava/llava-cli.cpp b/llama.cpp/llava/llava-cli.cpp
index bf32a01c..f15c4aa1 100644
--- a/llama.cpp/llava/llava-cli.cpp
+++ b/llama.cpp/llava/llava-cli.cpp
@@ -163,10 +163,35 @@ static void process_prompt(struct llava_context * ctx_llava, struct llava_image_
const int max_tgt_len = params->n_predict < 0 ? 256 : params->n_predict;
const bool add_bos = llama_should_add_bos_token(llama_get_model(ctx_llava->ctx_llama));
- // llava chat format is "<system_prompt>\nUSER:<image_embeddings>\n<textual_prompt>\nASSISTANT:"
- eval_string(ctx_llava->ctx_llama, "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\nUSER:", params->n_batch, &n_past, add_bos);
+ std::string system_prompt, user_prompt;
+ size_t image_pos = prompt.find("<image>");
+ if (image_pos != std::string::npos) {
+ // new templating mode: Provide the full prompt including system message and use <image> as a placeholder for the image
+
+ system_prompt = prompt.substr(0, image_pos);
+ user_prompt = prompt.substr(image_pos + std::string("<image>").length());
+ // We replace \n with actual newlines in user_prompt, just in case -e was not used in templating string
+ size_t pos = 0;
+ while ((pos = user_prompt.find("\\n", pos)) != std::string::npos) {
+ user_prompt.replace(pos, 2, "\n");
+ pos += 1; // Advance past the replaced newline
+ }
+ while ((pos = system_prompt.find("\\n", pos)) != std::string::npos) {
+ system_prompt.replace(pos, 2, "\n");
+ pos += 1; // Advance past the replaced newline
+ }
+
+ printf("system_prompt: %s\n", system_prompt.c_str());
+ printf("user_prompt: %s\n", user_prompt.c_str());
+ } else {
+ // llava-1.5 native mode
+ system_prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\nUSER:";
+ user_prompt = prompt + "\nASSISTANT:";
+ }
+
+ eval_string(ctx_llava->ctx_llama, system_prompt.c_str(), params->n_batch, &n_past, add_bos);
llava_eval_image_embed(ctx_llava->ctx_llama, image_embed, params->n_batch, &n_past);
- eval_string(ctx_llava->ctx_llama, (prompt + "\nASSISTANT:").c_str(), params->n_batch, &n_past, false);
+ eval_string(ctx_llava->ctx_llama, user_prompt.c_str(), params->n_batch, &n_past, false);
// generate the response
@@ -177,6 +202,7 @@ static void process_prompt(struct llava_context * ctx_llava, struct llava_image_
for (int i = 0; i < max_tgt_len; i++) {
const char * tmp = sample(ctx_sampling, ctx_llava->ctx_llama, &n_past);
if (strcmp(tmp, "</s>") == 0) break;
+ if (strstr(tmp, "###")) break; // Yi-VL behavior
printf("%s", tmp);
fflush(stdout);
diff --git a/llama.cpp/main/main.1 b/llama.cpp/main/main.1
index 67c497ff..06acc11a 100644
--- a/llama.cpp/main/main.1
+++ b/llama.cpp/main/main.1
@@ -477,7 +477,6 @@ layer (default): split layers and KV across GPUs
row: split rows across GPUs
.El
.It Fl ts Ar SPLIT , Fl Fl tensor-split Ar SPLIT
-
When using multiple GPUs this option controls how large tensors should
be split across all GPUs.
.Ar SPLIT
@@ -486,10 +485,8 @@ proportion of data that each GPU should get in order. For example,
\[dq]3,2\[dq] will assign 60% of the data to GPU 0 and 40% to GPU 1. By
default the data is split in proportion to VRAM but this may not be
optimal for performance. Requires cuBLAS.
-
How to split tensors across multiple GPUs, comma-separated list of
proportions, e.g. 3,1
-
.It Fl mg Ar i , Fl Fl main-gpu Ar i
The GPU to use for scratch and small tensors.
.It Fl nommq , Fl Fl no-mul-mat-q
@@ -704,6 +701,18 @@ Enable embedding vector output.
Default: disabled
.It Fl Fl nobrowser
Do not attempt to open a web browser tab at startup.
+.It Fl gan Ar N , Fl Fl grp-attn-n Ar N
+Set the group attention factor to extend context size through
+self-extend. The default value is
+.Ar 1
+which means disabled. This flag is used together with
+.Fl Fl grp-attn-w .
+.It Fl gaw Ar N , Fl Fl grp-attn-w Ar N
+Set the group attention width to extend context size through
+self-extend. The default value is
+.Ar 512 .
+This flag is used together with
+.Fl Fl grp-attn-n .
.El
.Sh LOG OPTIONS
The following log options are available:
diff --git a/llama.cpp/main/main.1.asc b/llama.cpp/main/main.1.asc
index 8475833a..49eefb57 100644
--- a/llama.cpp/main/main.1.asc
+++ b/llama.cpp/main/main.1.asc
@@ -428,17 +428,14 @@ OOPPTTIIOONNSS
-- row: split rows across GPUs
--ttss _S_P_L_I_T, ----tteennssoorr--sspplliitt _S_P_L_I_T
-
When using multiple GPUs this option controls how large tensors
should be split across all GPUs. _S_P_L_I_T is a comma-separated list
of non-negative values that assigns the proportion of data that
each GPU should get in order. For example, "3,2" will assign 60%
of the data to GPU 0 and 40% to GPU 1. By default the data is
split in proportion to VRAM but this may not be optimal for per‐
- formance. Requires cuBLAS.
-
- How to split tensors across multiple GPUs, comma-separated list
- of proportions, e.g. 3,1
+ formance. Requires cuBLAS. How to split tensors across multiple
+ GPUs, comma-separated list of proportions, e.g. 3,1
--mmgg _i, ----mmaaiinn--ggppuu _i
The GPU to use for scratch and small tensors.
@@ -689,6 +686,16 @@ SSEERRVVEERR OOPPTTIIOONNSS
----nnoobbrroowwsseerr
Do not attempt to open a web browser tab at startup.
+ --ggaann _N, ----ggrrpp--aattttnn--nn _N
+ Set the group attention factor to extend context size through
+ self-extend. The default value is _1 which means disabled. This
+ flag is used together with ----ggrrpp--aattttnn--ww.
+
+ --ggaaww _N, ----ggrrpp--aattttnn--ww _N
+ Set the group attention width to extend context size through
+ self-extend. The default value is _5_1_2. This flag is used to‐
+ gether with ----ggrrpp--aattttnn--nn.
+
LLOOGG OOPPTTIIOONNSS
The following log options are available:
diff --git a/llama.cpp/sampling.cpp b/llama.cpp/sampling.cpp
index e6a332f9..c3c12d71 100644
--- a/llama.cpp/sampling.cpp
+++ b/llama.cpp/sampling.cpp
@@ -16,6 +16,7 @@ struct llama_sampling_context * llama_sampling_init(const struct llama_sampling_
// will be empty (default) if there are parse errors
if (result->parsed_grammar.rules.empty()) {
fprintf(stderr, "%s: failed to parse grammar\n", __func__);
+ delete result;
return nullptr;
}
diff --git a/llama.cpp/server/README.md b/llama.cpp/server/README.md
index c256a9c5..1c92a204 100644
--- a/llama.cpp/server/README.md
+++ b/llama.cpp/server/README.md
@@ -23,12 +23,15 @@ Command line options:
- `--host`: Set the hostname or ip address to listen. Default `127.0.0.1`.
- `--port`: Set the port to listen. Default: `8080`.
- `--path`: path from which to serve static files (default examples/server/public)
+- `--api-key`: Set an api key for request authorization. By default the server responds to every request. With an api key set, the requests must have the Authorization header set with the api key as Bearer token. May be used multiple times to enable multiple valid keys.
+- `--api-key-file`: path to file containing api keys delimited by new lines. If set, requests must include one of the keys for access. May be used in conjunction with `--api-key`'s.
- `--embedding`: Enable embedding extraction, Default: disabled.
- `-np N`, `--parallel N`: Set the number of slots for process requests (default: 1)
- `-cb`, `--cont-batching`: enable continuous batching (a.k.a dynamic batching) (default: disabled)
- `-spf FNAME`, `--system-prompt-file FNAME` Set a file to load "a system prompt (initial prompt of all slots), this is useful for chat applications. [See more](#change-system-prompt-on-runtime)
- `--mmproj MMPROJ_FILE`: Path to a multimodal projector file for LLaVA.
-
+- `--grp-attn-n`: Set the group attention factor to extend context size through self-extend(default: 1=disabled), used together with group attention width `--grp-attn-w`
+- `--grp-attn-w`: Set the group attention width to extend context size through self-extend(default: 512), used together with group attention factor `--grp-attn-n`
## Build
server is build alongside everything else from the root of the project
@@ -109,6 +112,10 @@ node index.js
```
## API Endpoints
+- **GET** `/health`: Returns the current state of the server:
+ - `{"status": "loading model"}` if the model is still being loaded.
+ - `{"status": "error"}` if the model failed to load.
+ - `{"status": "ok"}` if the model is successfully loaded and the server is ready for further requests mentioned below.
- **POST** `/completion`: Given a `prompt`, it returns the predicted completion.
@@ -148,6 +155,8 @@ node index.js
`frequency_penalty`: Repeat alpha frequency penalty (default: 0.0, 0.0 = disabled);
+ `penalty_prompt`: This will replace the `prompt` for the purpose of the penalty evaluation. Can be either `null`, a string or an array of numbers representing tokens (default: `null` = use the original `prompt`).
+
`mirostat`: Enable Mirostat sampling, controlling perplexity during text generation (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0).
`mirostat_tau`: Set the Mirostat target entropy, parameter tau (default: 5.0).
@@ -164,43 +173,52 @@ node index.js
`n_probs`: If greater than 0, the response also contains the probabilities of top N tokens for each generated token (default: 0)
- `image_data`: An array of objects to hold base64-encoded image `data` and its `id`s to be reference in `prompt`. You can determine the place of the image in the prompt as in the following: `USER:[img-12]Describe the image in detail.\nASSISTANT:` In this case, `[img-12]` will be replaced by the embeddings of the image id 12 in the following `image_data` array: `{..., "image_data": [{"data": "<BASE64_STRING>", "id": 12}]}`. Use `image_data` only with multimodal models, e.g., LLaVA.
-
- *Result JSON:*
-
- Note: When using streaming mode (`stream`) only `content` and `stop` will be returned until end of completion.
-
- `content`: Completion result as a string (excluding `stopping_word` if any). In case of streaming mode, will contain the next token as a string.
-
- `stop`: Boolean for use with `stream` to check whether the generation has stopped (Note: This is not related to stopping words array `stop` from input options)
-
- `generation_settings`: The provided options above excluding `prompt` but including `n_ctx`, `model`
+ `image_data`: An array of objects to hold base64-encoded image `data` and its `id`s to be reference in `prompt`. You can determine the place of the image in the prompt as in the following: `USER:[img-12]Describe the image in detail.\nASSISTANT:`. In this case, `[img-12]` will be replaced by the embeddings of the image with id `12` in the following `image_data` array: `{..., "image_data": [{"data": "<BASE64_STRING>", "id": 12}]}`. Use `image_data` only with multimodal models, e.g., LLaVA.
- `model`: The path to the model loaded with `-m`
-
- `prompt`: The provided `prompt`
-
- `stopped_eos`: Indicating whether the completion has stopped because it encountered the EOS token
-
- `stopped_limit`: Indicating whether the completion stopped because `n_predict` tokens were generated before stop words or EOS was encountered
-
- `stopped_word`: Indicating whether the completion stopped due to encountering a stopping word from `stop` JSON array provided
-
- `stopping_word`: The stopping word encountered which stopped the generation (or "" if not stopped due to a stopping word)
+ `slot_id`: Assign the completion task to an specific slot. If is -1 the task will be assigned to a Idle slot (default: -1)
- `timings`: Hash of timing information about the completion such as the number of tokens `predicted_per_second`
+ `cache_prompt`: Save the prompt and generation for avoid reprocess entire prompt if a part of this isn't change (default: false)
- `tokens_cached`: Number of tokens from the prompt which could be re-used from previous completion (`n_past`)
+ `system_prompt`: Change the system prompt (initial prompt of all slots), this is useful for chat applications. [See more](#change-system-prompt-on-runtime)
- `tokens_evaluated`: Number of tokens evaluated in total from the prompt
+### Result JSON:
- `truncated`: Boolean indicating if the context size was exceeded during generation, i.e. the number of tokens provided in the prompt (`tokens_evaluated`) plus tokens generated (`tokens predicted`) exceeded the context size (`n_ctx`)
+* Note: When using streaming mode (`stream`) only `content` and `stop` will be returned until end of completion.
- `slot_id`: Assign the completion task to an specific slot. If is -1 the task will be assigned to a Idle slot (default: -1)
- `cache_prompt`: Save the prompt and generation for avoid reprocess entire prompt if a part of this isn't change (default: false)
+- `completion_probabilities`: An array of token probabilities for each completion. The array's length is `n_predict`. Each item in the array has the following structure:
- `system_prompt`: Change the system prompt (initial prompt of all slots), this is useful for chat applications. [See more](#change-system-prompt-on-runtime)
+```
+{
+ "content": "<the token selected by the model>",
+ "probs": [
+ {
+ "prob": float,
+ "tok_str": "<most likely token>"
+ },
+ {
+ "prob": float,
+ "tok_str": "<second most likely tonen>"
+ },
+ ...
+ ]
+},
+```
+Notice that each `probs` is an array of length `n_probs`.
+
+- `content`: Completion result as a string (excluding `stopping_word` if any). In case of streaming mode, will contain the next token as a string.
+- `stop`: Boolean for use with `stream` to check whether the generation has stopped (Note: This is not related to stopping words array `stop` from input options)
+- `generation_settings`: The provided options above excluding `prompt` but including `n_ctx`, `model`
+- `model`: The path to the model loaded with `-m`
+- `prompt`: The provided `prompt`
+- `stopped_eos`: Indicating whether the completion has stopped because it encountered the EOS token
+- `stopped_limit`: Indicating whether the completion stopped because `n_predict` tokens were generated before stop words or EOS was encountered
+- `stopped_word`: Indicating whether the completion stopped due to encountering a stopping word from `stop` JSON array provided
+- `stopping_word`: The stopping word encountered which stopped the generation (or "" if not stopped due to a stopping word)
+- `timings`: Hash of timing information about the completion such as the number of tokens `predicted_per_second`
+- `tokens_cached`: Number of tokens from the prompt which could be re-used from previous completion (`n_past`)
+- `tokens_evaluated`: Number of tokens evaluated in total from the prompt
+- `truncated`: Boolean indicating if the context size was exceeded during generation, i.e. the number of tokens provided in the prompt (`tokens_evaluated`) plus tokens generated (`tokens predicted`) exceeded the context size (`n_ctx`)
- **POST** `/tokenize`: Tokenize a given text.
@@ -222,6 +240,8 @@ node index.js
`content`: Set the text to process.
+ `image_data`: An array of objects to hold base64-encoded image `data` and its `id`s to be reference in `content`. You can determine the place of the image in the content as in the following: `Image: [img-21].\nCaption: This is a picture of a house`. In this case, `[img-21]` will be replaced by the embeddings of the image with id `21` in the following `image_data` array: `{..., "image_data": [{"data": "<BASE64_STRING>", "id": 21}]}`. Use `image_data` only with multimodal models, e.g., LLaVA.
+
- **POST** `/infill`: For code infilling. Takes a prefix and a suffix and returns the predicted completion as stream.
*Options:*
@@ -245,19 +265,21 @@ node index.js
You can use either Python `openai` library with appropriate checkpoints:
```python
- #!/usr/bin/env python3
- from openai import OpenAI
- client = OpenAI(
+ import openai
+
+ client = openai.OpenAI(
base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
api_key = "sk-no-key-required"
)
+
completion = client.chat.completions.create(
- model="LLaMA_CPP",
- messages=[
- {"role": "system", "content": "You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."},
- {"role": "user", "content": "Write a limerick about python exceptions"}
- ]
+ model="gpt-3.5-turbo",
+ messages=[
+ {"role": "system", "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."},
+ {"role": "user", "content": "Write a limerick about python exceptions"}
+ ]
)
+
print(completion.choices[0].message)
```
... or raw HTTP requests:
@@ -267,18 +289,18 @@ node index.js
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
- "model": "LLaMA_CPP",
- "messages": [
- {
- "role": "system",
- "content": "You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
- },
- {
- "role": "user",
- "content": "Write a limerick about python exceptions"
- }
- ]
- }' | json_pp
+ "model": "gpt-3.5-turbo",
+ "messages": [
+ {
+ "role": "system",
+ "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
+ },
+ {
+ "role": "user",
+ "content": "Write a limerick about python exceptions"
+ }
+ ]
+ }'
```
## More examples
diff --git a/llama.cpp/server/server.cpp b/llama.cpp/server/server.cpp
index d60304f7..e43daaf3 100644
--- a/llama.cpp/server/server.cpp
+++ b/llama.cpp/server/server.cpp
@@ -190,6 +190,12 @@ struct llama_client_slot
struct llama_sampling_params sparams;
llama_sampling_context *ctx_sampling = nullptr;
+ int32_t ga_i = 0; // group-attention state
+ int32_t ga_n = 1;// group-attention factor
+ int32_t ga_w = 512; // group-attention width
+
+ int32_t n_past_se = 0; // self-extend
+
// multimodal
std::vector<slot_image> images;
@@ -218,7 +224,8 @@ struct llama_client_slot
sent_count = 0;
sent_token_probs_index = 0;
infill = false;
-
+ ga_i = 0;
+ n_past_se = 0;
generated_token_probs.clear();
for (slot_image & img : images)
@@ -405,9 +412,26 @@ struct llama_server_context
slot.id = i;
slot.n_ctx = n_ctx_slot;
- slot.reset();
LOG_TEE(" -> Slot %i - max context: %i\n", slot.id, n_ctx_slot);
+
+ const int ga_n = params.grp_attn_n;
+ const int ga_w = params.grp_attn_w;
+
+ if (ga_n != 1) {
+ GGML_ASSERT(ga_n > 0 && "ga_n must be positive"); // NOLINT
+ GGML_ASSERT(ga_w % ga_n == 0 && "ga_w must be a multiple of ga_n"); // NOLINT
+ //GGML_ASSERT(n_ctx_train % ga_w == 0 && "n_ctx_train must be a multiple of ga_w"); // NOLINT
+ //GGML_ASSERT(n_ctx >= n_ctx_train * ga_n && "n_ctx must be at least n_ctx_train * ga_n"); // NOLINT
+ LOG_TEE(" -> Slot %i - self-extend: ga_n = %d, ga_w = %d\n", slot.id, ga_n, ga_w);
+ }
+
+ slot.ga_i = 0;
+ slot.ga_n = ga_n;
+ slot.ga_w = ga_w;
+
+ slot.reset();
+
slots.push_back(slot);
}
@@ -663,7 +687,7 @@ struct llama_server_context
while ((pos = prompt.find(pattern, pos)) != std::string::npos) {
size_t end_prefix = pos;
pos += pattern.length();
- size_t end_pos = prompt.find("]", pos);
+ size_t end_pos = prompt.find(']', pos);
if (end_pos != std::string::npos)
{
std::string image_id = prompt.substr(pos, end_pos - pos);
@@ -1355,32 +1379,35 @@ struct llama_server_context
for (llama_client_slot &slot : slots)
{
- if (slot.is_processing() && slot.cache_tokens.size() >= (size_t) slot.n_ctx)
+ if (slot.ga_n == 1)
{
- // Shift context
- const int n_left = slot.n_past - slot.params.n_keep - 1;
- const int n_discard = n_left / 2;
+ if (slot.is_processing() && slot.cache_tokens.size() >= (size_t) slot.n_ctx)
+ {
+ // Shift context
+ const int n_left = slot.n_past - slot.params.n_keep - 1;
+ const int n_discard = n_left / 2;
- LOG_TEE("slot %d: context shift - n_keep = %d, n_left = %d, n_discard = %d\n", slot.id, slot.params.n_keep, n_left, n_discard);
- llama_kv_cache_seq_rm (ctx, slot.id, slot.params.n_keep + 1 , slot.params.n_keep + n_discard + 1);
- llama_kv_cache_seq_shift(ctx, slot.id, slot.params.n_keep + 1 + n_discard, slot.n_past, -n_discard);
+ LOG_TEE("slot %d: context shift - n_keep = %d, n_left = %d, n_discard = %d\n", slot.id, slot.params.n_keep, n_left, n_discard);
+ llama_kv_cache_seq_rm (ctx, slot.id, slot.params.n_keep + 1 , slot.params.n_keep + n_discard + 1);
+ llama_kv_cache_seq_shift(ctx, slot.id, slot.params.n_keep + 1 + n_discard, slot.n_past, -n_discard);
- for (size_t i = slot.params.n_keep + 1 + n_discard; i < slot.cache_tokens.size(); i++)
- {
- slot.cache_tokens[i - n_discard] = slot.cache_tokens[i];
- }
+ for (size_t i = slot.params.n_keep + 1 + n_discard; i < slot.cache_tokens.size(); i++)
+ {
+ slot.cache_tokens[i - n_discard] = slot.cache_tokens[i];
+ }
- slot.cache_tokens.resize(slot.cache_tokens.size() - n_discard);
+ slot.cache_tokens.resize(slot.cache_tokens.size() - n_discard);
- slot.n_past -= n_discard;
+ slot.n_past -= n_discard;
- slot.truncated = true;
+ slot.truncated = true;
- LOG_VERBOSE("context shift", {
- {"n_ctx", n_ctx},
- {"n_keep", params.n_keep},
- {"n_left", n_left},
- });
+ LOG_VERBOSE("context shift", {
+ { "n_ctx", n_ctx },
+ { "n_keep", params.n_keep },
+ { "n_left", n_left },
+ });
+ }
}
}
@@ -1407,7 +1434,8 @@ struct llama_server_context
slot.i_batch = batch.n_tokens;
- llama_batch_add(batch, slot.sampled, system_tokens.size() + slot.n_past, { slot.id }, true);
+ const int32_t slot_npast = slot.n_past_se > 0 ? slot.n_past_se : slot.n_past;
+ llama_batch_add(batch, slot.sampled, system_tokens.size() + slot_npast, { slot.id }, true);
slot.n_past += 1;
}
@@ -1505,6 +1533,8 @@ struct llama_server_context
llama_sampling_reset(slot.ctx_sampling);
slot.n_past = 0;
+ slot.n_past_se = 0;
+ slot.ga_i = 0;
slot.num_prompt_tokens_processed = slot.num_prompt_tokens;
}
else
@@ -1518,6 +1548,25 @@ struct llama_server_context
slot.n_past = common_part(slot.cache_tokens, prompt_tokens);
slot.num_prompt_tokens_processed = slot.num_prompt_tokens - slot.n_past;
+ if (slot.ga_n != 1)
+ {
+ int ga_i = 0;
+ int32_t ga_n = slot.ga_n;
+ int32_t ga_w = slot.ga_w;
+ int32_t slot_npast = 0;
+ for (int k = 0; k < slot.n_past; ++k)
+ {
+ while (slot_npast >= ga_i + ga_w) {
+ const int bd = (ga_w/ga_n)*(ga_n - 1);
+ slot_npast -= bd;
+ ga_i += ga_w/ga_n;
+ }
+ slot_npast++;
+ }
+ slot.n_past_se = slot_npast;
+ slot.ga_i = ga_i;
+ }
+
LOG_TEE("slot %d : in cache: %i tokens | to process: %i tokens\n", slot.id, slot.n_past, slot.num_prompt_tokens_processed);
}
@@ -1532,6 +1581,10 @@ struct llama_server_context
// we have to evaluate at least 1 token to generate logits.
LOG_TEE("slot %d : we have to evaluate at least 1 token to generate logits\n", slot.id);
slot.n_past--;
+ if (slot.ga_i > 0)
+ {
+ slot.n_past_se--;
+ }
}
LOG_VERBOSE("prompt ingested", {
@@ -1544,9 +1597,22 @@ struct llama_server_context
// process the prefix of first image
std::vector<llama_token> prefix_tokens = has_images ? tokenize(slot.images[0].prefix_prompt, add_bos_token) : prompt_tokens;
+ int32_t slot_npast = slot.n_past_se > 0 ? slot.n_past_se : slot.n_past;
+ int ga_i = slot.ga_i;
+ int32_t ga_n = slot.ga_n;
+ int32_t ga_w = slot.ga_w;
for (; slot.n_past < (int) prefix_tokens.size(); ++slot.n_past)
{
- llama_batch_add(batch, prefix_tokens[slot.n_past], system_tokens.size() + slot.n_past, { slot.id }, false);
+ if (slot.ga_n != 1)
+ {
+ while (slot_npast >= ga_i + ga_w) {
+ const int bd = (ga_w/ga_n)*(ga_n - 1);
+ slot_npast -= bd;
+ ga_i += ga_w/ga_n;
+ }
+ }
+ llama_batch_add(batch, prefix_tokens[slot.n_past], system_tokens.size() + slot_npast, {slot.id }, false);
+ slot_npast += 1;
}
if (has_images && !ingest_images(slot, n_batch))
@@ -1576,6 +1642,36 @@ struct llama_server_context
for (int32_t i = 0; i < (int32_t) batch.n_tokens; i += n_batch)
{
const int32_t n_tokens = std::min(n_batch, (int32_t) (batch.n_tokens - i));
+
+ for (auto & slot : slots)
+ {
+ if (slot.ga_n != 1)
+ {
+ // context extension via Self-Extend
+ while (slot.n_past_se >= slot.ga_i + slot.ga_w)
+ {
+ const int ib = (slot.ga_n * slot.ga_i) / slot.ga_w;
+ const int bd = (slot.ga_w / slot.ga_n) * (slot.ga_n - 1);
+ const int dd = (slot.ga_w / slot.ga_n) - ib * bd - slot.ga_w;
+
+ LOG_TEE("\n");
+ LOG_TEE("shift: [%6d, %6d] + %6d -> [%6d, %6d]\n", slot.ga_i, slot.n_past_se, ib * bd, slot.ga_i + ib * bd, slot.n_past_se + ib * bd);
+ LOG_TEE("div: [%6d, %6d] / %6d -> [%6d, %6d]\n", slot.ga_i + ib * bd, slot.ga_i + ib * bd + slot.ga_w, slot.ga_n, (slot.ga_i + ib * bd) / slot.ga_n, (slot.ga_i + ib * bd + slot.ga_w) / slot.ga_n);
+ LOG_TEE("shift: [%6d, %6d] + %6d -> [%6d, %6d]\n", slot.ga_i + ib * bd + slot.ga_w, slot.n_past_se + ib * bd, dd, slot.ga_i + ib * bd + slot.ga_w + dd, slot.n_past_se + ib * bd + dd);
+
+ llama_kv_cache_seq_shift(ctx, slot.id, slot.ga_i, slot.n_past_se, ib * bd);
+ llama_kv_cache_seq_div(ctx, slot.id, slot.ga_i + ib * bd, slot.ga_i + ib * bd + slot.ga_w,slot.ga_n);
+ llama_kv_cache_seq_shift(ctx, slot.id, slot.ga_i + ib * bd + slot.ga_w,slot.n_past_se + ib * bd, dd);
+
+ slot.n_past_se -= bd;
+
+ slot.ga_i += slot.ga_w / slot.ga_n;
+
+ LOG_TEE("\nn_past_old = %d, n_past = %d, ga_i = %d\n\n", slot.n_past_se + bd, slot.n_past_se, slot.ga_i);
+ }
+ slot.n_past_se += n_tokens;
+ }
+ }
llama_batch batch_view =
{
n_tokens,
@@ -1589,6 +1685,7 @@ struct llama_server_context
};
const int ret = llama_decode(ctx, batch_view);
+
if (ret != 0)
{
if (n_batch == 1 || ret < 0)
@@ -1734,6 +1831,8 @@ static void server_print_usage(const char *argv0, const gpt_params &params,
printf(" --override-kv KEY=TYPE:VALUE\n");
printf(" advanced option to override model metadata by key. may be specified multiple times.\n");
printf(" types: int, float, bool. example: --override-kv tokenizer.ggml.add_bos_token=bool:false\n");
+ printf(" -gan N, --grp-attn-n N Set the group attention factor to extend context size through self-extend(default: 1=disabled), used together with group attention width `--grp-attn-w`");
+ printf(" -gaw N, --grp-attn-w N Set the group attention width to extend context size through self-extend(default: 512), used together with group attention factor `--grp-attn-n`");
printf("\n");
}
@@ -1919,6 +2018,25 @@ static void server_params_parse(int argc, char **argv, server_params &sparams,
}
params.n_threads = std::stoi(argv[i]);
}
+ else if (arg == "--grp-attn-n" || arg == "-gan")
+ {
+ if (++i >= argc) {
+ invalid_param = true;
+ break;
+ }
+
+ params.grp_attn_n = std::stoi(argv[i]);
+ }
+ else if (arg == "--grp-attn-w" || arg == "-gaw")
+ {
+ if (++i >= argc)
+ {
+ invalid_param = true;
+ break;
+ }
+
+ params.grp_attn_w = std::stoi(argv[i]);
+ }
else if (arg == "--threads-batch" || arg == "-tb")
{
if (++i >= argc)
diff --git a/llama.cpp/server/utils.h b/llama.cpp/server/utils.h
index a0a6235c..de8188ec 100644
--- a/llama.cpp/server/utils.h
+++ b/llama.cpp/server/utils.h
@@ -252,6 +252,7 @@ struct llama_server_queue {
}
// Start the main loop. This call is blocking
+ [[noreturn]]
void start_loop() {
while (true) {
// new task arrived