Skip to content

armgit / txtai

Public Imported from GitHub
0

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

Add GGML ANN documentation

davidmezzettid64c8d8f274b
diff --git a/docs/embeddings/configuration/ann.md b/docs/embeddings/configuration/ann.md
index 8acbc227..65c87bb2 100644
--- a/docs/embeddings/configuration/ann.md
+++ b/docs/embeddings/configuration/ann.md
@@ -4,7 +4,7 @@ Approximate Nearest Neighbor (ANN) index configuration for storing vector embedd
## backend
```yaml
-backend: faiss|hnsw|annoy|numpy|torch|pgvector|sqlite|custom
+backend: faiss|hnsw|annoy|ggml|numpy|torch|pgvector|sqlite|custom
```
Sets the ANN backend. Defaults to `faiss`. Additional backends are available via the [ann](../../../install/#ann) extras package. Set custom backends via setting this parameter to the fully resolvable class string.
@@ -62,6 +62,18 @@ annoy:
See [Annoy documentation](https://github.com/spotify/annoy#full-python-api) for more information on these parameters. Note that annoy indexes can not be modified after creation, upserts/deletes and other modifications are not supported.
+### ggml
+```yaml
+ggml:
+ gpu: enable GPU - defaults to True
+ quantize: sets the tensor quantization - defaults to F32
+ querysize: query buffer size - defaults to 64
+```
+
+The [GGML](https://github.com/ggml-org/ggml) backend is a k-nearest neighbors backend. It stores tensors using GGML and [GGUF](https://huggingface.co/docs/hub/en/gguf). It supports GPU-enabled operations and supports quantization. GGML is the framework used by [llama.cpp](https://github.com/ggml-org/llama.cpp).
+
+[See this](https://github.com/ggml-org/ggml/blob/master/include/ggml.h#L379) for a list of quantization types.
+
### numpy
The NumPy backend is a k-nearest neighbors backend. It's designed for simplicity and works well with smaller datasets that fit into memory.