💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Update documentation
davidmezzettic11fc7bf81fd
diff --git a/docs/embeddings/configuration/vectors.md b/docs/embeddings/configuration/vectors.mdindex 23a7e6f8..7fcc557f 100644--- a/docs/embeddings/configuration/vectors.md+++ b/docs/embeddings/configuration/vectors.md@@ -99,17 +99,16 @@ Instruction-based models use prefixes to modify how embeddings are computed. Thi[E5-base](https://huggingface.co/intfloat/e5-base) is an example of a model that accepts instructions. It takes `query: ` and `passage: ` prefixes and uses those to generate embeddings that work well for asymmetric search.-## tokenize+## models```yaml-tokenize: boolean+models: dict```-Enables string tokenization (defaults to false). This method applies tokenization rules that only work with English language text and may increase the quality of-English language sentence embeddings in some situations.+Loads and stores vector models in this cache. This is primarily used with subindexes but can be set on any embeddings instance. This prevents the same model from being loaded multiple times when working with multiple embeddings instances.-## models+## tokenize```yaml-models: dict+tokenize: boolean```-Loads and stores vector models in this cache. This is primarily used with subindexes but can be set on any embeddings instance. This prevents the same model from being loaded multiple times when working with multiple embeddings instances.+Enables string tokenization (defaults to false). This method applies tokenization rules that only work with English language text. It's not recommended for use with recent vector models.