Skip to content

armgit / txtai

Public Imported from GitHub
0

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

Change tokenization to default to false for TransformersVectors, closes #99

davidmezzetti246e4449f845
diff --git a/src/python/txtai/vectors/base.py b/src/python/txtai/vectors/base.py
index 45dc8cfd..a56a0343 100644
--- a/src/python/txtai/vectors/base.py
+++ b/src/python/txtai/vectors/base.py
@@ -18,7 +18,7 @@ class Vectors:
self.initialized = "dimensions" in config
# Enables optional string tokenization
- self.tokenize = "tokenize" not in config or config["tokenize"]
+ self.tokenize = config.get("tokenize")
# pylint: disable=E1111
self.model = self.load(config["path"])