Skip to content

Open Source alternative to Algolia + Pinecone and an Easier-to-Use alternative to ElasticSearch ⚡ 🔍 ✨ Fast, typo tolerant, in-memory fuzzy Search Engine for building delightful search experiences

Limit token size to handle bad data.

Kishore Nallan15d3da7203e3
diff --git a/src/index.cpp b/src/index.cpp
index 584ad2f9..c2595a20 100644
--- a/src/index.cpp
+++ b/src/index.cpp
@@ -1146,6 +1146,10 @@ void Index::tokenize_string_with_facets(const std::string& text, bool is_facet,
continue;
}
+ if(token.size() > 100) {
+ token.erase(100);
+ }
+
token_to_offsets[token].push_back(token_index + 1);
last_token = token;
@@ -1191,6 +1195,10 @@ void Index::tokenize_string_array_with_facets(const std::vector<std::string>& st
continue;
}
+ if(token.size() > 100) {
+ token.erase(100);
+ }
+
token_to_offsets[token].push_back(token_index + 1);
token_set.insert(token);
last_token = token;