Open Source alternative to Algolia + Pinecone and an Easier-to-Use alternative to ElasticSearch ⚡ 🔍 ✨ Fast, typo tolerant, in-memory fuzzy Search Engine for building delightful search experiences
C++ algoliadatastoreelasticsearchenterprise-searchfacetingfull-text-searchfuzzy-searchgeosearchin-memoryinstantsearchmerchandisingpineconesearchsearch-enginesemantic-searchsimilarity-searchsite-searchsynonymstypo-tolerancevector-search
Limit token size to handle bad data.
Kishore Nallan15d3da7203e3
diff --git a/src/index.cpp b/src/index.cppindex 584ad2f9..c2595a20 100644--- a/src/index.cpp+++ b/src/index.cpp@@ -1146,6 +1146,10 @@ void Index::tokenize_string_with_facets(const std::string& text, bool is_facet,continue;}+ if(token.size() > 100) {+ token.erase(100);+ }+token_to_offsets[token].push_back(token_index + 1);last_token = token;@@ -1191,6 +1195,10 @@ void Index::tokenize_string_array_with_facets(const std::vector<std::string>& stcontinue;}+ if(token.size() > 100) {+ token.erase(100);+ }+token_to_offsets[token].push_back(token_index + 1);token_set.insert(token);last_token = token;