web search done right
Commits on main
Refactor inverted index to it's own struct. Extra information such as favicon and other images are now separated from the inverted index and everything is stored in the index. This also allows us to easier keep a list of all pending image download jobs and execute these asynchronously (which we know do).
b07c675 Mikkel Denker committed 4 years ago
Index (almost) all text on page. Only use clean text for snippets unless absolutely necesarry.
7f42601 Mikkel Denker committed 4 years ago
Fixed bug that caused some queries to panic. Since we didn't check if the positions[i] vector had elements, some searches caused the weights and positions vector to have a different number of elements, thereby crashing on .zip_eq
7e31e98 Mikkel Denker committed 4 years ago
Stemmed terms now show up in the generated snippets. We use the language of the webpage to determine which stemmer to use.
0ed6bc2 Mikkel Denker committed 4 years ago