web search done right
Commits on main
Match webage terms across fields. If a webpage matches TermA in url and TermB in body it should still be a correct match even though no single field has both TermB and TermA.
e6c88d6 Mikkel Denker committed 4 years ago
Custom tantivy query implementation. Preparations for expanded spans ranking which allows us to rank documents based on term proximity. Overall this change gives more control over the search+ranking procedure.
d5efc66 Mikkel Denker committed 4 years ago
Initial JustText algorithm. Generate better snippets by guessing which parts of the website viable content. This also improves search accuracy as the BM25 score becomes way better (it is given cleaner data). Performance could probably be immensly improved, but it seems to work.
fda6c16 Mikkel Denker committed 4 years ago
Custom HTML lexer based on logos. There is no need to parse the entire html file into a tree since a stream of tags will be sufficient for extracting links+text. This should be faster than parsing. It wil also make it easier to implement the 'just_text' algorithm.
c500c0e Mikkel Denker committed 4 years ago
Separate stemmed and non-stemmed fields. This allows us to prioritize non-stemmed matches over stemmed matches.
0632110 Mikkel Denker committed 4 years ago
Merge all segments at the end of map, don't merge automatically. This hugely increases the indexing performance as we don't merge as often.
21149d8 Mikkel Denker committed 4 years ago
Very simple searching working. Still a lot of bugs to figure out, but it actually works
085d6ca Mikkel Denker committed 4 years ago
Indexing using mapreduce, centrality store, mapreduce worker state
da6c750 Mikkel Denker committed 4 years ago
Merge search indexes and make indexes serializable by freezing
ec0385b Mikkel Denker committed 4 years ago
limit warc files during webgraph building and other minor changes
4164cd3 Mikkel Denker committed 4 years ago