web search done right
Commits on main
skip links to/from same domain when calculating harmonic centralities
e4ae264 Mikkel Denker committed 2 years ago
move last worker folder into correct output location after indexing
de5c946 Mikkel Denker committed 2 years ago
simplified bm25f that uses same IDF weight across all fields
5beae3b Mikkel Denker committed 2 years ago
[crawler] don't re-crawl url when it redirects to another url
ca9b249 Mikkel Denker committed 2 years ago
[crawler] some sites seem to only host robots.txt files on the 'www.' subdomain without redirects
2376f33 Mikkel Denker committed 2 years ago
[crawler] continue wandering while budget > 0 and we still know about uncrawled urls
07369e1 Mikkel Denker committed 2 years ago
remove some telemetry hosts from list of ad servers so pages aren't mistakenly tagged as containing ads (#209)
a1c9721 Mikkel Denker committed 2 years ago
ability to convert some of the wander budget to scheduled urls
efa6f9f Mikkel Denker committed 2 years ago
optionally merge all webgraph segments into a single segment for improved read performance
817bda9 Mikkel Denker committed 2 years ago
store sort-key in webgraph edges to properly apply limits with multiple segments
295444d Mikkel Denker committed 2 years ago
no need to consider nodes where we have found a better path
07663c1 Mikkel Denker committed 2 years ago
no need to consider nodes where we have found a better path
90980a1 Mikkel Denker committed 2 years ago
simplify dijkstra to reduce allocations and btree searches
6bc7a5c Mikkel Denker committed 2 years ago
optionally store pages with 0 centrality from approx harmonic
4fbffe6 Mikkel Denker committed 2 years ago
reduce indirections in dht (and hopefully memory usage) by removing some arc's
4b459b5 Mikkel Denker committed 2 years ago
reduce memory usage and serialization/deserialization in dht by storing keys and values as enums instead of vec<u8>
f7a3545 Mikkel Denker committed 2 years ago