web search done right
Commits on main
increase coefficient for update_timestamp to show more recent articles from live index
e16e353 Mikkel Denker committed 2 years ago
add smoothing factor to timestamp to give it a half life of 3 days
c7ea2fd Mikkel Denker committed 2 years ago
re-open live index after compaction to reflect segment changes
bc45409 Mikkel Denker committed 2 years ago
harmonic centrality nearest neighbor calculation that uses the harmonic centrality of the highest neighbors node as a seed node proxy for the centrality of that node (with a discount factor)
4915160 Mikkel Denker committed 2 years ago
refactor GetSiteUrls into a response stream for reusability
9fc0293 Mikkel Denker committed 2 years ago
enforce edgelimit to only be applied in webgraph mod.rs so offset is only applied once
f6bca07 Mikkel Denker committed 2 years ago
implement EdgeLimit::LimitAndOffset in webgraph. to make sure the edges are returned in sorted order, we implement a new FlatSortedBy struct that takes a vec of iters and outputs the sorted order of the iter items. it assumes that each iter is already sorted so it doesn't have to traverse the entire result set
9414c42 Mikkel Denker committed 2 years ago
make sure no-follow links aren't used in /explore and to score host similarity
d533133 Mikkel Denker committed 2 years ago
only consider simhash duplicates if we don't have enough non-duplicates
c4ea75a Mikkel Denker committed 2 years ago
group urls by domain centrality in planner and process groups by highest centrality first
c5a465b Mikkel Denker committed 2 years ago
cleanup temporary directories automatically in tests (#228)
4e8c165 Mikkel Denker committed 2 years ago
Re-open live index after it has been downloaded from replica (#227)
950862b Mikkel Denker committed 2 years ago
normalize urls before they are crawled for better duplicates detection
0114e8d Mikkel Denker committed 2 years ago
skip awaiting webgraph nodes in dev environments as they may not exist
7814bf7 Mikkel Denker committed 2 years ago