web search done right
Commits on main
skip webgraph during search if there are no liked/disliked sites in query (no ranking signals would currently need the backlinks in that case)
c154093 Mikkel Denker committed 2 years ago
make sure webgraph merge doesn not exceed maximum number of allowed documents in tantivy
13a06b3 Mikkel Denker committed 2 years ago
when https version of robots.txt is 404 but http is unreachable, it should overall be treated as a 404 as the website has most likely just redirected all http traffic to https
3e732e0 Mikkel Denker committed 2 years ago
make sure urls discovered in crawler are deduplicated (extra precaution)
8df44a5 Mikkel Denker committed 2 years ago
centrality store in webgraph creation is now raw centrality instead of rank
dda40bd Mikkel Denker committed 2 years ago
[live_crawler] shuffle sites between each tick to make site prioritisation more fair
0b1f6ea Mikkel Denker committed 2 years ago
deduplicate cluster members to remove services that have been restarted
f6cd2e0 Mikkel Denker committed 2 years ago
[live_crawler] make sure frontpage check is skipped if disallowed in robots.txt
c662fe2 Mikkel Denker committed 2 years ago
[live_crawler] make sure unreachable robots.txt files aren't re-fetched
34230b5 Mikkel Denker committed 2 years ago
remove pages_by_host from webgraph as it isn't used anymore
2519bef Mikkel Denker committed 2 years ago
improve compaction performance in live index by performing initial segment merge on a read lock and only switch to the new segments on a write lock. this ensures that search requests can still be performed while the heavy part of merge is executing
3a127af Mikkel Denker committed 2 years ago
bump edge limit for ingoing nodes to 128 in SimilarHostsFinder
847c142 Mikkel Denker committed 2 years ago
index compaction shouldn’t update creation date as this would cause all segments to eventaully have same creation date
87fbd3d Mikkel Denker committed 2 years ago
make sure live index node doesn't try to replicate to itself
2492ebd Mikkel Denker committed 2 years ago