web search done right
Commits on main
Fix url root domain parsing and faster memory mapped crawldb.
ce1e15a Mikkel Denker committed 3 years ago
Deunsafication and testing of `sonic` protocol and introducing `sonic::service` (#72)
345bd9c Oliver Bøving committed 3 years ago
rocksdb bloom filters caused OOM errors. Updated filter sizes to more sane defaults
2e885e9 Mikkel Denker committed 3 years ago
less clones when inserting results into crawl coordinator
0eec601 Mikkel Denker committed 3 years ago
Some domain states got set to "Pending" even though a worker was already crawling them. This primarily happened for popular sites that has many incoming links from many different sites.
69e06bb Mikkel Denker committed 3 years ago
reduce number of connections from crawl workers to coordinator
93dcb50 Mikkel Denker committed 3 years ago
if '#' is before '?' in an url, the query part is simply from '?' to url.len()
f4cc94f Mikkel Denker committed 3 years ago
Moved some hardcoded constants into configuration files. Also a bunch of other stuff; I should really have split this commit into multiple parts...
4a2ca14 Mikkel Denker committed 3 years ago
Change user agent of crawler. Reddit seems to look for "bot" in the user agent. If they cannot find the substring, they return a page that updates the title with javascript. This causes us to have a bunch of reddit pages with the title "Reddit - Dive into anything" in the search results.
a545770 Mikkel Denker committed 3 years ago
Improve read performance for webgraph. We have split adjacency into a small_adjacency and full_adjacency (same for reverse) rocksdb databases. This allows us to read edge labels when we need them, which will increase the read performance for the webgraph edges at the expense of storage usage.
d9b0fd7 Mikkel Denker committed 3 years ago