web search done right
Commits on main
Disable webgraph compression. An insane amount of time was spent decompressing (about 80%) when fetching inbound links. This made it impractical to index backlink texts, which of course is very usefull information to have in the index.
3b8bdc2 Mikkel Denker committed 3 years ago
rkyv serialization in crawl-db to increase performance quite a bit
6226470 Mikkel Denker committed 3 years ago
simplify crawldb by removing all ids and storing urls/domains directly
5868119 Mikkel Denker committed 3 years ago
tune rocksdb options to reduce write amplification in crawl coordinator
5a562b6 Mikkel Denker committed 3 years ago
Remove mutable reference in sonic service handle fn. Each service should handle their required mutability with atomic/locking mechanisms. Otherwise, there would be no way for the sonic server to accept multiple incoming requests even if the server had available capacity to handle the request.
f365ee9 Mikkel Denker committed 3 years ago
Slight change of sonic protocol so close happens from client side. This should allow us to disable lingering for the socket since the client will only close the connection once it has received adequate data to satisfy the user request. No stray tcp packets should therefore hit the socket.
e9301e3 Mikkel Denker committed 3 years ago
Enable spell dictionary by default but reduce worst case compute
5c0487a Mikkel Denker committed 3 years ago
Refactor sonic service to use primitive sonic connections
0b11b20 Mikkel Denker committed 3 years ago
Various different optimizations. This commit should have been structured much better.
4bba8fe Mikkel Denker committed 3 years ago
Url states to disk. The idea is to reduce memory usage significantly. Don't know if it will be fast enough for big crawls.
9d4c01e Mikkel Denker committed 3 years ago