web search done right
Commits on main
Remove online/personal centrality. It simply wasn't good enough and was too computationally expensive. It is better to have a faster search experience and only rely on inbound similarity for the personalization part, as this tends to work quite well while also being way simpler. Kill your darlings and all that.
498e78b Mikkel Denker committed 3 years ago
added log messages to indicate where we are in centrality calculations
db7abf3 Mikkel Denker committed 3 years ago
try rkyv serialization in crawl coordinator to hopefully fix oom error
bdd9d41 Mikkel Denker committed 3 years ago
there seems to be a bug in bincode that causes an oom crash with an extreme amount of bytes tried allocated
142ddeb Mikkel Denker committed 3 years ago
Each worker now fetches 2*num_workers jobs. This will decrease the pressure on the coordinator
565e7e1 Mikkel Denker committed 3 years ago
Significantly decrease memory usage by storing url states on disk. Hopefully this will not hurt performance too much. It would allow us to make much larger crawls
864fe3a Mikkel Denker committed 3 years ago
Disable WAL in rocksdb for crawl coordinator. We don't care about data loss in this part. If a crawl coordinator fails, we will have to restart the crawl anyway. Full speeeeeeed
a6e7d85 Mikkel Denker committed 3 years ago
Don't show alice settings when alice is disabled. That would just be confusing, as the user might not know what alice is
14b9229 Mikkel Denker committed 3 years ago
Remove multiple warc writers again. It didn't seem to actually utilize the warc writers concurrently. It is easier to just spawn multiple workers on the same node in different tmux windows.
487552c Mikkel Denker committed 3 years ago
Remove ttl from robots.txt Makes sure that we don't retry download in any case
8331a77 Mikkel Denker committed 3 years ago
Re-wrote crawldb to be backed by rocksdb This should significantly increase throughput and at the very least allow for more performance tuning. Hopefully the crawl coordinator is no longer the bottleneck
a36a826 Mikkel Denker committed 3 years ago
less indexes to hopefully increase insert performance in crawldb
5b645e5 Mikkel Denker committed 3 years ago