web search done right
Commits on main
new spell correction is more accurate, so can have lower threshold for suggestions
26b1f08 Mikkel Denker committed 2 years ago
make sure active connections aren't reused from connection pool.
b2dd6c8 Mikkel Denker committed 2 years ago
fixed bug that caused error model in spell correction to always be empty
a1381d6 Mikkel Denker committed 2 years ago
use page centrality directly in crawl planner to prioritise pages.
5c94ded Mikkel Denker committed 2 years ago
no need to run workers independently anymore as they don't need to communicate
bc3fa69 Mikkel Denker committed 2 years ago
reduce communication between workers in approx centrality
d7aee00 Mikkel Denker committed 2 years ago
increase randomness of sampled nodes by using reservoir sampling to sample random nodes from graph
c178243 Mikkel Denker committed 2 years ago
avoid inf values in approximate centrality by pushing norm inside the sum so the values don't get too big
215696e Mikkel Denker committed 2 years ago
close connections in connection pool after a ttl of 60 seconds
9209ad2 Mikkel Denker committed 2 years ago
sort thesaurus entries by their in-degree to prioritize most used definitions
54b7eb1 Mikkel Denker committed 2 years ago
for some reason, poll fn seems to block sometimes when checking if tcpstream is closed. let's instead check by trying to read 0-bytes with a timeout and see if it returns an err
798ef2f Mikkel Denker committed 2 years ago
if queue is empty and batch.len() < BATCH_SIZE, approximate harmonic didn't make progress
e94f02d Mikkel Denker committed 2 years ago
chunk id2node requests into smaller parts to avoid timeout
df7ac44 Mikkel Denker committed 2 years ago
implement distributed version of approximated harmonic centrality the page graph still seems to be too big to calculate the exact centrality even when distributed across multiple workers (need more workers)
823efa6 Mikkel Denker committed 2 years ago
use binary heap for less cmp when merging spell correction dictionaries
d023552 Mikkel Denker committed 2 years ago
each worker in distributed harmonic should only use 1 thread for webgraph as they only need to traverse their edges which is not multithreaded anyway
c9750a8 Mikkel Denker committed 2 years ago
fix logic bug in 'sorted_k' function that caused it to prune wrong elements
1531c7a Mikkel Denker committed 2 years ago
remove retry strategy from ampc and add the missing features to the one from sonic so it can be re-used
0b57dbf Mikkel Denker committed 2 years ago