web search done right
Commits on main
each inbound link is now scaled by host centrality in derived_centrality. This should make the centrality metric more robust to linkbuilders
32d94b9 Mikkel Denker committed 3 years ago
Derived centrality. New centrality metric that is derived from the host harmonic centrality. This can hopefully be calculated on the enormous page webgraph.
4c49e2d Mikkel Denker committed 3 years ago
BTreeMap instead of DashMap in harmonic centrality calculation. Should reduce memory consumption but cannot be parallel
b3608f2 Mikkel Denker committed 3 years ago
Some linkbuilders buy a lot of domains. Relying on root domain is insufficient
a4dbe72 Mikkel Denker committed 3 years ago
Improve crawldb insert performance. Introduce a limit to maximum number of newly discovered domains per crawled domain. Also enables 'no_slowdown' for the inserts, even though the LOGs don't indicate that any write stalls is happening.
bb21f41 Mikkel Denker committed 3 years ago
metric to probabilistically estimate number of daily unique users
e7fa85f Mikkel Denker committed 3 years ago
Make result counts optional. This allows us to short circuit the query when we have reached max_docs_considered at the query level instead of at the collector level. This should increase search performance since the searcher then doesn't have to iterate all the results in order to return a count.
0353391 Mikkel Denker committed 3 years ago
Moved crawler seed urls to router. This ensures the correct coordinators gets the correct urls based on hash(domain) % coordinators.len()
b044f25 Mikkel Denker committed 3 years ago