Skip to content

web search done right

Commits on main

rust update and new git screenshot
Mikkel Denker committed 3 years ago
330c02a
3f69e3f
less approximations in similar sites finder
Mikkel Denker committed 3 years ago
c68c7ca
Remove www prefix when inserting into web graph
Mikkel Denker committed 3 years ago
c392f68
Retry urls that return 429 during crawl
Mikkel Denker committed 3 years ago
43bd806
1af9433
aa35730
test hardening
Mikkel Denker committed 3 years ago
62d8a57
url digits and slashes are computable before search
Mikkel Denker committed 3 years ago
d9bf123
number of slashes and digits in url as ranking signal
Mikkel Denker committed 3 years ago
1e2ee6b
formating and raw http for s3 endpoint
Mikkel Denker committed 3 years ago
4f1a707
5f73794
Normalize redirects in web graph
Mikkel Denker committed 3 years ago
af1e432
rename webgraph url level to page level
Mikkel Denker committed 3 years ago
ec76b7f
Lower domain weight after the urls have been sampled
Mikkel Denker committed 3 years ago
e858c00
harmonic centrality exact counting threshold
Mikkel Denker committed 3 years ago
0ea90ae
configurable webgraph level
Mikkel Denker committed 3 years ago
6aa3766
only search single replica with retries
Mikkel Denker committed 3 years ago
bf68865
f95271b
insert crawler job responses in batches
Mikkel Denker committed 3 years ago
e1c16e3
simplify ranking pipeline and fix offsets
Mikkel Denker committed 3 years ago
def23ac
use http instead of https as default protocol
Mikkel Denker committed 3 years ago
cd3a65f
fixed crawler sampling
Mikkel Denker committed 3 years ago
cb0e859