web search done right
Commits on main
New spell corrector training. I have just now gotten to the point where we can test this, so it definitely doesn't work yet but must o f the parts should be there.
f30e913 Mikkel Denker committed 2 years ago
Sonic: 1tb bound of body size. Sometimes weird requests would be interpreted as a header with a ridiculous high body size, which would cause an OOM when trying to allocate the buffer.
ca65ea9 Mikkel Denker committed 2 years ago
Deprecate old crawler docs. The crawler architecture has changed tremendously with the planner etc. The docs needs to be updated, but for now we will just hide them.
b096e7c Mikkel Denker committed 2 years ago
mark pages which likely contains ads or paywall in the serp
ff2c0ad Mikkel Denker committed 3 years ago
way faster bitvec similarity calculations by approximating intersection for sites with many inbound links with hyperloglog
63118f1 Mikkel Denker committed 3 years ago
Fix crawl plan. Too many spam urls were being prioritized
3054fe5 Mikkel Denker committed 3 years ago
no need to read edges in crawl planner, enough to know page ids
7ee67e9 Mikkel Denker committed 3 years ago
Make result likes host-level. liking en.wikipedia.org is not the same as liking wikipedia.org
3457231 Mikkel Denker committed 3 years ago