An open source, non-profit web search engine
Commits on main
Optimise by calling redis less and use our own faster but less accurate URL parsing
d0ec5e3 Daoud Clarke committed 2 years ago
Add a script to update URLs when running locally so we can profile the code
a5e688e Daoud Clarke committed 2 years ago
Temporarily stop recording batch stats to see if it helps with memory usage
3971d24 Daoud Clarke committed 2 years ago
Merge pull request #190 from mwmbl/add-resolved-url-and-anchor-text-when-crawling
37e6cab Daoud Clarke committed 2 years ago