web search done right
Change user agent of crawler. Reddit seems to look for "bot" in the user agent. If they cannot find the substring, they return a page that updates the title with javascript. This causes us to have a bunch of reddit pages with the title "Reddit - Dive into anything" in the search results.
Mikkel Denkera545770b6e4f
diff --git a/frontend/src/pages/crawler.mdx b/frontend/src/pages/crawler.mdxindex e1292e77..7cef6788 100644--- a/frontend/src/pages/crawler.mdx+++ b/frontend/src/pages/crawler.mdx@@ -4,14 +4,14 @@ title: Stract Crawler---# Stract Crawler-StractSearch is the name of our crawler that is used to collect pages from the web in order to index them in the search engine. It is written in Rust and the source code can be found [here](https://github.com/StractOrg/stract/tree/main/core/src/crawler). The crawler uses the user agent `Mozilla/5.0 (compatible; StractSearch/0.1; +https://trystract.com/webmasters)`.+Stract is an open source search engine. StractBot is the name of our crawler that is used to collect pages from the web in order to index them. It is written in Rust and the source code can be found [here](https://github.com/StractOrg/stract/tree/main/core/src/crawler). The crawler uses the user agent `Mozilla/5.0 (compatible; StractBot/0.1; +https://trystract.com/webmasters)`.## Politeness-StractSearch is a polite crawler. It respects the [robots.txt](https://en.wikipedia.org/wiki/Robots.txt) file of the website it is crawling and tries to not overload the server.+StractBot is a polite crawler. It respects the [robots.txt](https://en.wikipedia.org/wiki/Robots.txt) file of the website it is crawling and tries to not overload the server.It does this by waiting a certain amount of time between requests to the same domain. The waiting time is calculated by *min(max(fetchtime * politeness, 200 ms), 30 sec)* where *fetchtime* is the time it took to fetch the page. The crawler will wait at least 200 ms between requests and at most 30 seconds.This dynamic waiting time tries to prevent us from disrupting servers that cannot handle the load, while not giving unnecessary politeness to servers that can. The politeness factor starts at 1 and is doubled every time the crawler gets a 429 response from the server (to at most 1024).-The crawler looks for the token `StractSearch` in the robots.txt file to determine which pages (if any) the crawler is allowed to crawl.+The crawler looks for the token `StractBot` in the robots.txt file to determine which pages (if any) the crawler is allowed to crawl.## Contact us