Skip to content

web search done right

make sure doubling happens after min in crawl delay. this increases the politeness of the crawler even more, as we now e.g. wait minimum 10 seconds after first 429 response.

Mikkel Denker6af6085ebb80
diff --git a/core/src/crawler/worker.rs b/core/src/crawler/worker.rs
index 6ed56289..a42f070b 100644
--- a/core/src/crawler/worker.rs
+++ b/core/src/crawler/worker.rs
@@ -424,12 +424,14 @@ impl<S: DatumStream> JobExecutor<S> {
let res = self.fetch(url.clone()).await?;
let fetch_time = start.elapsed();
- let mut delay = fetch_time.mul_f32(self.politeness_factor);
+ let mut delay = fetch_time;
if delay < Duration::from_millis(self.config.min_crawl_delay_ms) {
delay = Duration::from_millis(self.config.min_crawl_delay_ms);
}
+ delay = delay.mul_f32(self.politeness_factor);
+
if delay > Duration::from_millis(self.config.max_crawl_delay_ms) {
delay = Duration::from_millis(self.config.max_crawl_delay_ms);
}
diff --git a/frontend/src/routes/settings/optics/+page.svelte b/frontend/src/routes/settings/optics/+page.svelte
index 9a08352b..a0b55d13 100644
--- a/frontend/src/routes/settings/optics/+page.svelte
+++ b/frontend/src/routes/settings/optics/+page.svelte
@@ -35,7 +35,7 @@
if (e instanceof TypeError) {
error = e;
}
- console.error(e);
+ console.error('Failed to add optic', e);
}
};
diff --git a/frontend/src/routes/webmasters/README.md b/frontend/src/routes/webmasters/README.md
index b182b216..d833c40d 100644
--- a/frontend/src/routes/webmasters/README.md
+++ b/frontend/src/routes/webmasters/README.md
@@ -7,7 +7,7 @@ The crawler uses the user agent `Mozilla/5.0 (compatible; StractBot/0.1; open so
## Politeness
StractBot is a polite crawler. It respects the [robots.txt](https://en.wikipedia.org/wiki/Robots.txt) file of the website it is crawling and tries to not overload the server.
-It does this by waiting a certain amount of time between requests to the same domain. The waiting time is calculated by _min(max(fetchtime \* politeness, 5 sec), 60 sec)_ where _fetchtime_ is the time it took to fetch the page. The crawler will wait at least 5 sec between requests and at most 60 seconds.
+It does this by waiting a certain amount of time between requests to the same domain. The waiting time is calculated by _min(politeness \* max(fetchtime, 5 sec), 60 sec)_ where _fetchtime_ is the time it took to fetch the page. The crawler will wait at least 5 sec between requests and at most 60 seconds.
This dynamic waiting time tries to prevent us from disrupting servers that cannot handle the load, while not giving unnecessary politeness to servers that can. The politeness factor starts at 1 and is doubled every time the crawler gets a 429 response from the server (to at most 2048).
The crawler looks for the token `StractBot` in the robots.txt file to determine which pages (if any) the crawler is allowed to crawl.