Skip to content

web search done right

detail how to exclude stractbot using robots.txt

Mikkel Denkerc43da8cf5c0f
diff --git a/frontend/src/routes/webmasters/README.md b/frontend/src/routes/webmasters/README.md
index 4a12f379..2ba693ef 100644
--- a/frontend/src/routes/webmasters/README.md
+++ b/frontend/src/routes/webmasters/README.md
@@ -32,7 +32,24 @@ If the crawler hasn't received any 429 responses from a server, it may gradually
### Robots.txt
-The crawler looks for the token StractBot in the robots.txt file to determine which pages (if any) it is allowed to crawl. The robots.txt file is cached for 1 hour, so changes to the file should be respected quite quickly.
+The crawler looks for the token StractBot in the [robots.txt](https://www.robotstxt.org/about.html) file to determine which pages (if any) it is allowed to crawl.
+
+If you want to restrict access to part of your site, add the following to your robots.txt file
+
+```
+User-agent: StractBot
+Disallow: /private
+```
+
+This will ensure that StractBot doesn't access any page with the '/private' prefix.
+You can also restrict StractBot from accessing any page on your site
+
+```
+User-agent: StractBot
+Disallow: /
+```
+
+The robots.txt file is cached for 1 hour, so changes to the file should be respected fairly quickly.
## Contact us