Skip to content

The Unified and Autonomous Development Platform

Updates robots.txt to not allow visiting non informational page

robin shenc838a8bc4045
diff --git a/product/system/site/robots.txt b/product/system/site/robots.txt
index c1ea0f9617..4b2751ff2e 100644
--- a/product/system/site/robots.txt
+++ b/product/system/site/robots.txt
@@ -1,2 +1,10 @@
User-agent: *
-Disallow:
\ No newline at end of file
+Disallow: /*/*/blob
+Disallow: /*/*/commits
+Disallow: /*/*/commit
+Disallow: /*/*/archive
+Disallow: /*/*/branches
+Disallow: /*/*/tags
+Disallow: /*/*/pulls
+Disallow: /*/*/comments
+Disallow: /*/*/compare