Skip to content

Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.

docs: add custom robots.txt with AI crawler allowances and sitemap reference (#634)

Gökhan Geyik0ed5a97ed2c6
diff --git a/docs/robots.txt b/docs/robots.txt
new file mode 100644
index 000000000..4d80ba875
--- /dev/null
+++ b/docs/robots.txt
@@ -0,0 +1,129 @@
+# ============================================
+# Ansvisor Docs Robots.txt
+# ============================================
+
+Sitemap: https://docs.ansvisor.com/sitemap.xml
+
+User-agent: *
+Disallow: /cdn-cgi/
+Disallow: /_next/
+Allow: /_next/image
+Allow: /
+
+# --------------------------------------------
+# OpenAI
+# --------------------------------------------
+
+User-agent: GPTBot
+Allow: /
+
+User-agent: OAI-SearchBot
+Allow: /
+
+User-agent: ChatGPT-User
+Allow: /
+
+# --------------------------------------------
+# Google
+# --------------------------------------------
+
+User-agent: Googlebot
+Allow: /
+
+User-agent: Google-Extended
+Allow: /
+
+User-agent: GoogleOther
+Allow: /
+
+# --------------------------------------------
+# Microsoft / Copilot
+# --------------------------------------------
+
+User-agent: Bingbot
+Allow: /
+
+User-agent: AdIdxBot
+Allow: /
+
+# --------------------------------------------
+# Anthropic
+# --------------------------------------------
+
+User-agent: ClaudeBot
+Allow: /
+
+User-agent: Claude-SearchBot
+Allow: /
+
+User-agent: Anthropic-ai
+Allow: /
+
+# --------------------------------------------
+# Perplexity
+# --------------------------------------------
+
+User-agent: PerplexityBot
+Allow: /
+
+User-agent: Perplexity-User
+Allow: /
+
+# --------------------------------------------
+# Apple
+# --------------------------------------------
+
+User-agent: Applebot
+Allow: /
+
+User-agent: Applebot-Extended
+Allow: /
+
+# --------------------------------------------
+# Common Crawl
+# --------------------------------------------
+
+User-agent: CCBot
+Allow: /
+
+# --------------------------------------------
+# Exa
+# --------------------------------------------
+
+User-agent: ExaBot
+Allow: /
+
+# --------------------------------------------
+# Firecrawl
+# --------------------------------------------
+
+User-agent: FirecrawlAgent
+Allow: /
+
+# --------------------------------------------
+# Amazon
+# --------------------------------------------
+
+User-agent: Amazonbot
+Allow: /
+
+# --------------------------------------------
+# Meta
+# --------------------------------------------
+
+User-agent: Meta-ExternalAgent
+Allow: /
+
+# --------------------------------------------
+# ByteDance
+# --------------------------------------------
+
+User-agent: Bytespider
+Allow: /
+
+# --------------------------------------------
+# You.com
+# --------------------------------------------
+
+User-agent: YouBot
+Allow: /