Open-source agentic AI Search Execution Platform. Track AI visibility, find opportunities, and let agents turn signals into prioritized actions and drafted fixes across ChatGPT, Claude, Gemini, Google AI Overviews, Google AI Mode, Perplexity, Grok, and Microsoft Copilot.
TypeScript aeoai-agentai-searchanswer-engine-optimizationbrand-monitoringchatgptclaudecopilotdockergeminigeogoogle-ai-overviewsgrokmcpnextjsopen-sourceperplexityseosupabasetypescript
docs: add custom robots.txt with AI crawler allowances and sitemap reference (#634)
Gökhan Geyik0ed5a97ed2c6
diff --git a/docs/robots.txt b/docs/robots.txtnew file mode 100644index 000000000..4d80ba875--- /dev/null+++ b/docs/robots.txt@@ -0,0 +1,129 @@+# ============================================+# Ansvisor Docs Robots.txt+# ============================================++Sitemap: https://docs.ansvisor.com/sitemap.xml++User-agent: *+Disallow: /cdn-cgi/+Disallow: /_next/+Allow: /_next/image+Allow: /++# --------------------------------------------+# OpenAI+# --------------------------------------------++User-agent: GPTBot+Allow: /++User-agent: OAI-SearchBot+Allow: /++User-agent: ChatGPT-User+Allow: /++# --------------------------------------------+# --------------------------------------------++User-agent: Googlebot+Allow: /++User-agent: Google-Extended+Allow: /++User-agent: GoogleOther+Allow: /++# --------------------------------------------+# Microsoft / Copilot+# --------------------------------------------++User-agent: Bingbot+Allow: /++User-agent: AdIdxBot+Allow: /++# --------------------------------------------+# Anthropic+# --------------------------------------------++User-agent: ClaudeBot+Allow: /++User-agent: Claude-SearchBot+Allow: /++User-agent: Anthropic-ai+Allow: /++# --------------------------------------------+# Perplexity+# --------------------------------------------++User-agent: PerplexityBot+Allow: /++User-agent: Perplexity-User+Allow: /++# --------------------------------------------+# Apple+# --------------------------------------------++User-agent: Applebot+Allow: /++User-agent: Applebot-Extended+Allow: /++# --------------------------------------------+# Common Crawl+# --------------------------------------------++User-agent: CCBot+Allow: /++# --------------------------------------------+# Exa+# --------------------------------------------++User-agent: ExaBot+Allow: /++# --------------------------------------------+# Firecrawl+# --------------------------------------------++User-agent: FirecrawlAgent+Allow: /++# --------------------------------------------+# Amazon+# --------------------------------------------++User-agent: Amazonbot+Allow: /++# --------------------------------------------+# Meta+# --------------------------------------------++User-agent: Meta-ExternalAgent+Allow: /++# --------------------------------------------+# ByteDance+# --------------------------------------------++User-agent: Bytespider+Allow: /++# --------------------------------------------+# You.com+# --------------------------------------------++User-agent: YouBot+Allow: /