Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

fix: improve robots.txt HTML filtering to check content structure (#1880)

devin-ai-integration[bot]ba3e4cd39fb3
diff --git a/apps/api/src/lib/robots-txt.ts b/apps/api/src/lib/robots-txt.ts
index 468850860..0b601d87c 100644
--- a/apps/api/src/lib/robots-txt.ts
+++ b/apps/api/src/lib/robots-txt.ts
@@ -35,7 +35,7 @@ export async function fetchRobotsTxt(
(x) => x[0].toLowerCase() === "content-type",
) ?? [])[1] ?? "";
- if (contentType.includes("text/html") ||
+ if ((contentType.includes("text/html") && response.data.trim().startsWith("<")) ||
contentType.includes("application/json") ||
contentType.includes("application/xml")) {
return "";