Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
fix: improve robots.txt HTML filtering to check content structure (#1880)
devin-ai-integration[bot]ba3e4cd39fb3
diff --git a/apps/api/src/lib/robots-txt.ts b/apps/api/src/lib/robots-txt.tsindex 468850860..0b601d87c 100644--- a/apps/api/src/lib/robots-txt.ts+++ b/apps/api/src/lib/robots-txt.ts@@ -35,7 +35,7 @@ export async function fetchRobotsTxt((x) => x[0].toLowerCase() === "content-type",) ?? [])[1] ?? "";- if (contentType.includes("text/html") ||+ if ((contentType.includes("text/html") && response.data.trim().startsWith("<")) ||contentType.includes("application/json") ||contentType.includes("application/xml")) {return "";