Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
fix(crawler): treat XML files as sitemaps (temporarily)
Gergő Móricz350d00d27a6c
diff --git a/apps/api/src/scraper/WebScraper/crawler.ts b/apps/api/src/scraper/WebScraper/crawler.tsindex 3fe53e4da..c00fed78a 100644--- a/apps/api/src/scraper/WebScraper/crawler.ts+++ b/apps/api/src/scraper/WebScraper/crawler.ts@@ -362,7 +362,7 @@ export class WebCrawler {};- const sitemapUrl = url.endsWith("/sitemap.xml")+ const sitemapUrl = url.endsWith(".xml")? url: `${url}/sitemap.xml`;