Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

fix(crawler): sitemaps poisoning crawls with unrelated links (#1334)

Gergő Móriczc7ae50d2d0be
diff --git a/apps/api/src/scraper/WebScraper/crawler.ts b/apps/api/src/scraper/WebScraper/crawler.ts
index ba4793d87..29a35e3bf 100644
--- a/apps/api/src/scraper/WebScraper/crawler.ts
+++ b/apps/api/src/scraper/WebScraper/crawler.ts
@@ -271,7 +271,7 @@ export class WebCrawler {
return urlsHandler(urls);
} else {
let filteredLinks = this.filterLinks(
- [...new Set(urls)],
+ [...new Set(urls)].filter(x => this.filterURL(x, this.initialUrl) !== null),
leftOfLimit,
this.maxCrawledDepth,
fromMap,