Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
fix(crawler): sitemaps poisoning crawls with unrelated links (#1334)
Gergő Móriczc7ae50d2d0be
diff --git a/apps/api/src/scraper/WebScraper/crawler.ts b/apps/api/src/scraper/WebScraper/crawler.tsindex ba4793d87..29a35e3bf 100644--- a/apps/api/src/scraper/WebScraper/crawler.ts+++ b/apps/api/src/scraper/WebScraper/crawler.ts@@ -271,7 +271,7 @@ export class WebCrawler {return urlsHandler(urls);} else {let filteredLinks = this.filterLinks(- [...new Set(urls)],+ [...new Set(urls)].filter(x => this.filterURL(x, this.initialUrl) !== null),leftOfLimit,this.maxCrawledDepth,fromMap,