Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
fix(api): sitemap max age for map requests
delong3ac3997a79762
diff --git a/apps/api/src/scraper/WebScraper/crawler.ts b/apps/api/src/scraper/WebScraper/crawler.tsindex 0cf1f45a6..da4f6b1ff 100644--- a/apps/api/src/scraper/WebScraper/crawler.ts+++ b/apps/api/src/scraper/WebScraper/crawler.ts@@ -494,10 +494,7 @@ export class WebCrawler {);});- // Allow sitemaps to be cached for 48 hours if they are requested from /map- // - mogery- const maxAge = fromMap && !onlySitemap ? 48 * 60 * 60 * 1000 : 0;-+ const maxAge = 48 * 60 * 60 * 1000;try {const robotsSitemaps = this.robots.getSitemaps();this.logger.debug("Attempting to fetch sitemap links", {