Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

fix(api): sitemap max age for map requests

delong3ac3997a79762
diff --git a/apps/api/src/scraper/WebScraper/crawler.ts b/apps/api/src/scraper/WebScraper/crawler.ts
index 0cf1f45a6..da4f6b1ff 100644
--- a/apps/api/src/scraper/WebScraper/crawler.ts
+++ b/apps/api/src/scraper/WebScraper/crawler.ts
@@ -494,10 +494,7 @@ export class WebCrawler {
);
});
- // Allow sitemaps to be cached for 48 hours if they are requested from /map
- // - mogery
- const maxAge = fromMap && !onlySitemap ? 48 * 60 * 60 * 1000 : 0;
-
+ const maxAge = 48 * 60 * 60 * 1000;
try {
const robotsSitemaps = this.robots.getSitemaps();
this.logger.debug("Attempting to fetch sitemap links", {