Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Revert "fix(v1/crawl): always use sitemap"

Nicolas339b19ce9d57
diff --git a/apps/api/src/controllers/v1/crawl.ts b/apps/api/src/controllers/v1/crawl.ts
index e49cfe90d..e0883fa8e 100644
--- a/apps/api/src/controllers/v1/crawl.ts
+++ b/apps/api/src/controllers/v1/crawl.ts
@@ -85,7 +85,9 @@ export async function crawlController(
await saveCrawl(id, sc);
- const sitemap = await crawler.tryGetSitemap();
+ const sitemap = sc.crawlerOptions.ignoreSitemap
+ ? null
+ : await crawler.tryGetSitemap();
if (sitemap !== null && sitemap.length > 0) {
let jobPriority = 20;