Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: log warning when Sitemap.load returns empty sitemap due to loading error (#3462)

Copilot2cf9c1cd9fca
diff --git a/packages/utils/src/internals/sitemap.ts b/packages/utils/src/internals/sitemap.ts
index 9280d4a40..cc74ca776 100644
--- a/packages/utils/src/internals/sitemap.ts
+++ b/packages/utils/src/internals/sitemap.ts
@@ -432,7 +432,8 @@ export class Sitemap {
for await (const item of parseSitemap(sources, proxyUrl, parseSitemapOptions)) {
urls.push(item.loc);
}
- } catch {
+ } catch (e) {
+ log.warning(`Sitemap.load: Failed to load sitemap, returning empty result. (${e})`);
return new Sitemap([]);
}