Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

fix: sitemaps support `application/xml` (#2408)

Jindřich Bärcbcf47a7b991
diff --git a/packages/utils/src/internals/robots.ts b/packages/utils/src/internals/robots.ts
index 30a69e519..fd390dd7a 100644
--- a/packages/utils/src/internals/robots.ts
+++ b/packages/utils/src/internals/robots.ts
@@ -14,7 +14,7 @@ let HTTPError: typeof HTTPErrorClass;
* **Example usage:**
* ```javascript
* // Load the robots.txt file
- * const robots = await RobotsFile.load('https://crawlee.dev/docs/introduction/first-crawler');
+ * const robots = await RobotsFile.find('https://crawlee.dev/docs/introduction/first-crawler');
*
* // Check if a URL should be crawled according to robots.txt
* const url = 'https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler';
@@ -46,7 +46,7 @@ export class RobotsFile {
}
/**
- * Allows providing the URL and robotx.txt content explicitly instead of loading it from the target site.
+ * Allows providing the URL and robots.txt content explicitly instead of loading it from the target site.
* @param url the URL for robots.txt file
* @param content contents of robots.txt
* @param [proxyUrl] a proxy to be used for fetching the robots.txt file
diff --git a/packages/utils/src/internals/sitemap.ts b/packages/utils/src/internals/sitemap.ts
index 3c9f89030..973d25ff0 100644
--- a/packages/utils/src/internals/sitemap.ts
+++ b/packages/utils/src/internals/sitemap.ts
@@ -169,7 +169,7 @@ export class Sitemap {
const parser = (() => {
const contentType = sitemapStream.response!.headers['content-type'];
- if (contentType === 'text/xml' || sitemapUrl.endsWith('.xml')) {
+ if (['text/xml', 'application/xml'].includes(contentType ?? '') || sitemapUrl.endsWith('.xml')) {
return Sitemap.createXmlParser(parsingState, () => resolve(undefined), reject);
}