Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

added docs.

Petr Patek71ba6338cfa4
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index 5979f83e6..f9a262c8f 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -261,17 +261,35 @@ The exceptions are logged to the request using the
and <a href="systemstatus"><code>SystemStatus</code></a> defaults are provided to account for the fact that <code>cheerio</code>
parses HTML synchronously and therefore blocks the event loop.</p>
</td></tr><tr>
-<td><code>[options.minConcurrency]</code></td><td><code>Object</code></td><td><code>1</code></td>
+<td><code>[options.minConcurrency]</code></td><td><code>Number</code></td><td><code>1</code></td>
</tr>
<tr>
<td colspan="3"><p>Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
<p> <em>WARNING:</em> If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
If you&#39;re not sure, just keep the default value and the concurrency will scale up automatically.</p>
</td></tr><tr>
-<td><code>[options.maxConcurrency]</code></td><td><code>Object</code></td><td><code>1000</code></td>
+<td><code>[options.maxConcurrency]</code></td><td><code>Number</code></td><td><code>1000</code></td>
</tr>
<tr>
<td colspan="3"><p>Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding <a href="autoscaledpool"><code>AutoscaledPool</code></a> option.</p>
+</td></tr><tr>
+<td><code>[options.useSessionPool]</code></td><td><code>Boolean</code></td><td><code>false</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
+ It also marks Session as bad after a request timeout.</p>
+</td></tr><tr>
+<td><code>[options.sessionPoolOptions]</code></td><td><code>Object</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Custom options passed to the underlying <a href="sessionpool"><code>SessionPool</code></a> constructor.</p>
+</td></tr><tr>
+<td><code>[options.persistCookiesPerSession]</code></td><td><code>Boolean</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Automatically saves cookies to Session. Works only if Session Pool is used.</p>
+<p> It parses cookie from response &quot;set-cookie&quot; header saves or updates cookies for session and once the session is used for next request.
+ It passes the &quot;Cookie&quot; header to the request with the session cookies.</p>
</td></tr></tbody>
</table>
<a name="CheerioCrawler+run"></a>
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index 1aaf314cd..2532f1161 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -255,13 +255,23 @@ const DEFAULT_OPTIONS = {
* are provided by `CheerioCrawler` and cannot be overridden. Reasonable {@link Snapshotter}
* and {@link SystemStatus} defaults are provided to account for the fact that `cheerio`
* parses HTML synchronously and therefore blocks the event loop.
- * @param {Object} [options.minConcurrency=1]
+ * @param {Number} [options.minConcurrency=1]
* Sets the minimum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
*
* *WARNING:* If you set this value too high with respect to the available system memory and CPU, your crawler will run extremely slow or crash.
* If you're not sure, just keep the default value and the concurrency will scale up automatically.
- * @param {Object} [options.maxConcurrency=1000]
+ * @param {Number} [options.maxConcurrency=1000]
* Sets the maximum concurrency (parallelism) for the crawl. Shortcut to the corresponding {@link AutoscaledPool} option.
+ * @param {Boolean} [options.useSessionPool=false]
+ * If set to true Crawler will automatically use Session Pool. It will automatically retire sessions on 403, 401 and 429 status codes.
+ * It also marks Session as bad after a request timeout.
+ * @param {Object} [options.sessionPoolOptions]
+ * Custom options passed to the underlying {@link SessionPool} constructor.
+ * @param {Boolean} [options.persistCookiesPerSession]
+ * Automatically saves cookies to Session. Works only if Session Pool is used.
+ *
+ * It parses cookie from response "set-cookie" header saves or updates cookies for session and once the session is used for next request.
+ * It passes the "Cookie" header to the request with the session cookies.
*/
class CheerioCrawler {
constructor(options = {}) {
diff --git a/src/utils.js b/src/utils.js
index 76c5530e6..eb86d2699 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -673,7 +673,7 @@ export const parseContentTypeFromResponse = (response) => {
/**
*
* @param response
- * @return {undefined|Cookie[]}
+ * @return {undefined|Array}
*/
export const getCookiesFromResponse = (response) => {
const { headers } = response;
@@ -690,7 +690,7 @@ export const getCookiesFromResponse = (response) => {
/**
* Gets cookie string for the "Cookie" header
- * @param cookies {[Cookie]}
+ * @param cookies {Array}
* @return {String|undefined}
*/
export const getCookieHeader = (cookies) => {
diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.js
index 94840d5bb..15c1d2f8e 100644
--- a/test/crawlers/cheerio_crawler.test.js
+++ b/test/crawlers/cheerio_crawler.test.js
@@ -920,7 +920,6 @@ describe('CheerioCrawler', () => {
});
await crawler.run();
- console.log(requests);
expect(sessions.length).toBeGreaterThan(1);
sessions.forEach((session) => {
expect(session.cookies).toHaveLength(1);