Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Added docs
Petr Pateka27bfe324a02
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 9bbfb52d3..c4e85c1ac 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -198,6 +198,18 @@ await crawler.run();<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>for the default implementation of this function.</p></td></tr><tr>+<td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>A function that executes before calling the request.+ This function is suitable for setting dynamic properties such as cookies to <a href="request"><code>Request</code></a>.</p>+<p> The function receives the following object as an argument:</p>+<pre><code>{+ request: Request+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>+<p> The function Should always return <a href="request"><code>Request</code></a>.</p>+</td></tr><tr><td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td></tr><tr>diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.jsindex 6ce89a7eb..fd299f96b 100644--- a/src/cheerio_crawler.js+++ b/src/cheerio_crawler.js@@ -184,7 +184,7 @@ const DEFAULT_OPTIONS = {* for the default implementation of this function.* @param {Function} [options.prepareRequestFunction]* A function that executes before calling the request.- * This function is suitable for setting dynamic properties such as cookies to {@link Request}.+ * This function is suitable for setting dynamic properties such as cookies to the {@link Request}.** The function receives the following object as an argument:* ```@@ -192,8 +192,9 @@ const DEFAULT_OPTIONS = {* request: Request* }* ```- * where the {@link Request} instance corresponds to the failed request.- * The function Should always return {@link Request}.+ * where the {@link Request} instance corresponds to the initialized request.+ *+ * The function should always return {@link Request}.** @param {Number} [options.maxRequestRetries=3]* Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.