Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Reword docs
Ondra Urbanab07d113e262
diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.jsindex fd299f96b..e9a128486 100644--- a/src/cheerio_crawler.js+++ b/src/cheerio_crawler.js@@ -140,6 +140,20 @@ const DEFAULT_OPTIONS = {* - `url`, `method`, `headers`: provided by `requestList` and/or `requestQueue`* - `strictSSL`: use `options.ignoreSslErrors`* - `proxy`: use `options.useApifyProxy` or `options.proxyUrls`+ *+ * @param {Function} [options.prepareRequestFunction]+ * A function that executes before the HTTP request is made to the target resource.+ * This function is suitable for setting dynamic properties such as cookies to the {@link Request}.+ *+ * The function receives the following object as an argument:+ * ```+ * {+ * request: Request+ * }+ * ```+ * where the {@link Request} instance corresponds to the initialized request.+ *+ * The function should always return {@link Request}.** @param {Number} [options.handlePageTimeoutSecs=60]* Timeout in which the function passed as `options.handlePageFunction` needs to finish, given in seconds.@@ -182,20 +196,6 @@ const DEFAULT_OPTIONS = {** See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>* for the default implementation of this function.- * @param {Function} [options.prepareRequestFunction]- * A function that executes before calling the request.- * This function is suitable for setting dynamic properties such as cookies to the {@link Request}.- *- * The function receives the following object as an argument:- * ```- * {- * request: Request- * }- * ```- * where the {@link Request} instance corresponds to the initialized request.- *- * The function should always return {@link Request}.- ** @param {Number} [options.maxRequestRetries=3]* Indicates how many times the request is retried if either `requestFunction` or `handlePageFunction` fails.* @param {Number} [options.maxRequestsPerCrawl]