Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Improved docs
Jan Curn877ae04dbd4e
diff --git a/src/basic_crawler.js b/src/basic_crawler.jsindex 29dbc29ba..f6b6fbbf5 100644--- a/src/basic_crawler.js+++ b/src/basic_crawler.js@@ -80,7 +80,16 @@ const DEFAULT_OPTIONS = {* }* ```* where the {@link Request} instance represents the URL to crawl.- * The function must return a promise.+ *+ * The function must return a promise, which is then awaited by the crawler.+ *+ * If the function throws an exception, the crawler will try to re-crawl the+ * request later, up to `option.maxRequestRetries` times.+ * If all the retries fail, the crawler calls the function+ * provided to the `options.handleFailedRequestFunction` parameter.+ * To make this work, you should **always**+ * let your function throw exceptions rather than catch them.+ * The exceptions are logged to the request using the {@link Request.pushErrorMessage} function.* @param {RequestList} options.requestList* Static list of URLs to be processed.* Either `requestList` or `requestQueue` option must be provided (or both).diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.jsindex 012f61c80..366bb297f 100644--- a/src/cheerio_crawler.js+++ b/src/cheerio_crawler.js@@ -98,11 +98,20 @@ const DEFAULT_OPTIONS = {* $: Cheerio, // the Cheerio object with parsed HTML* html: String // the raw HTML of the page* request: Request,- * response: Object // a http.IncomingMessage object with properties such as the `statusCode`+ * response: Object // An instance of Node's http.IncomingMessage object* }* ```* With the {@link Request} object representing the URL to crawl.- * If the function returns a promise, it is awaited.+ *+ * If the function returns a promise, it is awaited by the crawler.+ *+ * If the function throws an exception, the crawler will try to re-crawl the+ * request later, up to `option.maxRequestRetries` times.+ * If all the retries fail, the crawler calls the function+ * provided to the `options.handleFailedRequestFunction` parameter.+ * To make this work, you should **always**+ * let your function throw exceptions rather than catch them.+ * The exceptions are logged to the request using the {@link Request.pushErrorMessage} function.* @param {RequestList} options.requestList* Static list of URLs to be processed.* Either `requestList` or `requestQueue` option must be provided (or both).diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.jsindex 6075b7c0f..a9433e5a0 100644--- a/src/puppeteer_crawler.js+++ b/src/puppeteer_crawler.js@@ -93,6 +93,16 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;* <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-response" target="_blank"><code>Response</code></a>,* which is the main resource response as returned by `page.goto(request.url)`.* `puppeteerPool` is an instance of the {@link PuppeteerPool} used by this `PuppeteerCrawler`.+ *+ * The function must return a promise, which is then awaited by the crawler.+ *+ * If the function throws an exception, the crawler will try to re-crawl the+ * request later, up to `option.maxRequestRetries` times.+ * If all the retries fail, the crawler calls the function+ * provided to the `options.handleFailedRequestFunction` parameter.+ * To make this work, you should **always**+ * let your function throw exceptions rather than catch them.+ * The exceptions are logged to the request using the {@link Request.pushErrorMessage} function.* @param {RequestList} options.requestList* Static list of URLs to be processed.* Either `requestList` or `requestQueue` option must be provided (or both).diff --git a/src/puppeteer_utils.js b/src/puppeteer_utils.jsindex ccccaf3b7..dddae8926 100644--- a/src/puppeteer_utils.js+++ b/src/puppeteer_utils.js@@ -79,16 +79,18 @@ const injectFile = async (page, filePath) => {/*** Injects the <a href="https://jquery.com/" target="_blank"><code>jQuery</code></a> library into a Puppeteer page.- * jQuery is often useful for various web scraping and crawling tasks,- * e.g. to extract data from HTML elements using CSS selectors.+ * jQuery is often useful for various web scraping and crawling tasks.+ * For example, it can help extract text from HTML elements using CSS selectors.** Beware that the injected jQuery object will be set to the `window.$` variable and thus it might cause conflicts with- * libraries included by the page that use the same variable (e.g. another version of jQuery).+ * other libraries included by the page that use the same variable name (e.g. another version of jQuery).+ * This can affect functionality of page's scripts.*- * Also make sure that you're injecting jQuery after the page has loaded (after `page.goto()`). Otherwise,- * the navigation will override the existing environment and the library will no longer be available.+ * Also make sure that you're injecting jQuery after the page has loaded+ * (i.e. after [`page.goto()`](https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options)).+ * Otherwise, the navigation will override the existing environment and the library will no longer be available.*- * Example usage:+ * **Example usage:*** ```javascript* await Apify.utils.puppeteer.injectJQuery(page);* const title = await page.evaluate(() => {@@ -114,11 +116,22 @@ const injectJQuery = (page) => {/*** Injects the <a href="https://underscorejs.org/" target="_blank"><code>Underscore.js</code></a> library into a Puppeteer page.+ ** Beware that the injected Underscore object will be set to the `window._` variable and thus it might cause conflicts with- * libraries included by the page that use the same variable.+ * libraries included by the page that use the same variable name.+ * This can affect functionality of page's scripts.*- * Also make sure that you're injecting Underscore after the page has loaded (after `page.goto()`). Otherwise,- * the navigation will override the existing environment and the library will no longer be available.+ * Also make sure that you're injecting Underscore after the page has loaded+ * (i.e. after [`page.goto()`](https://pptr.dev/#?product=Puppeteer&show=api-pagegotourl-options)).+ * Otherwise, the navigation will override the existing environment and the library will no longer be available.+ *+ * **Example usage:**+ * ```javascript+ * await Apify.utils.puppeteer.injectUnderscore(page);+ * const escapedHtml = await page.evaluate(() => {+ * return _.escape('<h1>Hello</h1>');+ * });+ * ```** @param {Page} page Puppeteer [Page](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page) object.* @return {Promise}@@ -305,11 +318,11 @@ const enqueueLinks = async (...args) => {};/**- * Forces the browser tab to block loading certain page resources,- * using the `Page.setRequestInterception(value)` method.- * This is useful to speed up crawling of websites.+ * Forces the Puppeteer browser tab to block loading certain HTTP resources.+ * This is useful to speed up crawling of websites, since it reduces the amount+ * of data that need to be downloaded from the web.*- * The resource types to block can be controlled using the `resourceTypes` parameter,+ * The resource types to block can be specified using the `resourceTypes` parameter,* which indicates the types of resources as they are perceived by the rendering engine.* The following resource types are currently supported:* `document`, `stylesheet`, `image`, `media`, `font`, `script`, `texttrack`, `xhr`, `fetch`,@@ -320,6 +333,26 @@ const enqueueLinks = async (...args) => {* If the `resourceTypes` parameter is not provided,* by default the function blocks these resource types: `stylesheet`, `font`, `image`, `media`.*+ * Note that the `blockResources` function internally uses Puppeteer's+ * [`Page.setRequestInterception()`](https://pptr.dev/#?product=Puppeteer&show=api-pagesetrequestinterceptionvalue) function,+ * which can only be used once per `Page` object.+ *+ * **Example usage**+ * ```javascript+ * const Apify = require('apify');+ *+ * const browser = await Apify.launchPuppeteer();+ * const page = await browser.newPage();+ *+ * // Block all resources except for the main HTML document+ * await Apify.utils.puppeteer.blockResources(page,+ * ['stylesheet', 'image', 'media', 'font', 'script', 'texttrack', 'xhr',+ * 'fetch', 'eventsource', 'websocket', 'manifest', 'other']+ * );+ *+ * await page.goto('https://www.example.com');+ * ```+ ** @param {Page} page* Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.* @param {String[]} [resourceTypes=['stylesheet', 'font', 'image', 'media']]@@ -447,7 +480,8 @@ const compileScript = (scriptString, context = Object.create(null)) => {/**- * A namespace that contains various Puppeteer utilities.+ * A namespace that contains various utilities for+ * [Puppeteer](https://github.com/GoogleChrome/puppeteer) - the headless Chrome Node API.** **Example usage:***