Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Ondra Urband39b263212c3
diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex b20182111..041c2926e 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -886,10 +886,14 @@ const requestList = await Apify.openRequestList('my-name', sources);<td><code>sources</code></td><td><code>Array<Object></code> | <code>Array<string></code></td></tr><tr>-<td colspan="3"><p>Sources represent the URLs to crawl. It can either be a <code>string[]</code> with plain URLs or an <code>Object[]</code>.- The objects' contents can either be just plain objects, defining at least the 'url' property- or instances of the <a href="request"><code>Request</code></a> class. See the (<code>new RequestList</code>)(RequestList#new_RequestList_new)- options for details.</p>+<td colspan="3"><p>An array of sources of URLs for the <code>RequestList</code>.+ It can be either an array of plain objects that+ define the <code>url</code> property, or an array of instances of the <a href="request"><code>Request</code></a> class.+ Additionally, the <code>requestsFromUrl</code> property may be used instead of <code>url</code>,+ which will instruct <code>RequestList</code> to download the source URLs from a given remote location.+ The URLs will be parsed from the received response.</p>+<p> For details, see the (<code>RequestList</code>)[requestlist#new-exportsrequestlistoptions]+ constructor options.</p></td></tr><tr><td><code>[options]</code></td><td><code>Object</code></td></tr>diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex c5e17ca5d..04d771682 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -4,9 +4,14 @@ title: BasicCrawler---<a name="BasicCrawler"></a>-Provides a simple framework for parallel crawling of web pages,-whose URLs are fed either from a static list-or from a dynamic queue of URLs.+Provides a simple framework for parallel crawling of web pages.+The URLs to crawl are fed either from a static list of URLs+or from a dynamic queue of URLs enabling recursive crawling of websites.++`BasicCrawler` is a low-level tool that requires the user to implement the page+download and data extraction functionality themselves.+If you want a crawler that already facilitates this functionality,+please consider using [`PuppeteerCrawler`](puppeteercrawler) or [`CheerioCrawler`](cheeriocrawler).`BasicCrawler` invokes the user-provided [`handleRequestFunction()`](#new_BasicCrawler_new)for each [`Request`](request) object, which represents a single URL to crawl.diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex b3e1834f4..92ae578bc 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -6,6 +6,13 @@ title: CheerioCrawlerProvides a framework for the parallel crawling of web pages using plain HTTP requests and<a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> HTML parser.+The URLs to crawl are fed either from a static list of URLs+or from a dynamic queue of URLs enabling recursive crawling of websites.++Since `CheerioCrawler` uses raw HTTP requests to download web pages,+it is very fast and efficient on data bandwidth. However, if the target website requires JavaScript+to display the content, you might need to use [`PuppeteerCrawler`](puppeteercrawler) instead,+because it loads the pages using full-featured headless Chrome browser.`CheerioCrawler` downloads each URL using a plain HTTP request,parses the HTML content using <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex e0564a5d8..d7bd7fbd8 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -6,6 +6,13 @@ title: PuppeteerCrawlerProvides a simple framework for parallel crawling of web pagesusing headless Chrome with <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a>.+The URLs to crawl are fed either from a static list of URLs+or from a dynamic queue of URLs enabling recursive crawling of websites.++Since `PuppeteerCrawler` uses headless Chrome to download web pages and extract data,+it is useful for crawling of websites that require to execute JavaScript.+If the target website doesn't need JavaScript, consider using [`CheerioCrawler`](cheeriocrawler),+which downloads the pages using raw HTTP requests and is about 10x faster.The source URLs are represented using [`Request`](request) objects that are fed from[`RequestList`](requestlist) or [`RequestQueue`](requestqueue) instances provided by the [`requestList`](#new_PuppeteerCrawler_new)diff --git a/docs/api/RequestList.md b/docs/api/RequestList.mdindex 826a77261..e3148655f 100644--- a/docs/api/RequestList.md+++ b/docs/api/RequestList.md@@ -100,10 +100,10 @@ await requestList.reclaimRequest(request2);<td><code>options.sources</code></td><td><code>Array</code></td><td></td></tr><tr>-<td colspan="3"><p>An array of sources for the <code>RequestList</code>. Its contents can either be just plain objects,- defining at least the 'url' property or instances of the <a href="request"><code>Request</code></a> class.- Additionally a <code>requestsFromUrl</code> property may be used instead of <code>url</code>,- which will instruct the <code>RequestList</code> to download the sources from the given remote location.+<td colspan="3"><p>An array of sources of URLs for the <code>RequestList</code>. It can be either an array of plain objects that+ define the <code>url</code> property, or an array of instances of the <a href="request"><code>Request</code></a> class.+ Additionally, the <code>requestsFromUrl</code> property may be used instead of <code>url</code>,+ which will instruct <code>RequestList</code> to download the source URLs from a given remote location.The URLs will be parsed from the received response.</p><pre><code>[// One URL