Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Ondra Urband199fe75c77c
diff --git a/docs/api/Dataset.md b/docs/api/Dataset.mdindex 3ce616678..f0217b5f0 100644--- a/docs/api/Dataset.md+++ b/docs/api/Dataset.md@@ -114,6 +114,11 @@ has the following structure:}```+**NOTE**: If using dataset with local disk storage, the `format` option must be `json` and+the following options are not supported:+`unwind`, `disableBodyParser`, `attachment`, `bom` and `simplified`.+If you try to use them, you will receive an error.+<table><thead><tr>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 059741515..374817028 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -54,7 +54,7 @@ const crawler = new Apify.PuppeteerCrawler({})},handleFailedRequestFunction: async ({ request }) => {- // This function is called when crawling of a request failed too many time+ // This function is called when the crawling of a request failed too many timesawait Apify.pushData({url: request.url,succeeded: false,diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.mdindex 87ae1d9f4..79cac7229 100644--- a/docs/api/RequestQueue.md+++ b/docs/api/RequestQueue.md@@ -227,6 +227,12 @@ depending on the mode of operation.Returns the number of handled requests.+This function is just a convenient shortcut for:++```javascript+const { handledRequestCount } = await queue.getInfo();+```+<a name="RequestQueue+getInfo"></a>## `requestQueue.getInfo()` ⇒ `Promise<Object>`@@ -247,9 +253,9 @@ API endpoint.createdAt: new Date("2015-12-12T07:34:14.202Z"),modifiedAt: new Date("2015-12-13T08:36:13.202Z"),accessedAt: new Date("2015-12-14T08:36:13.202Z"),- totalRequestCount: 0,- handledRequestCount: 0,- pendingRequestCount: 0,+ totalRequestCount: 25,+ handledRequestCount: 5,+ pendingRequestCount: 20,}```