Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Build docs

Ondra Urband199fe75c77c
diff --git a/docs/api/Dataset.md b/docs/api/Dataset.md
index 3ce616678..f0217b5f0 100644
--- a/docs/api/Dataset.md
+++ b/docs/api/Dataset.md
@@ -114,6 +114,11 @@ has the following structure:
}
```
+**NOTE**: If using dataset with local disk storage, the `format` option must be `json` and
+the following options are not supported:
+`unwind`, `disableBodyParser`, `attachment`, `bom` and `simplified`.
+If you try to use them, you will receive an error.
+
<table>
<thead>
<tr>
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.md
index 059741515..374817028 100644
--- a/docs/api/PuppeteerCrawler.md
+++ b/docs/api/PuppeteerCrawler.md
@@ -54,7 +54,7 @@ const crawler = new Apify.PuppeteerCrawler({
})
},
handleFailedRequestFunction: async ({ request }) => {
- // This function is called when crawling of a request failed too many time
+ // This function is called when the crawling of a request failed too many times
await Apify.pushData({
url: request.url,
succeeded: false,
diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.md
index 87ae1d9f4..79cac7229 100644
--- a/docs/api/RequestQueue.md
+++ b/docs/api/RequestQueue.md
@@ -227,6 +227,12 @@ depending on the mode of operation.
Returns the number of handled requests.
+This function is just a convenient shortcut for:
+
+```javascript
+const { handledRequestCount } = await queue.getInfo();
+```
+
<a name="RequestQueue+getInfo"></a>
## `requestQueue.getInfo()` ⇒ `Promise<Object>`
@@ -247,9 +253,9 @@ API endpoint.
createdAt: new Date("2015-12-12T07:34:14.202Z"),
modifiedAt: new Date("2015-12-13T08:36:13.202Z"),
accessedAt: new Date("2015-12-14T08:36:13.202Z"),
- totalRequestCount: 0,
- handledRequestCount: 0,
- pendingRequestCount: 0,
+ totalRequestCount: 25,
+ handledRequestCount: 5,
+ pendingRequestCount: 20,
}
```