Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Fix lint, build docs
Ondra Urban4aa205c97191
diff --git a/docs/api/Dataset.md b/docs/api/Dataset.mdindex 5adde4d89..5bf9c75ff 100644--- a/docs/api/Dataset.md+++ b/docs/api/Dataset.md@@ -52,7 +52,7 @@ await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);- [`.forEach(iteratee, [options], [index])`](#Dataset+forEach) ⇒ `Promise`- [`.map(iteratee, options)`](#Dataset+map) ⇒ `Promise<Array>`- [`.reduce(iteratee, memo, options)`](#Dataset+reduce) ⇒ `Promise<*>`- - [`.delete()`](#Dataset+delete) ⇒ `Promise`+ - [`.drop()`](#Dataset+drop) ⇒ `Promise`<a name="Dataset+pushData"></a>@@ -411,8 +411,8 @@ If `iteratee()` returns a `Promise` then it's awaited before a next call.<td colspan="3"><p>How many items to load in one request.</p></td></tr></tbody></table>-<a name="Dataset+delete"></a>+<a name="Dataset+drop"></a>-## `dataset.delete()` ⇒ `Promise`+## `dataset.drop()` ⇒ `Promise`Removes the dataset either from the Apify cloud storage or from the local directory, depending on the mode of operation.diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.mdindex fe8251d09..14e6e5724 100644--- a/docs/api/KeyValueStore.md+++ b/docs/api/KeyValueStore.md@@ -57,14 +57,14 @@ await store.setValue('some-key', { foo: 'bar' });// text data returned as a string and other data is returned as binary bufferconst value = await store.getValue('some-key');-// Delete record-await store.drop('some-key');+// Drop (delete) the store+await store.drop();```- [KeyValueStore](keyvaluestore)- [`.getValue(key)`](#KeyValueStore+getValue) ⇒ `Promise<(Object|String|Buffer)>`- [`.setValue(key, value, [options])`](#KeyValueStore+setValue) ⇒ `Promise`- - [`.delete()`](#KeyValueStore+delete) ⇒ `Promise`+ - [`.drop()`](#KeyValueStore+drop) ⇒ `Promise`- [`.getPublicUrl(key)`](#KeyValueStore+getPublicUrl) ⇒ `string`- [`.forEachKey(iteratee, [options])`](#KeyValueStore+forEachKey) ⇒ `Promise`@@ -177,9 +177,9 @@ stored!<td colspan="3"><p>Specifies a custom MIME content type of the record.</p></td></tr></tbody></table>-<a name="KeyValueStore+delete"></a>+<a name="KeyValueStore+drop"></a>-## `keyValueStore.delete()` ⇒ `Promise`+## `keyValueStore.drop()` ⇒ `Promise`Removes the key-value store either from the Apify cloud storage or from the local directory, depending on the mode of operation.diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.mdindex 34085f677..55bdb228e 100644--- a/docs/api/RequestQueue.md+++ b/docs/api/RequestQueue.md@@ -71,7 +71,7 @@ await queue.reclaimRequest(request2);- [`.reclaimRequest(request, [options])`](#RequestQueue+reclaimRequest) ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)- [`.isEmpty()`](#RequestQueue+isEmpty) ⇒ `Promise<Boolean>`- [`.isFinished()`](#RequestQueue+isFinished) ⇒ `Promise<Boolean>`- - [`.delete()`](#RequestQueue+delete) ⇒ `Promise`+ - [`.drop()`](#RequestQueue+drop) ⇒ `Promise`- ~~[`.handledCount()`](#RequestQueue+handledCount) ⇒ `Promise<number>`~~- [`.getInfo()`](#RequestQueue+getInfo) ⇒ `Promise<Object>`@@ -211,9 +211,9 @@ activity in the queue, use [`RequestQueue.isFinished`](requestqueue.isfinished).Resolves to `true` if all requests were already handled and there are no more left. Due to the nature of distributed storage used by the queue, thefunction might occasionally return a false negative, but it will never return a false positive.-<a name="RequestQueue+delete"></a>+<a name="RequestQueue+drop"></a>-## `requestQueue.delete()` ⇒ `Promise`+## `requestQueue.drop()` ⇒ `Promise`Removes the queue either from the Apify Cloud storage or from the local directory, depending on the mode of operation.diff --git a/src/dataset.js b/src/dataset.jsindex 9025c8dd9..9ecce9ddd 100644--- a/src/dataset.js+++ b/src/dataset.js@@ -4,10 +4,10 @@ import fs from 'fs-extra';import _ from 'underscore';import { leftpad } from 'apify-shared/utilities';import LruCache from 'apify-shared/lru_cache';+import log from 'apify-shared/log';import { checkParamOrThrow } from 'apify-client/build/utils';import { ENV_VARS, LOCAL_STORAGE_SUBDIRS, MAX_PAYLOAD_SIZE_BYTES } from 'apify-shared/consts';import { apifyClient, ensureDirExists, openRemoteStorage, openLocalStorage, ensureTokenOrLocalStorageEnvExists } from './utils';-import log from 'apify-shared/log';export const LOCAL_STORAGE_SUBDIR = LOCAL_STORAGE_SUBDIRS.datasets;export const LOCAL_FILENAME_DIGITS = 9;